2024-09-11T06:01:25.606 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-11T06:01:25.627 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-11T06:01:25.727 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-09-10_21:16:03-rbd-squid-distro-default-smithi/7900605 branch: squid description: rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-zlib supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-stress-workunit-min-compat-client-octopus} email: ceph-qa@ceph.com first_in_suite: false job_id: '7900605' kernel: kdb: 1 sha1: distro ktype: distro last_in_suite: false machine_type: smithi meta: - desc: run two ceph clusters and install rbd-mirror - desc: configure the permissions for client.mirror - desc: configure the permissions for client.mirror - desc: 2 ceph clusters with 1 mon and 3 osds each - desc: run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon name: teuthology-2024-09-10_21:16:03-rbd-squid-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 3 size: 30 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: squid ceph: conf: client: debug rbd: 20 debug rbd_mirror: 15 log to stderr: false rbd default features: 125 client.mirror.0: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.1: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.2: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.3: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.4: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.5: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.6: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid global: mon client directed command retry: 5 mon warn on pool no app: false ms inject socket failures: 5000 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bluestore block size: 96636764160 bluestore compression algorithm: zlib bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME sha1: b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: squid sha1: daaf78de334647fc0c800f4ac495cc9475a26bae owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - cluster1.mon.a - cluster1.mgr.x - cluster2.mgr.x - cluster1.osd.0 - cluster1.osd.1 - cluster1.osd.2 - cluster1.client.0 - cluster2.client.0 - - cluster2.mon.a - cluster2.osd.0 - cluster2.osd.1 - cluster2.osd.2 - cluster1.client.mirror - cluster1.client.mirror.0 - cluster1.client.mirror.1 - cluster1.client.mirror.2 - cluster1.client.mirror.3 - cluster1.client.mirror.4 - cluster1.client.mirror.5 - cluster1.client.mirror.6 - cluster2.client.mirror - cluster2.client.mirror.0 - cluster2.client.mirror.1 - cluster2.client.mirror.2 - cluster2.client.mirror.3 - cluster2.client.mirror.4 - cluster2.client.mirror.5 - cluster2.client.mirror.6 seed: 6175 sha1: b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc sleep_before_teardown: 0 subset: 88/128 suite: rbd suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_daaf78de334647fc0c800f4ac495cc9475a26bae/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: daaf78de334647fc0c800f4ac495cc9475a26bae targets: smithi006.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBO0WRxR505R28vAPOaiTE3OCExO5dRsDtnACcGHqGLEups3p4DMCG5xRwBug/cQh6QcewlLrbWsCJohOpKXaaQg= smithi112.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJfTs0r1xAi00gNyDf/PIuEnNrqSpWYDR6gneJW9eH9P26G5v230dQe7xH8BPEZImnabz26XbGX7EdknHM19Ris= tasks: - install: extra_packages: - rbd-mirror - ceph: cluster: cluster1 - ceph: cluster: cluster2 - exec: cluster1.client.mirror.0: - sudo ceph --cluster cluster1 auth caps client.mirror.4 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.5 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.6 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.4 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.5 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.6 mon 'profile rbd-mirror' osd 'profile rbd' - exec: cluster1.client.mirror.0: - sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd' - workunit: clients: cluster1.client.mirror: - rbd/rbd_mirror_stress.sh env: CEPH_ARGS: '' RBD_MIRROR_MIN_COMPAT_CLIENT: octopus RBD_MIRROR_USE_EXISTING_CLUSTER: '1' teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-10_21:16:03 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-11T06:01:25.727 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_daaf78de334647fc0c800f4ac495cc9475a26bae/qa; will attempt to use it 2024-09-11T06:01:25.728 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_daaf78de334647fc0c800f4ac495cc9475a26bae/qa/tasks 2024-09-11T06:01:25.728 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-11T06:01:25.730 INFO:teuthology.task.internal:Checking packages... 2024-09-11T06:01:25.750 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash 'b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc' 2024-09-11T06:01:25.751 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-11T06:01:25.751 INFO:teuthology.packaging:ref: None 2024-09-11T06:01:25.751 INFO:teuthology.packaging:tag: None 2024-09-11T06:01:25.751 INFO:teuthology.packaging:branch: squid 2024-09-11T06:01:25.751 INFO:teuthology.packaging:sha1: b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc 2024-09-11T06:01:25.751 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=squid 2024-09-11T06:01:25.979 INFO:teuthology.task.internal:Found packages for ceph version 19.1.1-298-gb53b6d3e-1jammy 2024-09-11T06:01:25.981 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-11T06:01:25.987 INFO:teuthology.task.internal:no buildpackages task found 2024-09-11T06:01:25.988 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-11T06:01:25.993 INFO:teuthology.task.internal:Saving configuration 2024-09-11T06:01:26.010 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-11T06:01:26.017 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-11T06:01:26.039 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi006.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-10_21:16:03-rbd-squid-distro-default-smithi/7900605', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-11 05:55:49.572874', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOJ8YnYNNOwS/pkAio8XBoQxmOOCKPNAR0pjJVm0Dt7i'} 2024-09-11T06:01:26.063 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi112.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-10_21:16:03-rbd-squid-distro-default-smithi/7900605', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-11 05:55:49.574092', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHVXSj3nKvkEsP9H9uGQLBZATkqsy1YC5JG8WTF4EkUZ'} 2024-09-11T06:01:26.063 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-11T06:01:26.069 INFO:teuthology.task.internal:roles: ubuntu@smithi006.front.sepia.ceph.com - ['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'] 2024-09-11T06:01:26.069 INFO:teuthology.task.internal:roles: ubuntu@smithi112.front.sepia.ceph.com - ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6'] 2024-09-11T06:01:26.069 INFO:teuthology.run_tasks:Running task console_log... 2024-09-11T06:01:26.132 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f2e8b52af80>, signals=[15]) 2024-09-11T06:01:26.132 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-11T06:01:26.138 INFO:teuthology.task.internal:Opening connections... 2024-09-11T06:01:26.139 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi006.front.sepia.ceph.com 2024-09-11T06:01:26.140 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi006.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T06:01:26.222 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi112.front.sepia.ceph.com 2024-09-11T06:01:26.223 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T06:01:26.302 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-11T06:01:26.308 DEBUG:teuthology.orchestra.run.smithi006:> uname -m 2024-09-11T06:01:26.313 INFO:teuthology.orchestra.run.smithi006.stdout:x86_64 2024-09-11T06:01:26.313 DEBUG:teuthology.orchestra.run.smithi006:> cat /etc/os-release 2024-09-11T06:01:26.358 INFO:teuthology.orchestra.run.smithi006.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-09-11T06:01:26.359 INFO:teuthology.orchestra.run.smithi006.stdout:NAME="Ubuntu" 2024-09-11T06:01:26.359 INFO:teuthology.orchestra.run.smithi006.stdout:VERSION_ID="22.04" 2024-09-11T06:01:26.359 INFO:teuthology.orchestra.run.smithi006.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-09-11T06:01:26.359 INFO:teuthology.orchestra.run.smithi006.stdout:VERSION_CODENAME=jammy 2024-09-11T06:01:26.359 INFO:teuthology.orchestra.run.smithi006.stdout:ID=ubuntu 2024-09-11T06:01:26.359 INFO:teuthology.orchestra.run.smithi006.stdout:ID_LIKE=debian 2024-09-11T06:01:26.359 INFO:teuthology.orchestra.run.smithi006.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-11T06:01:26.359 INFO:teuthology.orchestra.run.smithi006.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-11T06:01:26.359 INFO:teuthology.orchestra.run.smithi006.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-11T06:01:26.359 INFO:teuthology.orchestra.run.smithi006.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-11T06:01:26.360 INFO:teuthology.orchestra.run.smithi006.stdout:UBUNTU_CODENAME=jammy 2024-09-11T06:01:26.360 INFO:teuthology.lock.ops:Updating smithi006.front.sepia.ceph.com on lock server 2024-09-11T06:01:26.382 DEBUG:teuthology.orchestra.run.smithi112:> uname -m 2024-09-11T06:01:26.387 INFO:teuthology.orchestra.run.smithi112.stdout:x86_64 2024-09-11T06:01:26.387 DEBUG:teuthology.orchestra.run.smithi112:> cat /etc/os-release 2024-09-11T06:01:26.433 INFO:teuthology.orchestra.run.smithi112.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-09-11T06:01:26.433 INFO:teuthology.orchestra.run.smithi112.stdout:NAME="Ubuntu" 2024-09-11T06:01:26.434 INFO:teuthology.orchestra.run.smithi112.stdout:VERSION_ID="22.04" 2024-09-11T06:01:26.434 INFO:teuthology.orchestra.run.smithi112.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-09-11T06:01:26.434 INFO:teuthology.orchestra.run.smithi112.stdout:VERSION_CODENAME=jammy 2024-09-11T06:01:26.434 INFO:teuthology.orchestra.run.smithi112.stdout:ID=ubuntu 2024-09-11T06:01:26.434 INFO:teuthology.orchestra.run.smithi112.stdout:ID_LIKE=debian 2024-09-11T06:01:26.434 INFO:teuthology.orchestra.run.smithi112.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-11T06:01:26.434 INFO:teuthology.orchestra.run.smithi112.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-11T06:01:26.434 INFO:teuthology.orchestra.run.smithi112.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-11T06:01:26.434 INFO:teuthology.orchestra.run.smithi112.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-11T06:01:26.434 INFO:teuthology.orchestra.run.smithi112.stdout:UBUNTU_CODENAME=jammy 2024-09-11T06:01:26.435 INFO:teuthology.lock.ops:Updating smithi112.front.sepia.ceph.com on lock server 2024-09-11T06:01:26.456 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-11T06:01:26.468 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-11T06:01:26.473 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-11T06:01:26.473 DEBUG:teuthology.orchestra.run.smithi006:> test '!' -e /home/ubuntu/cephtest 2024-09-11T06:01:26.475 DEBUG:teuthology.orchestra.run.smithi112:> test '!' -e /home/ubuntu/cephtest 2024-09-11T06:01:26.484 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-11T06:01:26.490 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-11T06:01:26.490 DEBUG:teuthology.orchestra.run.smithi006:> test -z $(ls -A /var/lib/ceph) 2024-09-11T06:01:26.520 DEBUG:teuthology.orchestra.run.smithi112:> test -z $(ls -A /var/lib/ceph) 2024-09-11T06:01:26.610 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-11T06:01:26.665 INFO:teuthology.run_tasks:Running task kernel... 2024-09-11T06:01:26.680 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-11T06:01:26.680 INFO:teuthology.task.kernel:config {'cluster1.mon.a': {'kdb': 1, 'sha1': 'distro'}, 'cluster2.mon.a': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-11T06:01:26.681 DEBUG:teuthology.orchestra.run.smithi006:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-11T06:01:26.681 DEBUG:teuthology.orchestra.run.smithi112:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-11T06:01:26.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T06:01:26.686 DEBUG:teuthology.orchestra.run.smithi006:> uname -r 2024-09-11T06:01:26.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T06:01:26.687 DEBUG:teuthology.orchestra.run.smithi112:> uname -r 2024-09-11T06:01:26.734 INFO:teuthology.orchestra.run.smithi112.stdout:5.15.0-56-generic 2024-09-11T06:01:26.734 INFO:teuthology.task.kernel:Running kernel on smithi112: 5.15.0-56-generic 2024-09-11T06:01:26.734 DEBUG:teuthology.orchestra.run.smithi112:> sudo apt-get clean 2024-09-11T06:01:26.735 INFO:teuthology.orchestra.run.smithi006.stdout:5.15.0-56-generic 2024-09-11T06:01:26.735 INFO:teuthology.task.kernel:Running kernel on smithi006: 5.15.0-56-generic 2024-09-11T06:01:26.736 DEBUG:teuthology.orchestra.run.smithi006:> sudo apt-get clean 2024-09-11T06:01:26.849 DEBUG:teuthology.orchestra.run.smithi112:> sudo apt-get update 2024-09-11T06:01:26.860 DEBUG:teuthology.orchestra.run.smithi006:> sudo apt-get update 2024-09-11T06:01:26.983 INFO:teuthology.orchestra.run.smithi112.stdout:Get:1 http://security.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2024-09-11T06:01:27.001 INFO:teuthology.orchestra.run.smithi006.stdout:Get:1 http://security.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2024-09-11T06:01:27.101 INFO:teuthology.orchestra.run.smithi112.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-11T06:01:27.110 INFO:teuthology.orchestra.run.smithi006.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-11T06:01:27.184 INFO:teuthology.orchestra.run.smithi112.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2024-09-11T06:01:27.192 INFO:teuthology.orchestra.run.smithi006.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2024-09-11T06:01:27.201 INFO:teuthology.orchestra.run.smithi112.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [533 kB] 2024-09-11T06:01:27.209 INFO:teuthology.orchestra.run.smithi006.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,771 kB] 2024-09-11T06:01:27.289 INFO:teuthology.orchestra.run.smithi112.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,771 kB] 2024-09-11T06:01:27.312 INFO:teuthology.orchestra.run.smithi006.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [533 kB] 2024-09-11T06:01:27.316 INFO:teuthology.orchestra.run.smithi006.stdout:Get:6 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [291 kB] 2024-09-11T06:01:27.318 INFO:teuthology.orchestra.run.smithi006.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [13.3 kB] 2024-09-11T06:01:27.319 INFO:teuthology.orchestra.run.smithi006.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [37.7 kB] 2024-09-11T06:01:27.319 INFO:teuthology.orchestra.run.smithi006.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [2,327 kB] 2024-09-11T06:01:27.327 INFO:teuthology.orchestra.run.smithi112.stdout:Get:6 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [291 kB] 2024-09-11T06:01:27.329 INFO:teuthology.orchestra.run.smithi112.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [13.3 kB] 2024-09-11T06:01:27.330 INFO:teuthology.orchestra.run.smithi112.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [37.7 kB] 2024-09-11T06:01:27.330 INFO:teuthology.orchestra.run.smithi112.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [2,327 kB] 2024-09-11T06:01:27.343 INFO:teuthology.orchestra.run.smithi006.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [400 kB] 2024-09-11T06:01:27.346 INFO:teuthology.orchestra.run.smithi006.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [584 B] 2024-09-11T06:01:27.346 INFO:teuthology.orchestra.run.smithi006.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [626 kB] 2024-09-11T06:01:27.351 INFO:teuthology.orchestra.run.smithi006.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [901 kB] 2024-09-11T06:01:27.357 INFO:teuthology.orchestra.run.smithi006.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [176 kB] 2024-09-11T06:01:27.359 INFO:teuthology.orchestra.run.smithi112.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [400 kB] 2024-09-11T06:01:27.359 INFO:teuthology.orchestra.run.smithi006.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [19.2 kB] 2024-09-11T06:01:27.359 INFO:teuthology.orchestra.run.smithi006.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-09-11T06:01:27.360 INFO:teuthology.orchestra.run.smithi006.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-09-11T06:01:27.362 INFO:teuthology.orchestra.run.smithi112.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [584 B] 2024-09-11T06:01:27.363 INFO:teuthology.orchestra.run.smithi112.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [626 kB] 2024-09-11T06:01:27.368 INFO:teuthology.orchestra.run.smithi112.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [901 kB] 2024-09-11T06:01:27.368 INFO:teuthology.orchestra.run.smithi006.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-09-11T06:01:27.373 INFO:teuthology.orchestra.run.smithi006.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [228 B] 2024-09-11T06:01:27.375 INFO:teuthology.orchestra.run.smithi112.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [176 kB] 2024-09-11T06:01:27.377 INFO:teuthology.orchestra.run.smithi112.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [19.2 kB] 2024-09-11T06:01:27.377 INFO:teuthology.orchestra.run.smithi112.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-09-11T06:01:27.377 INFO:teuthology.orchestra.run.smithi112.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-09-11T06:01:27.422 INFO:teuthology.orchestra.run.smithi112.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-09-11T06:01:27.422 INFO:teuthology.orchestra.run.smithi112.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [228 B] 2024-09-11T06:01:27.539 INFO:teuthology.orchestra.run.smithi112.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2024-09-11T06:01:27.547 INFO:teuthology.orchestra.run.smithi006.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2024-09-11T06:01:27.677 INFO:teuthology.orchestra.run.smithi112.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,988 kB] 2024-09-11T06:01:27.685 INFO:teuthology.orchestra.run.smithi006.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,988 kB] 2024-09-11T06:01:27.992 INFO:teuthology.orchestra.run.smithi112.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [690 kB] 2024-09-11T06:01:28.000 INFO:teuthology.orchestra.run.smithi006.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [690 kB] 2024-09-11T06:01:28.027 INFO:teuthology.orchestra.run.smithi112.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [349 kB] 2024-09-11T06:01:28.036 INFO:teuthology.orchestra.run.smithi006.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [349 kB] 2024-09-11T06:01:28.042 INFO:teuthology.orchestra.run.smithi112.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [17.8 kB] 2024-09-11T06:01:28.043 INFO:teuthology.orchestra.run.smithi112.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [2,386 kB] 2024-09-11T06:01:28.051 INFO:teuthology.orchestra.run.smithi006.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [17.8 kB] 2024-09-11T06:01:28.051 INFO:teuthology.orchestra.run.smithi006.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [2,386 kB] 2024-09-11T06:01:28.109 INFO:teuthology.orchestra.run.smithi112.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [39.3 kB] 2024-09-11T06:01:28.112 INFO:teuthology.orchestra.run.smithi112.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [410 kB] 2024-09-11T06:01:28.118 INFO:teuthology.orchestra.run.smithi006.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [39.3 kB] 2024-09-11T06:01:28.121 INFO:teuthology.orchestra.run.smithi006.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [410 kB] 2024-09-11T06:01:28.128 INFO:teuthology.orchestra.run.smithi112.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [616 B] 2024-09-11T06:01:28.128 INFO:teuthology.orchestra.run.smithi112.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,123 kB] 2024-09-11T06:01:28.136 INFO:teuthology.orchestra.run.smithi006.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [616 B] 2024-09-11T06:01:28.136 INFO:teuthology.orchestra.run.smithi006.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,123 kB] 2024-09-11T06:01:28.160 INFO:teuthology.orchestra.run.smithi112.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [731 kB] 2024-09-11T06:01:28.170 INFO:teuthology.orchestra.run.smithi006.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [731 kB] 2024-09-11T06:01:28.180 INFO:teuthology.orchestra.run.smithi112.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [261 kB] 2024-09-11T06:01:28.185 INFO:teuthology.orchestra.run.smithi112.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [26.2 kB] 2024-09-11T06:01:28.186 INFO:teuthology.orchestra.run.smithi112.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [43.3 kB] 2024-09-11T06:01:28.187 INFO:teuthology.orchestra.run.smithi112.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,752 B] 2024-09-11T06:01:28.189 INFO:teuthology.orchestra.run.smithi006.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [261 kB] 2024-09-11T06:01:28.194 INFO:teuthology.orchestra.run.smithi006.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [26.2 kB] 2024-09-11T06:01:28.195 INFO:teuthology.orchestra.run.smithi006.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [43.3 kB] 2024-09-11T06:01:28.196 INFO:teuthology.orchestra.run.smithi006.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,752 B] 2024-09-11T06:01:28.238 INFO:teuthology.orchestra.run.smithi112.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.8 kB] 2024-09-11T06:01:28.238 INFO:teuthology.orchestra.run.smithi112.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [444 B] 2024-09-11T06:01:28.238 INFO:teuthology.orchestra.run.smithi112.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.9 kB] 2024-09-11T06:01:28.240 INFO:teuthology.orchestra.run.smithi112.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.8 kB] 2024-09-11T06:01:28.245 INFO:teuthology.orchestra.run.smithi006.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.8 kB] 2024-09-11T06:01:28.245 INFO:teuthology.orchestra.run.smithi006.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [444 B] 2024-09-11T06:01:28.245 INFO:teuthology.orchestra.run.smithi006.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.8 kB] 2024-09-11T06:01:28.247 INFO:teuthology.orchestra.run.smithi006.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.9 kB] 2024-09-11T06:01:28.270 INFO:teuthology.orchestra.run.smithi112.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.1 kB] 2024-09-11T06:01:28.270 INFO:teuthology.orchestra.run.smithi112.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-09-11T06:01:28.270 INFO:teuthology.orchestra.run.smithi112.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.3 kB] 2024-09-11T06:01:28.271 INFO:teuthology.orchestra.run.smithi112.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.8 kB] 2024-09-11T06:01:28.272 INFO:teuthology.orchestra.run.smithi112.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB] 2024-09-11T06:01:28.278 INFO:teuthology.orchestra.run.smithi006.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.1 kB] 2024-09-11T06:01:28.279 INFO:teuthology.orchestra.run.smithi006.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-09-11T06:01:28.279 INFO:teuthology.orchestra.run.smithi006.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.3 kB] 2024-09-11T06:01:28.280 INFO:teuthology.orchestra.run.smithi006.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.8 kB] 2024-09-11T06:01:28.281 INFO:teuthology.orchestra.run.smithi006.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB] 2024-09-11T06:01:28.319 INFO:teuthology.orchestra.run.smithi112.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [672 B] 2024-09-11T06:01:28.326 INFO:teuthology.orchestra.run.smithi006.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [672 B] 2024-09-11T06:01:33.833 INFO:teuthology.orchestra.run.smithi112.stdout:Fetched 15.8 MB in 2s (7,508 kB/s) 2024-09-11T06:01:33.910 INFO:teuthology.orchestra.run.smithi006.stdout:Fetched 15.8 MB in 2s (7,131 kB/s) 2024-09-11T06:01:35.278 INFO:teuthology.orchestra.run.smithi006.stdout:Reading package lists... 2024-09-11T06:01:35.293 INFO:teuthology.orchestra.run.smithi112.stdout:Reading package lists... 2024-09-11T06:01:35.297 DEBUG:teuthology.orchestra.run.smithi006:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-11T06:01:35.312 DEBUG:teuthology.orchestra.run.smithi112:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-11T06:01:35.360 INFO:teuthology.orchestra.run.smithi006.stdout:Reading package lists... 2024-09-11T06:01:35.376 INFO:teuthology.orchestra.run.smithi112.stdout:Reading package lists... 2024-09-11T06:01:35.573 INFO:teuthology.orchestra.run.smithi006.stdout:Building dependency tree... 2024-09-11T06:01:35.573 INFO:teuthology.orchestra.run.smithi006.stdout:Reading state information... 2024-09-11T06:01:35.584 INFO:teuthology.orchestra.run.smithi112.stdout:Building dependency tree... 2024-09-11T06:01:35.585 INFO:teuthology.orchestra.run.smithi112.stdout:Reading state information... 2024-09-11T06:01:35.765 INFO:teuthology.orchestra.run.smithi112.stdout:The following packages were automatically installed and are no longer required: 2024-09-11T06:01:35.765 INFO:teuthology.orchestra.run.smithi112.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-11T06:01:35.765 INFO:teuthology.orchestra.run.smithi112.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-11T06:01:35.766 INFO:teuthology.orchestra.run.smithi112.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-11T06:01:35.766 INFO:teuthology.orchestra.run.smithi112.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-09-11T06:01:35.766 INFO:teuthology.orchestra.run.smithi112.stdout: python2.7-minimal 2024-09-11T06:01:35.766 INFO:teuthology.orchestra.run.smithi112.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-11T06:01:35.767 INFO:teuthology.orchestra.run.smithi006.stdout:The following packages were automatically installed and are no longer required: 2024-09-11T06:01:35.767 INFO:teuthology.orchestra.run.smithi006.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-11T06:01:35.767 INFO:teuthology.orchestra.run.smithi006.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-11T06:01:35.768 INFO:teuthology.orchestra.run.smithi112.stdout:The following additional packages will be installed: 2024-09-11T06:01:35.768 INFO:teuthology.orchestra.run.smithi112.stdout: linux-generic linux-headers-5.15.0-119 linux-headers-5.15.0-119-generic 2024-09-11T06:01:35.768 INFO:teuthology.orchestra.run.smithi112.stdout: linux-headers-generic linux-image-5.15.0-119-generic 2024-09-11T06:01:35.768 INFO:teuthology.orchestra.run.smithi112.stdout: linux-modules-5.15.0-119-generic linux-modules-extra-5.15.0-119-generic 2024-09-11T06:01:35.769 INFO:teuthology.orchestra.run.smithi006.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-11T06:01:35.769 INFO:teuthology.orchestra.run.smithi006.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-09-11T06:01:35.769 INFO:teuthology.orchestra.run.smithi006.stdout: python2.7-minimal 2024-09-11T06:01:35.769 INFO:teuthology.orchestra.run.smithi006.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-11T06:01:35.769 INFO:teuthology.orchestra.run.smithi112.stdout:Suggested packages: 2024-09-11T06:01:35.769 INFO:teuthology.orchestra.run.smithi112.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-09-11T06:01:35.769 INFO:teuthology.orchestra.run.smithi112.stdout:Recommended packages: 2024-09-11T06:01:35.770 INFO:teuthology.orchestra.run.smithi112.stdout: thermald 2024-09-11T06:01:35.770 INFO:teuthology.orchestra.run.smithi006.stdout:The following additional packages will be installed: 2024-09-11T06:01:35.770 INFO:teuthology.orchestra.run.smithi006.stdout: linux-generic linux-headers-5.15.0-119 linux-headers-5.15.0-119-generic 2024-09-11T06:01:35.770 INFO:teuthology.orchestra.run.smithi006.stdout: linux-headers-generic linux-image-5.15.0-119-generic 2024-09-11T06:01:35.771 INFO:teuthology.orchestra.run.smithi006.stdout: linux-modules-5.15.0-119-generic linux-modules-extra-5.15.0-119-generic 2024-09-11T06:01:35.772 INFO:teuthology.orchestra.run.smithi006.stdout:Suggested packages: 2024-09-11T06:01:35.772 INFO:teuthology.orchestra.run.smithi006.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-09-11T06:01:35.772 INFO:teuthology.orchestra.run.smithi006.stdout:Recommended packages: 2024-09-11T06:01:35.772 INFO:teuthology.orchestra.run.smithi006.stdout: thermald 2024-09-11T06:01:35.822 INFO:teuthology.orchestra.run.smithi006.stdout:The following NEW packages will be installed: 2024-09-11T06:01:35.822 INFO:teuthology.orchestra.run.smithi006.stdout: linux-headers-5.15.0-119 linux-headers-5.15.0-119-generic 2024-09-11T06:01:35.822 INFO:teuthology.orchestra.run.smithi006.stdout: linux-image-5.15.0-119-generic linux-modules-5.15.0-119-generic 2024-09-11T06:01:35.822 INFO:teuthology.orchestra.run.smithi006.stdout: linux-modules-extra-5.15.0-119-generic 2024-09-11T06:01:35.822 INFO:teuthology.orchestra.run.smithi112.stdout:The following NEW packages will be installed: 2024-09-11T06:01:35.822 INFO:teuthology.orchestra.run.smithi112.stdout: linux-headers-5.15.0-119 linux-headers-5.15.0-119-generic 2024-09-11T06:01:35.823 INFO:teuthology.orchestra.run.smithi112.stdout: linux-image-5.15.0-119-generic linux-modules-5.15.0-119-generic 2024-09-11T06:01:35.823 INFO:teuthology.orchestra.run.smithi112.stdout: linux-modules-extra-5.15.0-119-generic 2024-09-11T06:01:35.823 INFO:teuthology.orchestra.run.smithi112.stdout:The following packages will be upgraded: 2024-09-11T06:01:35.824 INFO:teuthology.orchestra.run.smithi006.stdout:The following packages will be upgraded: 2024-09-11T06:01:35.824 INFO:teuthology.orchestra.run.smithi112.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-11T06:01:35.824 INFO:teuthology.orchestra.run.smithi006.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-11T06:01:35.878 INFO:teuthology.orchestra.run.smithi006.stdout:3 upgraded, 5 newly installed, 0 to remove and 344 not upgraded. 2024-09-11T06:01:35.878 INFO:teuthology.orchestra.run.smithi006.stdout:Need to get 113 MB of archives. 2024-09-11T06:01:35.878 INFO:teuthology.orchestra.run.smithi006.stdout:After this operation, 583 MB of additional disk space will be used. 2024-09-11T06:01:35.878 INFO:teuthology.orchestra.run.smithi006.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-119-generic amd64 5.15.0-119.129 [22.6 MB] 2024-09-11T06:01:36.003 INFO:teuthology.orchestra.run.smithi112.stdout:3 upgraded, 5 newly installed, 0 to remove and 344 not upgraded. 2024-09-11T06:01:36.004 INFO:teuthology.orchestra.run.smithi112.stdout:Need to get 113 MB of archives. 2024-09-11T06:01:36.004 INFO:teuthology.orchestra.run.smithi112.stdout:After this operation, 583 MB of additional disk space will be used. 2024-09-11T06:01:36.004 INFO:teuthology.orchestra.run.smithi112.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-119-generic amd64 5.15.0-119.129 [22.6 MB] 2024-09-11T06:01:36.322 INFO:teuthology.orchestra.run.smithi006.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-119-generic amd64 5.15.0-119.129 [11.5 MB] 2024-09-11T06:01:36.431 INFO:teuthology.orchestra.run.smithi006.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-119-generic amd64 5.15.0-119.129 [63.9 MB] 2024-09-11T06:01:37.400 INFO:teuthology.orchestra.run.smithi006.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.119.119 [1,694 B] 2024-09-11T06:01:37.400 INFO:teuthology.orchestra.run.smithi006.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.119.119 [2,502 B] 2024-09-11T06:01:37.400 INFO:teuthology.orchestra.run.smithi006.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-119 all 5.15.0-119.129 [12.3 MB] 2024-09-11T06:01:37.501 INFO:teuthology.orchestra.run.smithi112.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-119-generic amd64 5.15.0-119.129 [11.5 MB] 2024-09-11T06:01:37.531 INFO:teuthology.orchestra.run.smithi006.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-119-generic amd64 5.15.0-119.129 [2,879 kB] 2024-09-11T06:01:37.555 INFO:teuthology.orchestra.run.smithi006.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.119.119 [2,350 B] 2024-09-11T06:01:37.850 INFO:teuthology.orchestra.run.smithi112.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-119-generic amd64 5.15.0-119.129 [63.9 MB] 2024-09-11T06:01:37.954 INFO:teuthology.orchestra.run.smithi006.stdout:Fetched 113 MB in 2s (65.8 MB/s) 2024-09-11T06:01:38.129 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package linux-modules-5.15.0-119-generic. 2024-09-11T06:01:40.028 INFO:teuthology.orchestra.run.smithi112.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.119.119 [1,694 B] 2024-09-11T06:01:40.029 INFO:teuthology.orchestra.run.smithi112.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.119.119 [2,502 B] 2024-09-11T06:01:40.029 INFO:teuthology.orchestra.run.smithi112.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-119 all 5.15.0-119.129 [12.3 MB] 2024-09-11T06:01:40.291 INFO:teuthology.orchestra.run.smithi006.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-09-11T06:01:40.295 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../0-linux-modules-5.15.0-119-generic_5.15.0-119.129_amd64.deb ... 2024-09-11T06:01:40.380 INFO:teuthology.orchestra.run.smithi112.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-119-generic amd64 5.15.0-119.129 [2,879 kB] 2024-09-11T06:01:40.463 INFO:teuthology.orchestra.run.smithi112.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.119.119 [2,350 B] 2024-09-11T06:01:40.551 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking linux-modules-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T06:01:40.855 INFO:teuthology.orchestra.run.smithi112.stdout:Fetched 113 MB in 5s (24.5 MB/s) 2024-09-11T06:01:41.015 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package linux-modules-5.15.0-119-generic. 2024-09-11T06:01:42.853 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package linux-image-5.15.0-119-generic. 2024-09-11T06:01:42.875 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../1-linux-image-5.15.0-119-generic_5.15.0-119.129_amd64.deb ... 2024-09-11T06:01:42.967 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking linux-image-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T06:01:42.986 INFO:teuthology.orchestra.run.smithi112.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-09-11T06:01:42.990 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../0-linux-modules-5.15.0-119-generic_5.15.0-119.129_amd64.deb ... 2024-09-11T06:01:43.129 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking linux-modules-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T06:01:43.448 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-119-generic. 2024-09-11T06:01:43.469 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-119-generic_5.15.0-119.129_amd64.deb ... 2024-09-11T06:01:43.506 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking linux-modules-extra-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T06:01:45.264 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package linux-image-5.15.0-119-generic. 2024-09-11T06:01:45.285 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../1-linux-image-5.15.0-119-generic_5.15.0-119.129_amd64.deb ... 2024-09-11T06:01:45.371 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking linux-image-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T06:01:45.784 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-119-generic. 2024-09-11T06:01:45.804 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-119-generic_5.15.0-119.129_amd64.deb ... 2024-09-11T06:01:45.834 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking linux-modules-extra-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T06:01:49.941 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../3-linux-generic_5.15.0.119.119_amd64.deb ... 2024-09-11T06:01:50.042 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking linux-generic (5.15.0.119.119) over (5.15.0.56.54) ... 2024-09-11T06:01:50.367 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.119.119_amd64.deb ... 2024-09-11T06:01:50.504 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking linux-image-generic (5.15.0.119.119) over (5.15.0.56.54) ... 2024-09-11T06:01:50.816 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package linux-headers-5.15.0-119. 2024-09-11T06:01:50.835 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../5-linux-headers-5.15.0-119_5.15.0-119.129_all.deb ... 2024-09-11T06:01:50.874 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking linux-headers-5.15.0-119 (5.15.0-119.129) ... 2024-09-11T06:01:52.510 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../3-linux-generic_5.15.0.119.119_amd64.deb ... 2024-09-11T06:01:52.638 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking linux-generic (5.15.0.119.119) over (5.15.0.56.54) ... 2024-09-11T06:01:52.962 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.119.119_amd64.deb ... 2024-09-11T06:01:53.107 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking linux-image-generic (5.15.0.119.119) over (5.15.0.56.54) ... 2024-09-11T06:01:53.402 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package linux-headers-5.15.0-119. 2024-09-11T06:01:53.424 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../5-linux-headers-5.15.0-119_5.15.0-119.129_all.deb ... 2024-09-11T06:01:53.468 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking linux-headers-5.15.0-119 (5.15.0-119.129) ... 2024-09-11T06:01:56.429 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package linux-headers-5.15.0-119-generic. 2024-09-11T06:01:56.467 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../6-linux-headers-5.15.0-119-generic_5.15.0-119.129_amd64.deb ... 2024-09-11T06:01:56.504 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking linux-headers-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T06:01:59.108 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package linux-headers-5.15.0-119-generic. 2024-09-11T06:01:59.145 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../6-linux-headers-5.15.0-119-generic_5.15.0-119.129_amd64.deb ... 2024-09-11T06:01:59.183 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking linux-headers-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T06:01:59.219 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.119.119_amd64.deb ... 2024-09-11T06:01:59.341 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking linux-headers-generic (5.15.0.119.119) over (5.15.0.56.54) ... 2024-09-11T06:01:59.633 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up linux-headers-5.15.0-119 (5.15.0-119.129) ... 2024-09-11T06:01:59.754 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up linux-headers-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T06:01:59.889 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up linux-headers-generic (5.15.0.119.119) ... 2024-09-11T06:02:00.018 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up linux-image-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T06:02:01.418 INFO:teuthology.orchestra.run.smithi006.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-09-11T06:02:01.418 INFO:teuthology.orchestra.run.smithi006.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-09-11T06:02:01.418 INFO:teuthology.orchestra.run.smithi006.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-119-generic 2024-09-11T06:02:01.418 INFO:teuthology.orchestra.run.smithi006.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-119-generic 2024-09-11T06:02:01.569 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up linux-modules-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T06:02:01.795 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.119.119_amd64.deb ... 2024-09-11T06:02:01.920 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking linux-headers-generic (5.15.0.119.119) over (5.15.0.56.54) ... 2024-09-11T06:02:02.245 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up linux-headers-5.15.0-119 (5.15.0-119.129) ... 2024-09-11T06:02:02.366 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up linux-headers-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T06:02:03.227 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up linux-headers-generic (5.15.0.119.119) ... 2024-09-11T06:02:03.313 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up linux-modules-extra-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T06:02:03.489 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up linux-image-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T06:02:04.731 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up linux-image-generic (5.15.0.119.119) ... 2024-09-11T06:02:04.790 INFO:teuthology.orchestra.run.smithi112.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-09-11T06:02:04.790 INFO:teuthology.orchestra.run.smithi112.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-09-11T06:02:04.790 INFO:teuthology.orchestra.run.smithi112.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-119-generic 2024-09-11T06:02:04.790 INFO:teuthology.orchestra.run.smithi112.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-119-generic 2024-09-11T06:02:04.857 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up linux-generic (5.15.0.119.119) ... 2024-09-11T06:02:04.947 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up linux-modules-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T06:02:04.983 INFO:teuthology.orchestra.run.smithi006.stdout:Processing triggers for linux-image-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T06:02:05.062 INFO:teuthology.orchestra.run.smithi006.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-11T06:02:05.063 INFO:teuthology.orchestra.run.smithi006.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-119-generic 2024-09-11T06:02:06.365 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up linux-modules-extra-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T06:02:07.741 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up linux-image-generic (5.15.0.119.119) ... 2024-09-11T06:02:07.867 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up linux-generic (5.15.0.119.119) ... 2024-09-11T06:02:07.993 INFO:teuthology.orchestra.run.smithi112.stdout:Processing triggers for linux-image-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T06:02:08.059 INFO:teuthology.orchestra.run.smithi112.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-11T06:02:08.059 INFO:teuthology.orchestra.run.smithi112.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-119-generic 2024-09-11T06:02:19.826 INFO:teuthology.orchestra.run.smithi006.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-11T06:02:19.826 INFO:teuthology.orchestra.run.smithi006.stdout:Sourcing file `/etc/default/grub' 2024-09-11T06:02:19.830 INFO:teuthology.orchestra.run.smithi006.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-11T06:02:19.860 INFO:teuthology.orchestra.run.smithi006.stdout:Generating grub configuration file ... 2024-09-11T06:02:20.215 INFO:teuthology.orchestra.run.smithi006.stdout:Found linux image: /boot/vmlinuz-5.15.0-119-generic 2024-09-11T06:02:20.252 INFO:teuthology.orchestra.run.smithi006.stdout:Found initrd image: /boot/initrd.img-5.15.0-119-generic 2024-09-11T06:02:20.617 INFO:teuthology.orchestra.run.smithi006.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-11T06:02:20.623 INFO:teuthology.orchestra.run.smithi006.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-11T06:02:20.698 INFO:teuthology.orchestra.run.smithi006.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-11T06:02:20.704 INFO:teuthology.orchestra.run.smithi006.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-11T06:02:20.880 INFO:teuthology.orchestra.run.smithi006.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-11T06:02:20.880 INFO:teuthology.orchestra.run.smithi006.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-09-11T06:02:20.881 INFO:teuthology.orchestra.run.smithi006.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-11T06:02:20.917 INFO:teuthology.orchestra.run.smithi006.stdout:done 2024-09-11T06:02:21.374 DEBUG:teuthology.orchestra.run.smithi006:> dpkg -s linux-image-generic 2024-09-11T06:02:21.395 INFO:teuthology.orchestra.run.smithi006.stdout:Package: linux-image-generic 2024-09-11T06:02:21.395 INFO:teuthology.orchestra.run.smithi006.stdout:Status: install ok installed 2024-09-11T06:02:21.395 INFO:teuthology.orchestra.run.smithi006.stdout:Priority: optional 2024-09-11T06:02:21.395 INFO:teuthology.orchestra.run.smithi006.stdout:Section: kernel 2024-09-11T06:02:21.395 INFO:teuthology.orchestra.run.smithi006.stdout:Installed-Size: 21 2024-09-11T06:02:21.395 INFO:teuthology.orchestra.run.smithi006.stdout:Maintainer: Ubuntu Kernel Team 2024-09-11T06:02:21.395 INFO:teuthology.orchestra.run.smithi006.stdout:Architecture: amd64 2024-09-11T06:02:21.395 INFO:teuthology.orchestra.run.smithi006.stdout:Source: linux-meta 2024-09-11T06:02:21.395 INFO:teuthology.orchestra.run.smithi006.stdout:Version: 5.15.0.119.119 2024-09-11T06:02:21.395 INFO:teuthology.orchestra.run.smithi006.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-119), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-11T06:02:21.395 INFO:teuthology.orchestra.run.smithi006.stdout:Depends: linux-image-5.15.0-119-generic, linux-modules-extra-5.15.0-119-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-11T06:02:21.396 INFO:teuthology.orchestra.run.smithi006.stdout:Recommends: thermald 2024-09-11T06:02:21.396 INFO:teuthology.orchestra.run.smithi006.stdout:Description: Generic Linux kernel image 2024-09-11T06:02:21.396 INFO:teuthology.orchestra.run.smithi006.stdout: This package will always depend on the latest generic kernel image 2024-09-11T06:02:21.396 INFO:teuthology.orchestra.run.smithi006.stdout: available. 2024-09-11T06:02:21.396 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-119-generic 2024-09-11T06:02:21.396 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-11T06:02:21.396 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-11T06:02:21.396 INFO:teuthology.task.kernel:Installing distro kernel on cluster1.mon.a... 2024-09-11T06:02:21.396 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi006.front.sepia.ceph.com, path=None, version=distro) 2024-09-11T06:02:21.397 DEBUG:teuthology.orchestra.run.smithi006:> sudo apt-get clean 2024-09-11T06:02:21.582 DEBUG:teuthology.orchestra.run.smithi006:> sudo apt-get update 2024-09-11T06:02:21.698 INFO:teuthology.orchestra.run.smithi006.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-11T06:02:21.708 INFO:teuthology.orchestra.run.smithi006.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-11T06:02:21.725 INFO:teuthology.orchestra.run.smithi006.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-11T06:02:21.820 INFO:teuthology.orchestra.run.smithi006.stdout:Hit:4 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-11T06:02:22.265 INFO:teuthology.orchestra.run.smithi112.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-11T06:02:22.265 INFO:teuthology.orchestra.run.smithi112.stdout:Sourcing file `/etc/default/grub' 2024-09-11T06:02:22.282 INFO:teuthology.orchestra.run.smithi112.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-11T06:02:22.302 INFO:teuthology.orchestra.run.smithi112.stdout:Generating grub configuration file ... 2024-09-11T06:02:22.538 INFO:teuthology.orchestra.run.smithi112.stdout:Found linux image: /boot/vmlinuz-5.15.0-119-generic 2024-09-11T06:02:22.558 INFO:teuthology.orchestra.run.smithi112.stdout:Found initrd image: /boot/initrd.img-5.15.0-119-generic 2024-09-11T06:02:22.837 INFO:teuthology.orchestra.run.smithi112.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-11T06:02:22.842 INFO:teuthology.orchestra.run.smithi112.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-11T06:02:22.903 INFO:teuthology.orchestra.run.smithi112.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-11T06:02:22.909 INFO:teuthology.orchestra.run.smithi112.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-11T06:02:23.058 INFO:teuthology.orchestra.run.smithi112.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-11T06:02:23.059 INFO:teuthology.orchestra.run.smithi112.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-09-11T06:02:23.059 INFO:teuthology.orchestra.run.smithi112.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-11T06:02:23.079 INFO:teuthology.orchestra.run.smithi006.stdout:Reading package lists... 2024-09-11T06:02:23.099 DEBUG:teuthology.orchestra.run.smithi006:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-11T06:02:23.116 INFO:teuthology.orchestra.run.smithi112.stdout:done 2024-09-11T06:02:23.164 INFO:teuthology.orchestra.run.smithi006.stdout:Reading package lists... 2024-09-11T06:02:23.379 INFO:teuthology.orchestra.run.smithi006.stdout:Building dependency tree... 2024-09-11T06:02:23.379 INFO:teuthology.orchestra.run.smithi006.stdout:Reading state information... 2024-09-11T06:02:23.511 DEBUG:teuthology.orchestra.run.smithi112:> dpkg -s linux-image-generic 2024-09-11T06:02:23.536 INFO:teuthology.orchestra.run.smithi112.stdout:Package: linux-image-generic 2024-09-11T06:02:23.536 INFO:teuthology.orchestra.run.smithi112.stdout:Status: install ok installed 2024-09-11T06:02:23.536 INFO:teuthology.orchestra.run.smithi112.stdout:Priority: optional 2024-09-11T06:02:23.536 INFO:teuthology.orchestra.run.smithi112.stdout:Section: kernel 2024-09-11T06:02:23.536 INFO:teuthology.orchestra.run.smithi112.stdout:Installed-Size: 21 2024-09-11T06:02:23.536 INFO:teuthology.orchestra.run.smithi112.stdout:Maintainer: Ubuntu Kernel Team 2024-09-11T06:02:23.536 INFO:teuthology.orchestra.run.smithi112.stdout:Architecture: amd64 2024-09-11T06:02:23.537 INFO:teuthology.orchestra.run.smithi112.stdout:Source: linux-meta 2024-09-11T06:02:23.537 INFO:teuthology.orchestra.run.smithi112.stdout:Version: 5.15.0.119.119 2024-09-11T06:02:23.537 INFO:teuthology.orchestra.run.smithi112.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-119), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-11T06:02:23.537 INFO:teuthology.orchestra.run.smithi112.stdout:Depends: linux-image-5.15.0-119-generic, linux-modules-extra-5.15.0-119-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-11T06:02:23.537 INFO:teuthology.orchestra.run.smithi112.stdout:Recommends: thermald 2024-09-11T06:02:23.537 INFO:teuthology.orchestra.run.smithi112.stdout:Description: Generic Linux kernel image 2024-09-11T06:02:23.537 INFO:teuthology.orchestra.run.smithi112.stdout: This package will always depend on the latest generic kernel image 2024-09-11T06:02:23.537 INFO:teuthology.orchestra.run.smithi112.stdout: available. 2024-09-11T06:02:23.537 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-119-generic 2024-09-11T06:02:23.537 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-11T06:02:23.538 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-11T06:02:23.538 INFO:teuthology.task.kernel:Installing distro kernel on cluster2.mon.a... 2024-09-11T06:02:23.538 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi112.front.sepia.ceph.com, path=None, version=distro) 2024-09-11T06:02:23.538 DEBUG:teuthology.orchestra.run.smithi112:> sudo apt-get clean 2024-09-11T06:02:23.562 INFO:teuthology.orchestra.run.smithi006.stdout:linux-image-generic is already the newest version (5.15.0.119.119). 2024-09-11T06:02:23.563 INFO:teuthology.orchestra.run.smithi006.stdout:The following packages were automatically installed and are no longer required: 2024-09-11T06:02:23.563 INFO:teuthology.orchestra.run.smithi006.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-11T06:02:23.563 INFO:teuthology.orchestra.run.smithi006.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-11T06:02:23.563 INFO:teuthology.orchestra.run.smithi006.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-11T06:02:23.563 INFO:teuthology.orchestra.run.smithi006.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-11T06:02:23.564 INFO:teuthology.orchestra.run.smithi006.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-11T06:02:23.564 INFO:teuthology.orchestra.run.smithi006.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-11T06:02:23.605 INFO:teuthology.orchestra.run.smithi006.stdout:0 upgraded, 0 newly installed, 0 to remove and 344 not upgraded. 2024-09-11T06:02:23.606 DEBUG:teuthology.orchestra.run.smithi006:> dpkg -s linux-image-generic 2024-09-11T06:02:23.622 INFO:teuthology.orchestra.run.smithi006.stdout:Package: linux-image-generic 2024-09-11T06:02:23.622 INFO:teuthology.orchestra.run.smithi006.stdout:Status: install ok installed 2024-09-11T06:02:23.622 INFO:teuthology.orchestra.run.smithi006.stdout:Priority: optional 2024-09-11T06:02:23.622 INFO:teuthology.orchestra.run.smithi006.stdout:Section: kernel 2024-09-11T06:02:23.622 INFO:teuthology.orchestra.run.smithi006.stdout:Installed-Size: 21 2024-09-11T06:02:23.622 INFO:teuthology.orchestra.run.smithi006.stdout:Maintainer: Ubuntu Kernel Team 2024-09-11T06:02:23.623 INFO:teuthology.orchestra.run.smithi006.stdout:Architecture: amd64 2024-09-11T06:02:23.623 INFO:teuthology.orchestra.run.smithi006.stdout:Source: linux-meta 2024-09-11T06:02:23.623 INFO:teuthology.orchestra.run.smithi006.stdout:Version: 5.15.0.119.119 2024-09-11T06:02:23.623 INFO:teuthology.orchestra.run.smithi006.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-119), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-11T06:02:23.623 INFO:teuthology.orchestra.run.smithi006.stdout:Depends: linux-image-5.15.0-119-generic, linux-modules-extra-5.15.0-119-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-11T06:02:23.623 INFO:teuthology.orchestra.run.smithi006.stdout:Recommends: thermald 2024-09-11T06:02:23.623 INFO:teuthology.orchestra.run.smithi006.stdout:Description: Generic Linux kernel image 2024-09-11T06:02:23.624 INFO:teuthology.orchestra.run.smithi006.stdout: This package will always depend on the latest generic kernel image 2024-09-11T06:02:23.624 INFO:teuthology.orchestra.run.smithi006.stdout: available. 2024-09-11T06:02:23.624 DEBUG:teuthology.orchestra.run.smithi006:> mktemp 2024-09-11T06:02:23.671 INFO:teuthology.orchestra.run.smithi006.stdout:/tmp/tmp.YvXgszMoPB 2024-09-11T06:02:23.671 DEBUG:teuthology.orchestra.run.smithi006:> sudo cp /boot/grub/grub.cfg /tmp/tmp.YvXgszMoPB 2024-09-11T06:02:23.732 DEBUG:teuthology.orchestra.run.smithi112:> sudo apt-get update 2024-09-11T06:02:23.743 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 0666 /tmp/tmp.YvXgszMoPB 2024-09-11T06:02:23.846 INFO:teuthology.orchestra.run.smithi112.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-11T06:02:23.853 DEBUG:teuthology.orchestra.remote:smithi006:/tmp/tmp.YvXgszMoPB is 10KB 2024-09-11T06:02:23.866 DEBUG:teuthology.orchestra.run.smithi006:> rm -fr /tmp/tmp.YvXgszMoPB 2024-09-11T06:02:23.871 DEBUG:teuthology.orchestra.run.smithi006:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-11T06:02:23.927 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-09-11T06:02:23.927 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-11T06:02:23.928 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-11T06:02:23.970 INFO:teuthology.orchestra.run.smithi112.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-11T06:02:23.995 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-119-generic 2024-09-11T06:02:23.995 DEBUG:teuthology.orchestra.run.smithi006:> sudo update-grub 2024-09-11T06:02:24.059 INFO:teuthology.orchestra.run.smithi112.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-11T06:02:24.133 INFO:teuthology.orchestra.run.smithi112.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-11T06:02:25.307 INFO:teuthology.orchestra.run.smithi006.stderr:Sourcing file `/etc/default/grub' 2024-09-11T06:02:25.308 INFO:teuthology.orchestra.run.smithi006.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-11T06:02:25.312 INFO:teuthology.orchestra.run.smithi112.stdout:Reading package lists... 2024-09-11T06:02:25.317 INFO:teuthology.orchestra.run.smithi006.stderr:Generating grub configuration file ... 2024-09-11T06:02:25.331 DEBUG:teuthology.orchestra.run.smithi112:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-11T06:02:25.393 INFO:teuthology.orchestra.run.smithi112.stdout:Reading package lists... 2024-09-11T06:02:25.489 INFO:teuthology.orchestra.run.smithi006.stderr:Found linux image: /boot/vmlinuz-5.15.0-119-generic 2024-09-11T06:02:25.495 INFO:teuthology.orchestra.run.smithi006.stderr:Found initrd image: /boot/initrd.img-5.15.0-119-generic 2024-09-11T06:02:25.595 INFO:teuthology.orchestra.run.smithi112.stdout:Building dependency tree... 2024-09-11T06:02:25.596 INFO:teuthology.orchestra.run.smithi112.stdout:Reading state information... 2024-09-11T06:02:25.760 INFO:teuthology.orchestra.run.smithi112.stdout:linux-image-generic is already the newest version (5.15.0.119.119). 2024-09-11T06:02:25.760 INFO:teuthology.orchestra.run.smithi112.stdout:The following packages were automatically installed and are no longer required: 2024-09-11T06:02:25.761 INFO:teuthology.orchestra.run.smithi112.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-11T06:02:25.761 INFO:teuthology.orchestra.run.smithi112.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-11T06:02:25.761 INFO:teuthology.orchestra.run.smithi112.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-11T06:02:25.761 INFO:teuthology.orchestra.run.smithi112.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-11T06:02:25.761 INFO:teuthology.orchestra.run.smithi112.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-11T06:02:25.761 INFO:teuthology.orchestra.run.smithi112.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-11T06:02:25.770 INFO:teuthology.orchestra.run.smithi006.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-11T06:02:25.776 INFO:teuthology.orchestra.run.smithi006.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-11T06:02:25.802 INFO:teuthology.orchestra.run.smithi112.stdout:0 upgraded, 0 newly installed, 0 to remove and 344 not upgraded. 2024-09-11T06:02:25.803 DEBUG:teuthology.orchestra.run.smithi112:> dpkg -s linux-image-generic 2024-09-11T06:02:25.819 INFO:teuthology.orchestra.run.smithi112.stdout:Package: linux-image-generic 2024-09-11T06:02:25.819 INFO:teuthology.orchestra.run.smithi112.stdout:Status: install ok installed 2024-09-11T06:02:25.820 INFO:teuthology.orchestra.run.smithi112.stdout:Priority: optional 2024-09-11T06:02:25.820 INFO:teuthology.orchestra.run.smithi112.stdout:Section: kernel 2024-09-11T06:02:25.820 INFO:teuthology.orchestra.run.smithi112.stdout:Installed-Size: 21 2024-09-11T06:02:25.820 INFO:teuthology.orchestra.run.smithi112.stdout:Maintainer: Ubuntu Kernel Team 2024-09-11T06:02:25.820 INFO:teuthology.orchestra.run.smithi112.stdout:Architecture: amd64 2024-09-11T06:02:25.820 INFO:teuthology.orchestra.run.smithi112.stdout:Source: linux-meta 2024-09-11T06:02:25.820 INFO:teuthology.orchestra.run.smithi112.stdout:Version: 5.15.0.119.119 2024-09-11T06:02:25.820 INFO:teuthology.orchestra.run.smithi112.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-119), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-11T06:02:25.820 INFO:teuthology.orchestra.run.smithi112.stdout:Depends: linux-image-5.15.0-119-generic, linux-modules-extra-5.15.0-119-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-11T06:02:25.820 INFO:teuthology.orchestra.run.smithi112.stdout:Recommends: thermald 2024-09-11T06:02:25.820 INFO:teuthology.orchestra.run.smithi112.stdout:Description: Generic Linux kernel image 2024-09-11T06:02:25.820 INFO:teuthology.orchestra.run.smithi112.stdout: This package will always depend on the latest generic kernel image 2024-09-11T06:02:25.821 INFO:teuthology.orchestra.run.smithi112.stdout: available. 2024-09-11T06:02:25.821 DEBUG:teuthology.orchestra.run.smithi112:> mktemp 2024-09-11T06:02:25.823 INFO:teuthology.orchestra.run.smithi006.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-11T06:02:25.829 INFO:teuthology.orchestra.run.smithi006.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-11T06:02:25.868 INFO:teuthology.orchestra.run.smithi112.stdout:/tmp/tmp.3zrcdkBhFv 2024-09-11T06:02:25.869 DEBUG:teuthology.orchestra.run.smithi112:> sudo cp /boot/grub/grub.cfg /tmp/tmp.3zrcdkBhFv 2024-09-11T06:02:25.924 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod 0666 /tmp/tmp.3zrcdkBhFv 2024-09-11T06:02:25.945 INFO:teuthology.orchestra.run.smithi006.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-11T06:02:25.945 INFO:teuthology.orchestra.run.smithi006.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-09-11T06:02:25.945 INFO:teuthology.orchestra.run.smithi006.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-11T06:02:25.956 INFO:teuthology.orchestra.run.smithi006.stderr:done 2024-09-11T06:02:25.975 DEBUG:teuthology.orchestra.run.smithi006:> sudo shutdown -r now 2024-09-11T06:02:26.040 DEBUG:teuthology.orchestra.remote:smithi112:/tmp/tmp.3zrcdkBhFv is 10KB 2024-09-11T06:02:26.052 DEBUG:teuthology.orchestra.run.smithi112:> rm -fr /tmp/tmp.3zrcdkBhFv 2024-09-11T06:02:26.057 DEBUG:teuthology.orchestra.run.smithi112:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-11T06:02:26.112 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-11T06:02:26.112 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-11T06:02:26.112 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-11T06:02:26.187 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-119-generic 2024-09-11T06:02:26.187 DEBUG:teuthology.orchestra.run.smithi112:> sudo update-grub 2024-09-11T06:02:27.376 INFO:teuthology.orchestra.run.smithi112.stderr:Sourcing file `/etc/default/grub' 2024-09-11T06:02:27.377 INFO:teuthology.orchestra.run.smithi112.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-11T06:02:27.378 INFO:teuthology.orchestra.run.smithi112.stderr:Generating grub configuration file ... 2024-09-11T06:02:27.572 INFO:teuthology.orchestra.run.smithi112.stderr:Found linux image: /boot/vmlinuz-5.15.0-119-generic 2024-09-11T06:02:27.578 INFO:teuthology.orchestra.run.smithi112.stderr:Found initrd image: /boot/initrd.img-5.15.0-119-generic 2024-09-11T06:02:27.815 INFO:teuthology.orchestra.run.smithi112.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-11T06:02:27.821 INFO:teuthology.orchestra.run.smithi112.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-11T06:02:27.861 INFO:teuthology.orchestra.run.smithi112.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-11T06:02:27.866 INFO:teuthology.orchestra.run.smithi112.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-11T06:02:27.999 INFO:teuthology.orchestra.run.smithi112.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-11T06:02:27.999 INFO:teuthology.orchestra.run.smithi112.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-09-11T06:02:28.000 INFO:teuthology.orchestra.run.smithi112.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-11T06:02:28.013 INFO:teuthology.orchestra.run.smithi112.stderr:done 2024-09-11T06:02:28.031 DEBUG:teuthology.orchestra.run.smithi112:> sudo shutdown -r now 2024-09-11T06:02:56.000 INFO:teuthology.task.kernel:Checking client cluster1.mon.a for new kernel version... 2024-09-11T06:02:56.001 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi006.front.sepia.ceph.com' 2024-09-11T06:02:56.002 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi006.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T06:02:58.035 INFO:teuthology.task.kernel:Checking client cluster2.mon.a for new kernel version... 2024-09-11T06:02:58.036 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi112.front.sepia.ceph.com' 2024-09-11T06:02:58.036 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T06:03:14.365 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.6 2024-09-11T06:03:16.413 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.112 2024-09-11T06:03:23.373 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi006.front.sepia.ceph.com' 2024-09-11T06:03:23.374 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi006.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T06:03:25.415 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi112.front.sepia.ceph.com' 2024-09-11T06:03:25.416 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T06:03:26.429 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.6 2024-09-11T06:03:28.477 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.112 2024-09-11T06:03:38.432 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi006.front.sepia.ceph.com' 2024-09-11T06:03:38.433 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi006.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T06:03:40.480 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi112.front.sepia.ceph.com' 2024-09-11T06:03:40.481 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T06:03:58.913 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.112 2024-09-11T06:04:12.989 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.6 2024-09-11T06:04:13.915 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi112.front.sepia.ceph.com' 2024-09-11T06:04:13.916 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T06:04:14.154 DEBUG:teuthology.orchestra.run.smithi112:> true 2024-09-11T06:04:15.220 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi112.front.sepia.ceph.com' 2024-09-11T06:04:15.220 INFO:teuthology.task.kernel:Checking kernel version of cluster2.mon.a, want "5.15.0-119-generic"... 2024-09-11T06:04:15.220 DEBUG:teuthology.orchestra.run.smithi112:> uname -r 2024-09-11T06:04:15.226 INFO:teuthology.orchestra.run.smithi112.stdout:5.15.0-119-generic 2024-09-11T06:04:15.226 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-119-generic vs 5.15.0-119-generic 2024-09-11T06:04:15.226 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-11T06:04:15.226 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-11T06:04:16.227 INFO:teuthology.task.kernel:Enabling kdb on cluster2.mon.a... 2024-09-11T06:04:16.227 DEBUG:teuthology.orchestra.run.smithi112:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-11T06:04:16.295 INFO:teuthology.orchestra.run.smithi112.stdout:ttyS1 2024-09-11T06:04:16.309 DEBUG:teuthology.parallel:result is None 2024-09-11T06:04:27.990 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi006.front.sepia.ceph.com' 2024-09-11T06:04:27.991 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi006.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T06:04:28.279 DEBUG:teuthology.orchestra.run.smithi006:> true 2024-09-11T06:04:28.885 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi006.front.sepia.ceph.com' 2024-09-11T06:04:28.885 INFO:teuthology.task.kernel:Checking kernel version of cluster1.mon.a, want "5.15.0-119-generic"... 2024-09-11T06:04:28.886 DEBUG:teuthology.orchestra.run.smithi006:> uname -r 2024-09-11T06:04:28.935 INFO:teuthology.orchestra.run.smithi006.stdout:5.15.0-119-generic 2024-09-11T06:04:28.935 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-119-generic vs 5.15.0-119-generic 2024-09-11T06:04:28.935 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-11T06:04:28.935 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-11T06:04:29.936 INFO:teuthology.task.kernel:Enabling kdb on cluster1.mon.a... 2024-09-11T06:04:29.937 DEBUG:teuthology.orchestra.run.smithi006:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-11T06:04:30.018 INFO:teuthology.orchestra.run.smithi006.stdout:ttyS1 2024-09-11T06:04:30.055 DEBUG:teuthology.parallel:result is None 2024-09-11T06:04:30.055 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-11T06:04:30.065 INFO:teuthology.task.internal:Creating test directory... 2024-09-11T06:04:30.065 DEBUG:teuthology.orchestra.run.smithi006:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-11T06:04:30.067 DEBUG:teuthology.orchestra.run.smithi112:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-11T06:04:30.073 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-11T06:04:30.086 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-11T06:04:30.099 INFO:teuthology.task.internal:Creating archive directory... 2024-09-11T06:04:30.100 DEBUG:teuthology.orchestra.run.smithi006:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-11T06:04:30.114 DEBUG:teuthology.orchestra.run.smithi112:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-11T06:04:30.204 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-11T06:04:30.210 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-11T06:04:30.211 DEBUG:teuthology.orchestra.run.smithi006:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-11T06:04:30.213 DEBUG:teuthology.orchestra.run.smithi112:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-11T06:04:30.231 INFO:teuthology.orchestra.run.smithi006.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-11T06:04:30.233 INFO:teuthology.orchestra.run.smithi112.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-11T06:04:30.239 INFO:teuthology.orchestra.run.smithi006.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-11T06:04:30.240 INFO:teuthology.orchestra.run.smithi112.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-11T06:04:30.242 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-11T06:04:30.248 INFO:teuthology.task.internal:Configuring sudo... 2024-09-11T06:04:30.248 DEBUG:teuthology.orchestra.run.smithi006:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-11T06:04:30.283 DEBUG:teuthology.orchestra.run.smithi112:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-11T06:04:30.296 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-11T06:04:30.306 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-11T06:04:30.306 DEBUG:teuthology.orchestra.run.smithi006:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-11T06:04:30.339 DEBUG:teuthology.orchestra.run.smithi112:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-11T06:04:30.344 DEBUG:teuthology.orchestra.run.smithi006:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-11T06:04:30.390 DEBUG:teuthology.orchestra.run.smithi006:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-11T06:04:30.438 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-09-11T06:04:30.438 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-11T06:04:30.513 DEBUG:teuthology.orchestra.run.smithi112:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-11T06:04:30.519 DEBUG:teuthology.orchestra.run.smithi112:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-11T06:04:30.568 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-11T06:04:30.568 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-11T06:04:30.637 DEBUG:teuthology.orchestra.run.smithi006:> sudo service rsyslog restart 2024-09-11T06:04:30.640 DEBUG:teuthology.orchestra.run.smithi112:> sudo service rsyslog restart 2024-09-11T06:04:30.725 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-11T06:04:30.732 INFO:teuthology.task.internal:Starting timer... 2024-09-11T06:04:30.732 INFO:teuthology.run_tasks:Running task pcp... 2024-09-11T06:04:30.807 INFO:teuthology.run_tasks:Running task selinux... 2024-09-11T06:04:30.815 DEBUG:teuthology.task.selinux:Excluding smithi006: OS 'ubuntu' does not support SELinux 2024-09-11T06:04:30.815 DEBUG:teuthology.task.selinux:Excluding smithi112: OS 'ubuntu' does not support SELinux 2024-09-11T06:04:30.815 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-11T06:04:30.815 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-09-11T06:04:30.815 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-11T06:04:30.815 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-11T06:04:30.824 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-11T06:04:30.838 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-11T06:04:30.839 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi006.front.sepia.ceph.com,smithi112.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-11T06:12:05.340 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi006.front.sepia.ceph.com'), Remote(name='ubuntu@smithi112.front.sepia.ceph.com')] 2024-09-11T06:12:05.342 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi006.front.sepia.ceph.com' 2024-09-11T06:12:05.343 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi006.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T06:12:05.430 DEBUG:teuthology.orchestra.run.smithi006:> true 2024-09-11T06:12:05.509 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi006.front.sepia.ceph.com' 2024-09-11T06:12:05.509 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi112.front.sepia.ceph.com' 2024-09-11T06:12:05.510 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T06:12:05.590 DEBUG:teuthology.orchestra.run.smithi112:> true 2024-09-11T06:12:05.668 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi112.front.sepia.ceph.com' 2024-09-11T06:12:05.669 INFO:teuthology.run_tasks:Running task clock... 2024-09-11T06:12:05.680 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-11T06:12:05.680 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-11T06:12:05.681 DEBUG:teuthology.orchestra.run.smithi006:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-11T06:12:05.683 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-11T06:12:05.683 DEBUG:teuthology.orchestra.run.smithi112:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-11T06:12:05.710 INFO:teuthology.orchestra.run.smithi006.stdout:11 Sep 06:12:05 ntpd[18350]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-09-11T06:12:05.710 INFO:teuthology.orchestra.run.smithi006.stdout:11 Sep 06:12:05 ntpd[18350]: Command line: ntpd -gq 2024-09-11T06:12:05.710 INFO:teuthology.orchestra.run.smithi006.stdout:11 Sep 06:12:05 ntpd[18350]: ---------------------------------------------------- 2024-09-11T06:12:05.710 INFO:teuthology.orchestra.run.smithi006.stdout:11 Sep 06:12:05 ntpd[18350]: ntp-4 is maintained by Network Time Foundation, 2024-09-11T06:12:05.710 INFO:teuthology.orchestra.run.smithi006.stdout:11 Sep 06:12:05 ntpd[18350]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-09-11T06:12:05.710 INFO:teuthology.orchestra.run.smithi006.stdout:11 Sep 06:12:05 ntpd[18350]: corporation. Support and training for ntp-4 are 2024-09-11T06:12:05.711 INFO:teuthology.orchestra.run.smithi006.stdout:11 Sep 06:12:05 ntpd[18350]: available at https://www.nwtime.org/support 2024-09-11T06:12:05.711 INFO:teuthology.orchestra.run.smithi006.stdout:11 Sep 06:12:05 ntpd[18350]: ---------------------------------------------------- 2024-09-11T06:12:05.712 INFO:teuthology.orchestra.run.smithi006.stdout:11 Sep 06:12:05 ntpd[18350]: proto: precision = 0.045 usec (-24) 2024-09-11T06:12:05.712 INFO:teuthology.orchestra.run.smithi006.stdout:11 Sep 06:12:05 ntpd[18350]: basedate set to 2022-02-04 2024-09-11T06:12:05.712 INFO:teuthology.orchestra.run.smithi006.stdout:11 Sep 06:12:05 ntpd[18350]: gps base set to 2022-02-06 (week 2196) 2024-09-11T06:12:05.712 INFO:teuthology.orchestra.run.smithi006.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-11T06:12:05.712 INFO:teuthology.orchestra.run.smithi006.stdout:11 Sep 06:12:05 ntpd[18350]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-11T06:12:05.713 INFO:teuthology.orchestra.run.smithi006.stdout:11 Sep 06:12:05 ntpd[18350]: restrict ::: KOD does nothing without LIMITED. 2024-09-11T06:12:05.713 INFO:teuthology.orchestra.run.smithi006.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-11T06:12:05.714 INFO:teuthology.orchestra.run.smithi006.stdout:11 Sep 06:12:05 ntpd[18350]: Listen and drop on 0 v6wildcard [::]:123 2024-09-11T06:12:05.714 INFO:teuthology.orchestra.run.smithi006.stdout:11 Sep 06:12:05 ntpd[18350]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-11T06:12:05.714 INFO:teuthology.orchestra.run.smithi006.stdout:11 Sep 06:12:05 ntpd[18350]: Listen normally on 2 lo 127.0.0.1:123 2024-09-11T06:12:05.714 INFO:teuthology.orchestra.run.smithi006.stdout:11 Sep 06:12:05 ntpd[18350]: Listen normally on 3 ens1f0 172.21.15.6:123 2024-09-11T06:12:05.714 INFO:teuthology.orchestra.run.smithi006.stdout:11 Sep 06:12:05 ntpd[18350]: Listen normally on 4 lo [::1]:123 2024-09-11T06:12:05.714 INFO:teuthology.orchestra.run.smithi006.stdout:11 Sep 06:12:05 ntpd[18350]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:febd:15e4%4]:123 2024-09-11T06:12:05.714 INFO:teuthology.orchestra.run.smithi006.stdout:11 Sep 06:12:05 ntpd[18350]: Listening on routing socket on fd #22 for interface updates 2024-09-11T06:12:05.733 INFO:teuthology.orchestra.run.smithi112.stdout:11 Sep 06:12:05 ntpd[18434]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-09-11T06:12:05.733 INFO:teuthology.orchestra.run.smithi112.stdout:11 Sep 06:12:05 ntpd[18434]: Command line: ntpd -gq 2024-09-11T06:12:05.734 INFO:teuthology.orchestra.run.smithi112.stdout:11 Sep 06:12:05 ntpd[18434]: ---------------------------------------------------- 2024-09-11T06:12:05.734 INFO:teuthology.orchestra.run.smithi112.stdout:11 Sep 06:12:05 ntpd[18434]: ntp-4 is maintained by Network Time Foundation, 2024-09-11T06:12:05.734 INFO:teuthology.orchestra.run.smithi112.stdout:11 Sep 06:12:05 ntpd[18434]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-09-11T06:12:05.734 INFO:teuthology.orchestra.run.smithi112.stdout:11 Sep 06:12:05 ntpd[18434]: corporation. Support and training for ntp-4 are 2024-09-11T06:12:05.734 INFO:teuthology.orchestra.run.smithi112.stdout:11 Sep 06:12:05 ntpd[18434]: available at https://www.nwtime.org/support 2024-09-11T06:12:05.734 INFO:teuthology.orchestra.run.smithi112.stdout:11 Sep 06:12:05 ntpd[18434]: ---------------------------------------------------- 2024-09-11T06:12:05.735 INFO:teuthology.orchestra.run.smithi112.stdout:11 Sep 06:12:05 ntpd[18434]: proto: precision = 0.044 usec (-24) 2024-09-11T06:12:05.735 INFO:teuthology.orchestra.run.smithi112.stdout:11 Sep 06:12:05 ntpd[18434]: basedate set to 2022-02-04 2024-09-11T06:12:05.735 INFO:teuthology.orchestra.run.smithi112.stdout:11 Sep 06:12:05 ntpd[18434]: gps base set to 2022-02-06 (week 2196) 2024-09-11T06:12:05.735 INFO:teuthology.orchestra.run.smithi112.stdout:11 Sep 06:12:05 ntpd[18434]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-11T06:12:05.735 INFO:teuthology.orchestra.run.smithi112.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-11T06:12:05.735 INFO:teuthology.orchestra.run.smithi112.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-11T06:12:05.736 INFO:teuthology.orchestra.run.smithi112.stdout:11 Sep 06:12:05 ntpd[18434]: restrict ::: KOD does nothing without LIMITED. 2024-09-11T06:12:05.736 INFO:teuthology.orchestra.run.smithi112.stdout:11 Sep 06:12:05 ntpd[18434]: Listen and drop on 0 v6wildcard [::]:123 2024-09-11T06:12:05.736 INFO:teuthology.orchestra.run.smithi112.stdout:11 Sep 06:12:05 ntpd[18434]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-11T06:12:05.736 INFO:teuthology.orchestra.run.smithi112.stdout:11 Sep 06:12:05 ntpd[18434]: Listen normally on 2 lo 127.0.0.1:123 2024-09-11T06:12:05.736 INFO:teuthology.orchestra.run.smithi112.stdout:11 Sep 06:12:05 ntpd[18434]: Listen normally on 3 enp3s0f1 172.21.15.112:123 2024-09-11T06:12:05.737 INFO:teuthology.orchestra.run.smithi112.stdout:11 Sep 06:12:05 ntpd[18434]: Listen normally on 4 lo [::1]:123 2024-09-11T06:12:05.737 INFO:teuthology.orchestra.run.smithi112.stdout:11 Sep 06:12:05 ntpd[18434]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:cb69%5]:123 2024-09-11T06:12:05.737 INFO:teuthology.orchestra.run.smithi112.stdout:11 Sep 06:12:05 ntpd[18434]: Listening on routing socket on fd #22 for interface updates 2024-09-11T06:12:07.711 INFO:teuthology.orchestra.run.smithi006.stderr:11 Sep 06:12:07 ntpd[18350]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T06:12:07.711 INFO:teuthology.orchestra.run.smithi006.stderr:11 Sep 06:12:07 ntpd[18350]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T06:12:07.711 INFO:teuthology.orchestra.run.smithi006.stderr:11 Sep 06:12:07 ntpd[18350]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T06:12:07.711 INFO:teuthology.orchestra.run.smithi006.stderr:11 Sep 06:12:07 ntpd[18350]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T06:12:07.711 INFO:teuthology.orchestra.run.smithi006.stderr:11 Sep 06:12:07 ntpd[18350]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T06:12:07.711 INFO:teuthology.orchestra.run.smithi006.stderr:11 Sep 06:12:07 ntpd[18350]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T06:12:07.711 INFO:teuthology.orchestra.run.smithi006.stderr:11 Sep 06:12:07 ntpd[18350]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T06:12:07.712 INFO:teuthology.orchestra.run.smithi006.stderr:11 Sep 06:12:07 ntpd[18350]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T06:12:08.735 INFO:teuthology.orchestra.run.smithi112.stderr:11 Sep 06:12:08 ntpd[18434]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T06:12:08.735 INFO:teuthology.orchestra.run.smithi112.stderr:11 Sep 06:12:08 ntpd[18434]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T06:12:08.735 INFO:teuthology.orchestra.run.smithi112.stderr:11 Sep 06:12:08 ntpd[18434]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T06:12:08.735 INFO:teuthology.orchestra.run.smithi112.stderr:11 Sep 06:12:08 ntpd[18434]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T06:12:08.736 INFO:teuthology.orchestra.run.smithi112.stderr:11 Sep 06:12:08 ntpd[18434]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T06:12:08.736 INFO:teuthology.orchestra.run.smithi112.stderr:11 Sep 06:12:08 ntpd[18434]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T06:12:08.736 INFO:teuthology.orchestra.run.smithi112.stderr:11 Sep 06:12:08 ntpd[18434]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T06:12:08.736 INFO:teuthology.orchestra.run.smithi112.stderr:11 Sep 06:12:08 ntpd[18434]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T06:12:08.736 INFO:teuthology.orchestra.run.smithi112.stderr:11 Sep 06:12:08 ntpd[18434]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T06:12:08.736 INFO:teuthology.orchestra.run.smithi112.stderr:11 Sep 06:12:08 ntpd[18434]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T06:12:08.736 INFO:teuthology.orchestra.run.smithi112.stderr:11 Sep 06:12:08 ntpd[18434]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T06:12:08.736 INFO:teuthology.orchestra.run.smithi112.stderr:11 Sep 06:12:08 ntpd[18434]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T06:12:09.711 INFO:teuthology.orchestra.run.smithi006.stderr:11 Sep 06:12:09 ntpd[18350]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T06:12:09.711 INFO:teuthology.orchestra.run.smithi006.stderr:11 Sep 06:12:09 ntpd[18350]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T06:12:09.711 INFO:teuthology.orchestra.run.smithi006.stderr:11 Sep 06:12:09 ntpd[18350]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T06:12:09.711 INFO:teuthology.orchestra.run.smithi006.stderr:11 Sep 06:12:09 ntpd[18350]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T06:12:09.711 INFO:teuthology.orchestra.run.smithi006.stderr:11 Sep 06:12:09 ntpd[18350]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T06:12:09.711 INFO:teuthology.orchestra.run.smithi006.stderr:11 Sep 06:12:09 ntpd[18350]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T06:12:09.735 INFO:teuthology.orchestra.run.smithi112.stderr:11 Sep 06:12:09 ntpd[18434]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T06:12:09.735 INFO:teuthology.orchestra.run.smithi112.stderr:11 Sep 06:12:09 ntpd[18434]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T06:12:09.735 INFO:teuthology.orchestra.run.smithi112.stderr:11 Sep 06:12:09 ntpd[18434]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T06:12:09.735 INFO:teuthology.orchestra.run.smithi112.stderr:11 Sep 06:12:09 ntpd[18434]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T06:12:09.735 INFO:teuthology.orchestra.run.smithi112.stderr:11 Sep 06:12:09 ntpd[18434]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T06:12:09.735 INFO:teuthology.orchestra.run.smithi112.stderr:11 Sep 06:12:09 ntpd[18434]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T06:12:09.735 INFO:teuthology.orchestra.run.smithi112.stderr:11 Sep 06:12:09 ntpd[18434]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T06:12:09.735 INFO:teuthology.orchestra.run.smithi112.stderr:11 Sep 06:12:09 ntpd[18434]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T06:12:10.711 INFO:teuthology.orchestra.run.smithi006.stderr:11 Sep 06:12:10 ntpd[18350]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T06:12:10.712 INFO:teuthology.orchestra.run.smithi006.stderr:11 Sep 06:12:10 ntpd[18350]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T06:12:10.712 INFO:teuthology.orchestra.run.smithi006.stderr:11 Sep 06:12:10 ntpd[18350]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T06:12:10.712 INFO:teuthology.orchestra.run.smithi006.stderr:11 Sep 06:12:10 ntpd[18350]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T06:12:10.712 INFO:teuthology.orchestra.run.smithi006.stderr:11 Sep 06:12:10 ntpd[18350]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T06:12:10.712 INFO:teuthology.orchestra.run.smithi006.stderr:11 Sep 06:12:10 ntpd[18350]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T06:12:10.712 INFO:teuthology.orchestra.run.smithi006.stderr:11 Sep 06:12:10 ntpd[18350]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T06:12:10.712 INFO:teuthology.orchestra.run.smithi006.stderr:11 Sep 06:12:10 ntpd[18350]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T06:12:10.712 INFO:teuthology.orchestra.run.smithi006.stderr:11 Sep 06:12:10 ntpd[18350]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T06:12:10.712 INFO:teuthology.orchestra.run.smithi006.stderr:11 Sep 06:12:10 ntpd[18350]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T06:12:10.712 INFO:teuthology.orchestra.run.smithi006.stderr:11 Sep 06:12:10 ntpd[18350]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T06:12:10.712 INFO:teuthology.orchestra.run.smithi006.stderr:11 Sep 06:12:10 ntpd[18350]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T06:12:10.735 INFO:teuthology.orchestra.run.smithi112.stderr:11 Sep 06:12:10 ntpd[18434]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T06:12:10.735 INFO:teuthology.orchestra.run.smithi112.stderr:11 Sep 06:12:10 ntpd[18434]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T06:12:10.735 INFO:teuthology.orchestra.run.smithi112.stderr:11 Sep 06:12:10 ntpd[18434]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T06:12:10.735 INFO:teuthology.orchestra.run.smithi112.stderr:11 Sep 06:12:10 ntpd[18434]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T06:12:10.735 INFO:teuthology.orchestra.run.smithi112.stderr:11 Sep 06:12:10 ntpd[18434]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T06:12:10.735 INFO:teuthology.orchestra.run.smithi112.stderr:11 Sep 06:12:10 ntpd[18434]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T06:12:10.735 INFO:teuthology.orchestra.run.smithi112.stderr:11 Sep 06:12:10 ntpd[18434]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T06:12:10.735 INFO:teuthology.orchestra.run.smithi112.stderr:11 Sep 06:12:10 ntpd[18434]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T06:12:10.735 INFO:teuthology.orchestra.run.smithi112.stderr:11 Sep 06:12:10 ntpd[18434]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T06:12:10.735 INFO:teuthology.orchestra.run.smithi112.stderr:11 Sep 06:12:10 ntpd[18434]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T06:12:10.735 INFO:teuthology.orchestra.run.smithi112.stderr:11 Sep 06:12:10 ntpd[18434]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T06:12:10.736 INFO:teuthology.orchestra.run.smithi112.stderr:11 Sep 06:12:10 ntpd[18434]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T06:12:11.711 INFO:teuthology.orchestra.run.smithi006.stderr:11 Sep 06:12:11 ntpd[18350]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T06:12:11.711 INFO:teuthology.orchestra.run.smithi006.stderr:11 Sep 06:12:11 ntpd[18350]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T06:12:11.711 INFO:teuthology.orchestra.run.smithi006.stderr:11 Sep 06:12:11 ntpd[18350]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T06:12:11.711 INFO:teuthology.orchestra.run.smithi006.stderr:11 Sep 06:12:11 ntpd[18350]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T06:12:11.711 INFO:teuthology.orchestra.run.smithi006.stderr:11 Sep 06:12:11 ntpd[18350]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T06:12:11.711 INFO:teuthology.orchestra.run.smithi006.stderr:11 Sep 06:12:11 ntpd[18350]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T06:12:11.734 INFO:teuthology.orchestra.run.smithi112.stderr:11 Sep 06:12:11 ntpd[18434]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T06:12:11.734 INFO:teuthology.orchestra.run.smithi112.stderr:11 Sep 06:12:11 ntpd[18434]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T06:12:11.734 INFO:teuthology.orchestra.run.smithi112.stderr:11 Sep 06:12:11 ntpd[18434]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T06:12:11.734 INFO:teuthology.orchestra.run.smithi112.stderr:11 Sep 06:12:11 ntpd[18434]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T06:12:11.734 INFO:teuthology.orchestra.run.smithi112.stderr:11 Sep 06:12:11 ntpd[18434]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T06:12:11.735 INFO:teuthology.orchestra.run.smithi112.stderr:11 Sep 06:12:11 ntpd[18434]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T06:12:12.713 INFO:teuthology.orchestra.run.smithi006.stderr:11 Sep 06:12:12 ntpd[18350]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T06:12:12.714 INFO:teuthology.orchestra.run.smithi006.stderr:11 Sep 06:12:12 ntpd[18350]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T06:12:12.714 INFO:teuthology.orchestra.run.smithi006.stderr:11 Sep 06:12:12 ntpd[18350]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T06:12:12.714 INFO:teuthology.orchestra.run.smithi006.stderr:11 Sep 06:12:12 ntpd[18350]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T06:12:12.714 INFO:teuthology.orchestra.run.smithi006.stderr:11 Sep 06:12:12 ntpd[18350]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T06:12:12.714 INFO:teuthology.orchestra.run.smithi006.stderr:11 Sep 06:12:12 ntpd[18350]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T06:12:12.714 INFO:teuthology.orchestra.run.smithi006.stderr:11 Sep 06:12:12 ntpd[18350]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T06:12:12.714 INFO:teuthology.orchestra.run.smithi006.stderr:11 Sep 06:12:12 ntpd[18350]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T06:12:12.714 INFO:teuthology.orchestra.run.smithi006.stderr:11 Sep 06:12:12 ntpd[18350]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T06:12:12.714 INFO:teuthology.orchestra.run.smithi006.stderr:11 Sep 06:12:12 ntpd[18350]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T06:12:12.714 INFO:teuthology.orchestra.run.smithi006.stderr:11 Sep 06:12:12 ntpd[18350]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T06:12:12.714 INFO:teuthology.orchestra.run.smithi006.stderr:11 Sep 06:12:12 ntpd[18350]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T06:12:12.735 INFO:teuthology.orchestra.run.smithi112.stderr:11 Sep 06:12:12 ntpd[18434]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T06:12:12.735 INFO:teuthology.orchestra.run.smithi112.stderr:11 Sep 06:12:12 ntpd[18434]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T06:12:12.735 INFO:teuthology.orchestra.run.smithi112.stderr:11 Sep 06:12:12 ntpd[18434]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T06:12:12.735 INFO:teuthology.orchestra.run.smithi112.stderr:11 Sep 06:12:12 ntpd[18434]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T06:12:12.735 INFO:teuthology.orchestra.run.smithi112.stderr:11 Sep 06:12:12 ntpd[18434]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T06:12:12.735 INFO:teuthology.orchestra.run.smithi112.stderr:11 Sep 06:12:12 ntpd[18434]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T06:12:12.735 INFO:teuthology.orchestra.run.smithi112.stderr:11 Sep 06:12:12 ntpd[18434]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T06:12:12.735 INFO:teuthology.orchestra.run.smithi112.stderr:11 Sep 06:12:12 ntpd[18434]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T06:12:12.736 INFO:teuthology.orchestra.run.smithi112.stderr:11 Sep 06:12:12 ntpd[18434]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T06:12:12.736 INFO:teuthology.orchestra.run.smithi112.stderr:11 Sep 06:12:12 ntpd[18434]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T06:12:12.736 INFO:teuthology.orchestra.run.smithi112.stderr:11 Sep 06:12:12 ntpd[18434]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T06:12:12.736 INFO:teuthology.orchestra.run.smithi112.stderr:11 Sep 06:12:12 ntpd[18434]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T06:12:13.711 INFO:teuthology.orchestra.run.smithi006.stderr:11 Sep 06:12:13 ntpd[18350]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T06:12:13.711 INFO:teuthology.orchestra.run.smithi006.stderr:11 Sep 06:12:13 ntpd[18350]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T06:12:13.711 INFO:teuthology.orchestra.run.smithi006.stderr:11 Sep 06:12:13 ntpd[18350]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T06:12:13.711 INFO:teuthology.orchestra.run.smithi006.stderr:11 Sep 06:12:13 ntpd[18350]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T06:12:13.711 INFO:teuthology.orchestra.run.smithi006.stderr:11 Sep 06:12:13 ntpd[18350]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T06:12:13.711 INFO:teuthology.orchestra.run.smithi006.stderr:11 Sep 06:12:13 ntpd[18350]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T06:12:13.712 INFO:teuthology.orchestra.run.smithi006.stdout:11 Sep 06:12:13 ntpd[18350]: ntpd: time slew -0.000607 s 2024-09-11T06:12:13.712 INFO:teuthology.orchestra.run.smithi006.stdout:ntpd: time slew -0.000607s 2024-09-11T06:12:13.712 INFO:teuthology.orchestra.run.smithi006.stderr:11 Sep 06:12:13 ntpd[18350]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-11T06:12:13.712 INFO:teuthology.orchestra.run.smithi006.stderr:11 Sep 06:12:13 ntpd[18350]: can't open /var/log/ntpstats/loopstats.20240911: Permission denied 2024-09-11T06:12:13.734 INFO:teuthology.orchestra.run.smithi112.stderr:11 Sep 06:12:13 ntpd[18434]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T06:12:13.734 INFO:teuthology.orchestra.run.smithi112.stderr:11 Sep 06:12:13 ntpd[18434]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T06:12:13.734 INFO:teuthology.orchestra.run.smithi112.stderr:11 Sep 06:12:13 ntpd[18434]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T06:12:13.734 INFO:teuthology.orchestra.run.smithi112.stderr:11 Sep 06:12:13 ntpd[18434]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T06:12:13.734 INFO:teuthology.orchestra.run.smithi112.stderr:11 Sep 06:12:13 ntpd[18434]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T06:12:13.734 INFO:teuthology.orchestra.run.smithi112.stderr:11 Sep 06:12:13 ntpd[18434]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T06:12:13.764 INFO:teuthology.orchestra.run.smithi006.stdout: remote refid st t when poll reach delay offset jitter 2024-09-11T06:12:13.765 INFO:teuthology.orchestra.run.smithi006.stdout:============================================================================== 2024-09-11T06:12:13.765 INFO:teuthology.orchestra.run.smithi006.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-11T06:12:13.765 INFO:teuthology.orchestra.run.smithi006.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-11T06:12:13.765 INFO:teuthology.orchestra.run.smithi006.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-11T06:12:13.765 INFO:teuthology.orchestra.run.smithi006.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-11T06:12:14.734 INFO:teuthology.orchestra.run.smithi112.stderr:11 Sep 06:12:14 ntpd[18434]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T06:12:14.735 INFO:teuthology.orchestra.run.smithi112.stderr:11 Sep 06:12:14 ntpd[18434]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T06:12:14.735 INFO:teuthology.orchestra.run.smithi112.stderr:11 Sep 06:12:14 ntpd[18434]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T06:12:14.735 INFO:teuthology.orchestra.run.smithi112.stderr:11 Sep 06:12:14 ntpd[18434]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T06:12:14.735 INFO:teuthology.orchestra.run.smithi112.stderr:11 Sep 06:12:14 ntpd[18434]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T06:12:14.735 INFO:teuthology.orchestra.run.smithi112.stderr:11 Sep 06:12:14 ntpd[18434]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T06:12:14.735 INFO:teuthology.orchestra.run.smithi112.stderr:11 Sep 06:12:14 ntpd[18434]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T06:12:14.735 INFO:teuthology.orchestra.run.smithi112.stderr:11 Sep 06:12:14 ntpd[18434]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T06:12:14.735 INFO:teuthology.orchestra.run.smithi112.stdout:11 Sep 06:12:14 ntpd[18434]: ntpd: time slew -0.001317 s 2024-09-11T06:12:14.735 INFO:teuthology.orchestra.run.smithi112.stdout:ntpd: time slew -0.001317s 2024-09-11T06:12:14.736 INFO:teuthology.orchestra.run.smithi112.stderr:11 Sep 06:12:14 ntpd[18434]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-11T06:12:14.736 INFO:teuthology.orchestra.run.smithi112.stderr:11 Sep 06:12:14 ntpd[18434]: can't open /var/log/ntpstats/loopstats.20240911: Permission denied 2024-09-11T06:12:14.793 INFO:teuthology.orchestra.run.smithi112.stdout: remote refid st t when poll reach delay offset jitter 2024-09-11T06:12:14.793 INFO:teuthology.orchestra.run.smithi112.stdout:============================================================================== 2024-09-11T06:12:14.793 INFO:teuthology.orchestra.run.smithi112.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-11T06:12:14.793 INFO:teuthology.orchestra.run.smithi112.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-11T06:12:14.793 INFO:teuthology.orchestra.run.smithi112.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-11T06:12:14.793 INFO:teuthology.orchestra.run.smithi112.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-11T06:12:14.794 INFO:teuthology.run_tasks:Running task install... 2024-09-11T06:12:14.806 DEBUG:teuthology.task.install:project ceph 2024-09-11T06:12:14.806 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc'}} 2024-09-11T06:12:14.806 DEBUG:teuthology.task.install:config {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': 'b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc'} 2024-09-11T06:12:14.806 INFO:teuthology.task.install:Using flavor: default 2024-09-11T06:12:14.811 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-11T06:12:14.811 INFO:teuthology.task.install:extra packages: ['rbd-mirror'] 2024-09-11T06:12:14.812 DEBUG:teuthology.orchestra.run.smithi006:> sudo apt-key list | grep Ceph 2024-09-11T06:12:14.812 DEBUG:teuthology.orchestra.run.smithi112:> sudo apt-key list | grep Ceph 2024-09-11T06:12:14.876 INFO:teuthology.orchestra.run.smithi006.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-09-11T06:12:14.896 INFO:teuthology.orchestra.run.smithi112.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-09-11T06:12:14.906 INFO:teuthology.orchestra.run.smithi006.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-11T06:12:14.906 INFO:teuthology.orchestra.run.smithi006.stdout:uid [ unknown] Ceph.com (release key) 2024-09-11T06:12:14.907 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, rbd-mirror on remote deb x86_64 2024-09-11T06:12:14.908 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc 2024-09-11T06:12:14.925 INFO:teuthology.orchestra.run.smithi112.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-11T06:12:14.925 INFO:teuthology.orchestra.run.smithi112.stdout:uid [ unknown] Ceph.com (release key) 2024-09-11T06:12:14.926 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, rbd-mirror on remote deb x86_64 2024-09-11T06:12:14.926 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc 2024-09-11T06:12:15.059 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default/ 2024-09-11T06:12:15.059 INFO:teuthology.task.install.deb:Package version is 19.1.1-298-gb53b6d3e-1jammy 2024-09-11T06:12:15.069 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default/ 2024-09-11T06:12:15.069 INFO:teuthology.task.install.deb:Package version is 19.1.1-298-gb53b6d3e-1jammy 2024-09-11T06:12:15.169 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-09-11T06:12:15.169 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-11T06:12:15.183 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-11T06:12:15.183 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-11T06:12:15.184 DEBUG:teuthology.orchestra.run.smithi006:> sudo apt-get update 2024-09-11T06:12:15.198 DEBUG:teuthology.orchestra.run.smithi112:> sudo apt-get update 2024-09-11T06:12:15.423 INFO:teuthology.orchestra.run.smithi006.stdout:Ign:1 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy InRelease 2024-09-11T06:12:15.423 INFO:teuthology.orchestra.run.smithi112.stdout:Ign:1 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy InRelease 2024-09-11T06:12:15.445 INFO:teuthology.orchestra.run.smithi006.stdout:Get:2 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-09-11T06:12:15.445 INFO:teuthology.orchestra.run.smithi112.stdout:Get:2 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-09-11T06:12:15.468 INFO:teuthology.orchestra.run.smithi112.stdout:Ign:3 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy Release.gpg 2024-09-11T06:12:15.468 INFO:teuthology.orchestra.run.smithi006.stdout:Ign:3 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy Release.gpg 2024-09-11T06:12:15.488 INFO:teuthology.orchestra.run.smithi006.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-11T06:12:15.488 INFO:teuthology.orchestra.run.smithi006.stdout:Hit:5 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-11T06:12:15.490 INFO:teuthology.orchestra.run.smithi006.stdout:Get:6 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,404 B] 2024-09-11T06:12:15.490 INFO:teuthology.orchestra.run.smithi112.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-11T06:12:15.491 INFO:teuthology.orchestra.run.smithi112.stdout:Hit:5 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-11T06:12:15.491 INFO:teuthology.orchestra.run.smithi112.stdout:Get:6 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.9 kB] 2024-09-11T06:12:15.521 INFO:teuthology.orchestra.run.smithi006.stdout:Get:7 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.9 kB] 2024-09-11T06:12:15.521 INFO:teuthology.orchestra.run.smithi112.stdout:Get:7 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,404 B] 2024-09-11T06:12:15.560 INFO:teuthology.orchestra.run.smithi006.stdout:Hit:8 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-11T06:12:15.563 INFO:teuthology.orchestra.run.smithi112.stdout:Hit:8 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-11T06:12:15.642 INFO:teuthology.orchestra.run.smithi006.stdout:Hit:9 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-11T06:12:15.644 INFO:teuthology.orchestra.run.smithi112.stdout:Hit:9 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-11T06:12:15.866 INFO:teuthology.orchestra.run.smithi112.stdout:Fetched 29.0 kB in 0s (58.9 kB/s) 2024-09-11T06:12:15.910 INFO:teuthology.orchestra.run.smithi006.stdout:Fetched 29.0 kB in 1s (54.6 kB/s) 2024-09-11T06:12:16.881 INFO:teuthology.orchestra.run.smithi112.stdout:Reading package lists... 2024-09-11T06:12:16.897 DEBUG:teuthology.orchestra.run.smithi112:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.1.1-298-gb53b6d3e-1jammy cephadm=19.1.1-298-gb53b6d3e-1jammy ceph-mds=19.1.1-298-gb53b6d3e-1jammy ceph-mgr=19.1.1-298-gb53b6d3e-1jammy ceph-common=19.1.1-298-gb53b6d3e-1jammy ceph-fuse=19.1.1-298-gb53b6d3e-1jammy ceph-test=19.1.1-298-gb53b6d3e-1jammy ceph-volume=19.1.1-298-gb53b6d3e-1jammy radosgw=19.1.1-298-gb53b6d3e-1jammy python3-rados=19.1.1-298-gb53b6d3e-1jammy python3-rgw=19.1.1-298-gb53b6d3e-1jammy python3-cephfs=19.1.1-298-gb53b6d3e-1jammy python3-rbd=19.1.1-298-gb53b6d3e-1jammy libcephfs2=19.1.1-298-gb53b6d3e-1jammy libcephfs-dev=19.1.1-298-gb53b6d3e-1jammy librados2=19.1.1-298-gb53b6d3e-1jammy librbd1=19.1.1-298-gb53b6d3e-1jammy rbd-fuse=19.1.1-298-gb53b6d3e-1jammy rbd-mirror=19.1.1-298-gb53b6d3e-1jammy 2024-09-11T06:12:16.955 INFO:teuthology.orchestra.run.smithi006.stdout:Reading package lists... 2024-09-11T06:12:16.960 INFO:teuthology.orchestra.run.smithi112.stdout:Reading package lists... 2024-09-11T06:12:16.971 DEBUG:teuthology.orchestra.run.smithi006:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.1.1-298-gb53b6d3e-1jammy cephadm=19.1.1-298-gb53b6d3e-1jammy ceph-mds=19.1.1-298-gb53b6d3e-1jammy ceph-mgr=19.1.1-298-gb53b6d3e-1jammy ceph-common=19.1.1-298-gb53b6d3e-1jammy ceph-fuse=19.1.1-298-gb53b6d3e-1jammy ceph-test=19.1.1-298-gb53b6d3e-1jammy ceph-volume=19.1.1-298-gb53b6d3e-1jammy radosgw=19.1.1-298-gb53b6d3e-1jammy python3-rados=19.1.1-298-gb53b6d3e-1jammy python3-rgw=19.1.1-298-gb53b6d3e-1jammy python3-cephfs=19.1.1-298-gb53b6d3e-1jammy python3-rbd=19.1.1-298-gb53b6d3e-1jammy libcephfs2=19.1.1-298-gb53b6d3e-1jammy libcephfs-dev=19.1.1-298-gb53b6d3e-1jammy librados2=19.1.1-298-gb53b6d3e-1jammy librbd1=19.1.1-298-gb53b6d3e-1jammy rbd-fuse=19.1.1-298-gb53b6d3e-1jammy rbd-mirror=19.1.1-298-gb53b6d3e-1jammy 2024-09-11T06:12:17.036 INFO:teuthology.orchestra.run.smithi006.stdout:Reading package lists... 2024-09-11T06:12:17.170 INFO:teuthology.orchestra.run.smithi112.stdout:Building dependency tree... 2024-09-11T06:12:17.171 INFO:teuthology.orchestra.run.smithi112.stdout:Reading state information... 2024-09-11T06:12:17.248 INFO:teuthology.orchestra.run.smithi006.stdout:Building dependency tree... 2024-09-11T06:12:17.249 INFO:teuthology.orchestra.run.smithi006.stdout:Reading state information... 2024-09-11T06:12:17.376 INFO:teuthology.orchestra.run.smithi112.stdout:The following packages were automatically installed and are no longer required: 2024-09-11T06:12:17.376 INFO:teuthology.orchestra.run.smithi112.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-11T06:12:17.376 INFO:teuthology.orchestra.run.smithi112.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-11T06:12:17.376 INFO:teuthology.orchestra.run.smithi112.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-11T06:12:17.377 INFO:teuthology.orchestra.run.smithi112.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-11T06:12:17.377 INFO:teuthology.orchestra.run.smithi112.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-11T06:12:17.377 INFO:teuthology.orchestra.run.smithi112.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-11T06:12:17.378 INFO:teuthology.orchestra.run.smithi112.stdout:The following additional packages will be installed: 2024-09-11T06:12:17.378 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-11T06:12:17.378 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-09-11T06:12:17.378 INFO:teuthology.orchestra.run.smithi112.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-09-11T06:12:17.378 INFO:teuthology.orchestra.run.smithi112.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-09-11T06:12:17.379 INFO:teuthology.orchestra.run.smithi112.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-09-11T06:12:17.379 INFO:teuthology.orchestra.run.smithi112.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-09-11T06:12:17.379 INFO:teuthology.orchestra.run.smithi112.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-09-11T06:12:17.379 INFO:teuthology.orchestra.run.smithi112.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-09-11T06:12:17.379 INFO:teuthology.orchestra.run.smithi112.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-09-11T06:12:17.379 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-09-11T06:12:17.379 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-11T06:12:17.380 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-11T06:12:17.380 INFO:teuthology.orchestra.run.smithi112.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-11T06:12:17.380 INFO:teuthology.orchestra.run.smithi112.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-11T06:12:17.380 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-11T06:12:17.380 INFO:teuthology.orchestra.run.smithi112.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-11T06:12:17.380 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-09-11T06:12:17.380 INFO:teuthology.orchestra.run.smithi112.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-09-11T06:12:17.380 INFO:teuthology.orchestra.run.smithi112.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-09-11T06:12:17.380 INFO:teuthology.orchestra.run.smithi112.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-09-11T06:12:17.380 INFO:teuthology.orchestra.run.smithi112.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-09-11T06:12:17.380 INFO:teuthology.orchestra.run.smithi112.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-09-11T06:12:17.380 INFO:teuthology.orchestra.run.smithi112.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-09-11T06:12:17.380 INFO:teuthology.orchestra.run.smithi112.stdout: xmlstarlet zip 2024-09-11T06:12:17.381 INFO:teuthology.orchestra.run.smithi112.stdout:Suggested packages: 2024-09-11T06:12:17.381 INFO:teuthology.orchestra.run.smithi112.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-09-11T06:12:17.381 INFO:teuthology.orchestra.run.smithi112.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-09-11T06:12:17.381 INFO:teuthology.orchestra.run.smithi112.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-09-11T06:12:17.381 INFO:teuthology.orchestra.run.smithi112.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-09-11T06:12:17.381 INFO:teuthology.orchestra.run.smithi112.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-09-11T06:12:17.381 INFO:teuthology.orchestra.run.smithi112.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-09-11T06:12:17.381 INFO:teuthology.orchestra.run.smithi112.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-09-11T06:12:17.381 INFO:teuthology.orchestra.run.smithi112.stdout:Recommended packages: 2024-09-11T06:12:17.381 INFO:teuthology.orchestra.run.smithi112.stdout: btrfs-tools 2024-09-11T06:12:17.463 INFO:teuthology.orchestra.run.smithi006.stdout:The following packages were automatically installed and are no longer required: 2024-09-11T06:12:17.463 INFO:teuthology.orchestra.run.smithi006.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-11T06:12:17.463 INFO:teuthology.orchestra.run.smithi006.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-11T06:12:17.463 INFO:teuthology.orchestra.run.smithi006.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-11T06:12:17.464 INFO:teuthology.orchestra.run.smithi006.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-11T06:12:17.464 INFO:teuthology.orchestra.run.smithi006.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-11T06:12:17.464 INFO:teuthology.orchestra.run.smithi006.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-11T06:12:17.465 INFO:teuthology.orchestra.run.smithi006.stdout:The following additional packages will be installed: 2024-09-11T06:12:17.465 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-11T06:12:17.465 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-09-11T06:12:17.465 INFO:teuthology.orchestra.run.smithi006.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-09-11T06:12:17.465 INFO:teuthology.orchestra.run.smithi006.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-09-11T06:12:17.466 INFO:teuthology.orchestra.run.smithi006.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-09-11T06:12:17.467 INFO:teuthology.orchestra.run.smithi006.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-09-11T06:12:17.467 INFO:teuthology.orchestra.run.smithi006.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-09-11T06:12:17.467 INFO:teuthology.orchestra.run.smithi006.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-09-11T06:12:17.467 INFO:teuthology.orchestra.run.smithi006.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-09-11T06:12:17.467 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-09-11T06:12:17.467 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-11T06:12:17.467 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-11T06:12:17.467 INFO:teuthology.orchestra.run.smithi006.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-11T06:12:17.467 INFO:teuthology.orchestra.run.smithi006.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-11T06:12:17.467 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-11T06:12:17.467 INFO:teuthology.orchestra.run.smithi006.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-11T06:12:17.467 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-09-11T06:12:17.467 INFO:teuthology.orchestra.run.smithi006.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-09-11T06:12:17.468 INFO:teuthology.orchestra.run.smithi006.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-09-11T06:12:17.468 INFO:teuthology.orchestra.run.smithi006.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-09-11T06:12:17.468 INFO:teuthology.orchestra.run.smithi006.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-09-11T06:12:17.468 INFO:teuthology.orchestra.run.smithi006.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-09-11T06:12:17.468 INFO:teuthology.orchestra.run.smithi006.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-09-11T06:12:17.468 INFO:teuthology.orchestra.run.smithi006.stdout: xmlstarlet zip 2024-09-11T06:12:17.468 INFO:teuthology.orchestra.run.smithi006.stdout:Suggested packages: 2024-09-11T06:12:17.468 INFO:teuthology.orchestra.run.smithi006.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-09-11T06:12:17.469 INFO:teuthology.orchestra.run.smithi006.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-09-11T06:12:17.469 INFO:teuthology.orchestra.run.smithi006.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-09-11T06:12:17.469 INFO:teuthology.orchestra.run.smithi006.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-09-11T06:12:17.469 INFO:teuthology.orchestra.run.smithi006.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-09-11T06:12:17.469 INFO:teuthology.orchestra.run.smithi006.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-09-11T06:12:17.469 INFO:teuthology.orchestra.run.smithi006.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-09-11T06:12:17.469 INFO:teuthology.orchestra.run.smithi006.stdout:Recommended packages: 2024-09-11T06:12:17.469 INFO:teuthology.orchestra.run.smithi006.stdout: btrfs-tools 2024-09-11T06:12:17.484 INFO:teuthology.orchestra.run.smithi112.stdout:The following NEW packages will be installed: 2024-09-11T06:12:17.485 INFO:teuthology.orchestra.run.smithi112.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-11T06:12:17.485 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-11T06:12:17.485 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-09-11T06:12:17.486 INFO:teuthology.orchestra.run.smithi112.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-09-11T06:12:17.486 INFO:teuthology.orchestra.run.smithi112.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-09-11T06:12:17.486 INFO:teuthology.orchestra.run.smithi112.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-09-11T06:12:17.486 INFO:teuthology.orchestra.run.smithi112.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-09-11T06:12:17.486 INFO:teuthology.orchestra.run.smithi112.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-09-11T06:12:17.486 INFO:teuthology.orchestra.run.smithi112.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-09-11T06:12:17.486 INFO:teuthology.orchestra.run.smithi112.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-09-11T06:12:17.486 INFO:teuthology.orchestra.run.smithi112.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-09-11T06:12:17.487 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-09-11T06:12:17.487 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-11T06:12:17.487 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-11T06:12:17.487 INFO:teuthology.orchestra.run.smithi112.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-11T06:12:17.487 INFO:teuthology.orchestra.run.smithi112.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-11T06:12:17.487 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-11T06:12:17.487 INFO:teuthology.orchestra.run.smithi112.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-11T06:12:17.487 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-09-11T06:12:17.487 INFO:teuthology.orchestra.run.smithi112.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-09-11T06:12:17.487 INFO:teuthology.orchestra.run.smithi112.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-09-11T06:12:17.487 INFO:teuthology.orchestra.run.smithi112.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-09-11T06:12:17.487 INFO:teuthology.orchestra.run.smithi112.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-09-11T06:12:17.487 INFO:teuthology.orchestra.run.smithi112.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-09-11T06:12:17.487 INFO:teuthology.orchestra.run.smithi112.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-09-11T06:12:17.488 INFO:teuthology.orchestra.run.smithi112.stdout: radosgw rbd-fuse rbd-mirror socat unzip xmlstarlet zip 2024-09-11T06:12:17.579 INFO:teuthology.orchestra.run.smithi006.stdout:The following NEW packages will be installed: 2024-09-11T06:12:17.579 INFO:teuthology.orchestra.run.smithi006.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-11T06:12:17.579 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-11T06:12:17.579 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-09-11T06:12:17.579 INFO:teuthology.orchestra.run.smithi006.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-09-11T06:12:17.579 INFO:teuthology.orchestra.run.smithi006.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-09-11T06:12:17.579 INFO:teuthology.orchestra.run.smithi006.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-09-11T06:12:17.580 INFO:teuthology.orchestra.run.smithi006.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-09-11T06:12:17.580 INFO:teuthology.orchestra.run.smithi006.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-09-11T06:12:17.581 INFO:teuthology.orchestra.run.smithi006.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-09-11T06:12:17.581 INFO:teuthology.orchestra.run.smithi006.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-09-11T06:12:17.581 INFO:teuthology.orchestra.run.smithi006.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-09-11T06:12:17.581 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-09-11T06:12:17.581 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-11T06:12:17.581 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-11T06:12:17.581 INFO:teuthology.orchestra.run.smithi006.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-11T06:12:17.581 INFO:teuthology.orchestra.run.smithi006.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-11T06:12:17.581 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-11T06:12:17.581 INFO:teuthology.orchestra.run.smithi006.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-11T06:12:17.581 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-09-11T06:12:17.581 INFO:teuthology.orchestra.run.smithi006.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-09-11T06:12:17.581 INFO:teuthology.orchestra.run.smithi006.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-09-11T06:12:17.582 INFO:teuthology.orchestra.run.smithi006.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-09-11T06:12:17.582 INFO:teuthology.orchestra.run.smithi006.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-09-11T06:12:17.582 INFO:teuthology.orchestra.run.smithi006.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-09-11T06:12:17.582 INFO:teuthology.orchestra.run.smithi006.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-09-11T06:12:17.582 INFO:teuthology.orchestra.run.smithi006.stdout: radosgw rbd-fuse rbd-mirror socat unzip xmlstarlet zip 2024-09-11T06:12:17.629 INFO:teuthology.orchestra.run.smithi112.stdout:0 upgraded, 119 newly installed, 0 to remove and 331 not upgraded. 2024-09-11T06:12:17.629 INFO:teuthology.orchestra.run.smithi112.stdout:Need to get 184 MB of archives. 2024-09-11T06:12:17.629 INFO:teuthology.orchestra.run.smithi112.stdout:After this operation, 842 MB of additional disk space will be used. 2024-09-11T06:12:17.629 INFO:teuthology.orchestra.run.smithi112.stdout:Get:1 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.1.1-298-gb53b6d3e-1jammy [3,588 kB] 2024-09-11T06:12:17.674 INFO:teuthology.orchestra.run.smithi112.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-09-11T06:12:17.727 INFO:teuthology.orchestra.run.smithi006.stdout:0 upgraded, 119 newly installed, 0 to remove and 331 not upgraded. 2024-09-11T06:12:17.727 INFO:teuthology.orchestra.run.smithi006.stdout:Need to get 184 MB of archives. 2024-09-11T06:12:17.727 INFO:teuthology.orchestra.run.smithi006.stdout:After this operation, 842 MB of additional disk space will be used. 2024-09-11T06:12:17.727 INFO:teuthology.orchestra.run.smithi006.stdout:Get:1 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.1.1-298-gb53b6d3e-1jammy [3,588 kB] 2024-09-11T06:12:17.771 INFO:teuthology.orchestra.run.smithi006.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-09-11T06:12:17.809 INFO:teuthology.orchestra.run.smithi112.stdout:Get:3 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.1.1-298-gb53b6d3e-1jammy [3,243 kB] 2024-09-11T06:12:17.872 INFO:teuthology.orchestra.run.smithi112.stdout:Get:4 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.1.1-298-gb53b6d3e-1jammy [719 kB] 2024-09-11T06:12:17.887 INFO:teuthology.orchestra.run.smithi112.stdout:Get:5 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.1.1-298-gb53b6d3e-1jammy [356 kB] 2024-09-11T06:12:17.902 INFO:teuthology.orchestra.run.smithi112.stdout:Get:6 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.1.1-298-gb53b6d3e-1jammy [32.8 kB] 2024-09-11T06:12:17.902 INFO:teuthology.orchestra.run.smithi112.stdout:Get:7 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.1.1-298-gb53b6d3e-1jammy [180 kB] 2024-09-11T06:12:17.905 INFO:teuthology.orchestra.run.smithi112.stdout:Get:8 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.1.1-298-gb53b6d3e-1jammy [67.6 kB] 2024-09-11T06:12:17.907 INFO:teuthology.orchestra.run.smithi112.stdout:Get:9 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.1.1-298-gb53b6d3e-1jammy [333 kB] 2024-09-11T06:12:17.912 INFO:teuthology.orchestra.run.smithi112.stdout:Get:10 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.1.1-298-gb53b6d3e-1jammy [6,875 kB] 2024-09-11T06:12:18.029 INFO:teuthology.orchestra.run.smithi006.stdout:Get:3 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.1.1-298-gb53b6d3e-1jammy [3,243 kB] 2024-09-11T06:12:18.066 INFO:teuthology.orchestra.run.smithi112.stdout:Get:11 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.1.1-298-gb53b6d3e-1jammy [112 kB] 2024-09-11T06:12:18.068 INFO:teuthology.orchestra.run.smithi112.stdout:Get:12 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.1.1-298-gb53b6d3e-1jammy [472 kB] 2024-09-11T06:12:18.075 INFO:teuthology.orchestra.run.smithi112.stdout:Get:13 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.1.1-298-gb53b6d3e-1jammy [25.9 MB] 2024-09-11T06:12:18.095 INFO:teuthology.orchestra.run.smithi112.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-09-11T06:12:18.107 INFO:teuthology.orchestra.run.smithi112.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-09-11T06:12:18.188 INFO:teuthology.orchestra.run.smithi112.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-09-11T06:12:18.191 INFO:teuthology.orchestra.run.smithi006.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-09-11T06:12:18.203 INFO:teuthology.orchestra.run.smithi006.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-09-11T06:12:18.207 INFO:teuthology.orchestra.run.smithi006.stdout:Get:6 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.1.1-298-gb53b6d3e-1jammy [719 kB] 2024-09-11T06:12:18.222 INFO:teuthology.orchestra.run.smithi006.stdout:Get:7 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.1.1-298-gb53b6d3e-1jammy [356 kB] 2024-09-11T06:12:18.270 INFO:teuthology.orchestra.run.smithi006.stdout:Get:8 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.1.1-298-gb53b6d3e-1jammy [32.8 kB] 2024-09-11T06:12:18.271 INFO:teuthology.orchestra.run.smithi006.stdout:Get:9 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.1.1-298-gb53b6d3e-1jammy [180 kB] 2024-09-11T06:12:18.275 INFO:teuthology.orchestra.run.smithi006.stdout:Get:10 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.1.1-298-gb53b6d3e-1jammy [67.6 kB] 2024-09-11T06:12:18.277 INFO:teuthology.orchestra.run.smithi006.stdout:Get:11 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.1.1-298-gb53b6d3e-1jammy [333 kB] 2024-09-11T06:12:18.284 INFO:teuthology.orchestra.run.smithi006.stdout:Get:12 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.1.1-298-gb53b6d3e-1jammy [6,875 kB] 2024-09-11T06:12:18.284 INFO:teuthology.orchestra.run.smithi006.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-09-11T06:12:18.417 INFO:teuthology.orchestra.run.smithi112.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-09-11T06:12:18.428 INFO:teuthology.orchestra.run.smithi112.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-09-11T06:12:18.457 INFO:teuthology.orchestra.run.smithi112.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-09-11T06:12:18.466 INFO:teuthology.orchestra.run.smithi112.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-09-11T06:12:18.467 INFO:teuthology.orchestra.run.smithi112.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-09-11T06:12:18.468 INFO:teuthology.orchestra.run.smithi112.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-09-11T06:12:18.486 INFO:teuthology.orchestra.run.smithi112.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-09-11T06:12:18.491 INFO:teuthology.orchestra.run.smithi112.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-09-11T06:12:18.495 INFO:teuthology.orchestra.run.smithi112.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-09-11T06:12:18.499 INFO:teuthology.orchestra.run.smithi112.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-09-11T06:12:18.514 INFO:teuthology.orchestra.run.smithi006.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-09-11T06:12:18.525 INFO:teuthology.orchestra.run.smithi006.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-09-11T06:12:18.554 INFO:teuthology.orchestra.run.smithi006.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-09-11T06:12:18.563 INFO:teuthology.orchestra.run.smithi006.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-09-11T06:12:18.564 INFO:teuthology.orchestra.run.smithi006.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-09-11T06:12:18.565 INFO:teuthology.orchestra.run.smithi006.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-09-11T06:12:18.579 INFO:teuthology.orchestra.run.smithi112.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-09-11T06:12:18.583 INFO:teuthology.orchestra.run.smithi006.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-09-11T06:12:18.584 INFO:teuthology.orchestra.run.smithi112.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-09-11T06:12:18.588 INFO:teuthology.orchestra.run.smithi006.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-09-11T06:12:18.588 INFO:teuthology.orchestra.run.smithi112.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-09-11T06:12:18.591 INFO:teuthology.orchestra.run.smithi112.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-09-11T06:12:18.592 INFO:teuthology.orchestra.run.smithi006.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-09-11T06:12:18.593 INFO:teuthology.orchestra.run.smithi112.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-09-11T06:12:18.593 INFO:teuthology.orchestra.run.smithi112.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-09-11T06:12:18.595 INFO:teuthology.orchestra.run.smithi112.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-09-11T06:12:18.596 INFO:teuthology.orchestra.run.smithi112.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-09-11T06:12:18.597 INFO:teuthology.orchestra.run.smithi112.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-09-11T06:12:18.597 INFO:teuthology.orchestra.run.smithi006.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-09-11T06:12:18.647 INFO:teuthology.orchestra.run.smithi006.stdout:Get:24 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.1.1-298-gb53b6d3e-1jammy [112 kB] 2024-09-11T06:12:18.650 INFO:teuthology.orchestra.run.smithi006.stdout:Get:25 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.1.1-298-gb53b6d3e-1jammy [472 kB] 2024-09-11T06:12:18.662 INFO:teuthology.orchestra.run.smithi112.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-09-11T06:12:18.678 INFO:teuthology.orchestra.run.smithi006.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-09-11T06:12:18.679 INFO:teuthology.orchestra.run.smithi112.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-09-11T06:12:18.680 INFO:teuthology.orchestra.run.smithi112.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-09-11T06:12:18.681 INFO:teuthology.orchestra.run.smithi112.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-09-11T06:12:18.682 INFO:teuthology.orchestra.run.smithi006.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-09-11T06:12:18.686 INFO:teuthology.orchestra.run.smithi112.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-09-11T06:12:18.686 INFO:teuthology.orchestra.run.smithi006.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-09-11T06:12:18.687 INFO:teuthology.orchestra.run.smithi112.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-09-11T06:12:18.690 INFO:teuthology.orchestra.run.smithi006.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-09-11T06:12:18.691 INFO:teuthology.orchestra.run.smithi006.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-09-11T06:12:18.691 INFO:teuthology.orchestra.run.smithi006.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-09-11T06:12:18.693 INFO:teuthology.orchestra.run.smithi006.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-09-11T06:12:18.694 INFO:teuthology.orchestra.run.smithi006.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-09-11T06:12:18.695 INFO:teuthology.orchestra.run.smithi006.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-09-11T06:12:18.700 INFO:teuthology.orchestra.run.smithi006.stdout:Get:35 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.1.1-298-gb53b6d3e-1jammy [25.9 MB] 2024-09-11T06:12:18.745 INFO:teuthology.orchestra.run.smithi112.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-09-11T06:12:18.746 INFO:teuthology.orchestra.run.smithi112.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-09-11T06:12:18.761 INFO:teuthology.orchestra.run.smithi006.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-09-11T06:12:18.777 INFO:teuthology.orchestra.run.smithi006.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-09-11T06:12:18.778 INFO:teuthology.orchestra.run.smithi006.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-09-11T06:12:18.778 INFO:teuthology.orchestra.run.smithi006.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-09-11T06:12:18.784 INFO:teuthology.orchestra.run.smithi006.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-09-11T06:12:18.785 INFO:teuthology.orchestra.run.smithi006.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-09-11T06:12:18.844 INFO:teuthology.orchestra.run.smithi006.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-09-11T06:12:18.845 INFO:teuthology.orchestra.run.smithi006.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-09-11T06:12:18.845 INFO:teuthology.orchestra.run.smithi006.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-11T06:12:18.846 INFO:teuthology.orchestra.run.smithi006.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-09-11T06:12:18.855 INFO:teuthology.orchestra.run.smithi112.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-11T06:12:18.856 INFO:teuthology.orchestra.run.smithi112.stdout:Get:45 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.1.1-298-gb53b6d3e-1jammy [5,145 kB] 2024-09-11T06:12:18.856 INFO:teuthology.orchestra.run.smithi112.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-09-11T06:12:18.857 INFO:teuthology.orchestra.run.smithi112.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-webob all 1:1.8.6-1.1ubuntu0.1 [86.7 kB] 2024-09-11T06:12:18.860 INFO:teuthology.orchestra.run.smithi112.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-09-11T06:12:18.863 INFO:teuthology.orchestra.run.smithi112.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-09-11T06:12:18.863 INFO:teuthology.orchestra.run.smithi112.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-09-11T06:12:18.872 INFO:teuthology.orchestra.run.smithi112.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-09-11T06:12:18.906 INFO:teuthology.orchestra.run.smithi112.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-09-11T06:12:18.924 INFO:teuthology.orchestra.run.smithi006.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-webob all 1:1.8.6-1.1ubuntu0.1 [86.7 kB] 2024-09-11T06:12:18.926 INFO:teuthology.orchestra.run.smithi006.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-09-11T06:12:18.927 INFO:teuthology.orchestra.run.smithi006.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-09-11T06:12:18.928 INFO:teuthology.orchestra.run.smithi006.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-09-11T06:12:18.939 INFO:teuthology.orchestra.run.smithi006.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-09-11T06:12:18.939 INFO:teuthology.orchestra.run.smithi112.stdout:Get:53 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.1.1-298-gb53b6d3e-1jammy [245 kB] 2024-09-11T06:12:18.943 INFO:teuthology.orchestra.run.smithi112.stdout:Get:54 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.1.1-298-gb53b6d3e-1jammy [124 kB] 2024-09-11T06:12:18.946 INFO:teuthology.orchestra.run.smithi112.stdout:Get:55 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.1.1-298-gb53b6d3e-1jammy [1,555 kB] 2024-09-11T06:12:18.954 INFO:teuthology.orchestra.run.smithi112.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-09-11T06:12:18.955 INFO:teuthology.orchestra.run.smithi112.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-09-11T06:12:18.957 INFO:teuthology.orchestra.run.smithi112.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-09-11T06:12:18.961 INFO:teuthology.orchestra.run.smithi112.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-idna all 3.3-1ubuntu0.1 [52.1 kB] 2024-09-11T06:12:18.988 INFO:teuthology.orchestra.run.smithi112.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-09-11T06:12:18.991 INFO:teuthology.orchestra.run.smithi112.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-09-11T06:12:18.993 INFO:teuthology.orchestra.run.smithi112.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.2 [181 kB] 2024-09-11T06:12:18.994 INFO:teuthology.orchestra.run.smithi112.stdout:Get:63 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.1.1-298-gb53b6d3e-1jammy [6,143 kB] 2024-09-11T06:12:18.998 INFO:teuthology.orchestra.run.smithi112.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-09-11T06:12:18.999 INFO:teuthology.orchestra.run.smithi112.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-09-11T06:12:19.005 INFO:teuthology.orchestra.run.smithi006.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-09-11T06:12:19.006 INFO:teuthology.orchestra.run.smithi006.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-09-11T06:12:19.007 INFO:teuthology.orchestra.run.smithi006.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-09-11T06:12:19.010 INFO:teuthology.orchestra.run.smithi006.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-09-11T06:12:19.014 INFO:teuthology.orchestra.run.smithi006.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-idna all 3.3-1ubuntu0.1 [52.1 kB] 2024-09-11T06:12:19.068 INFO:teuthology.orchestra.run.smithi112.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-09-11T06:12:19.073 INFO:teuthology.orchestra.run.smithi112.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-09-11T06:12:19.087 INFO:teuthology.orchestra.run.smithi006.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-09-11T06:12:19.089 INFO:teuthology.orchestra.run.smithi006.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-09-11T06:12:19.091 INFO:teuthology.orchestra.run.smithi006.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.2 [181 kB] 2024-09-11T06:12:19.096 INFO:teuthology.orchestra.run.smithi006.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-09-11T06:12:19.097 INFO:teuthology.orchestra.run.smithi006.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-09-11T06:12:19.169 INFO:teuthology.orchestra.run.smithi006.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-09-11T06:12:19.173 INFO:teuthology.orchestra.run.smithi006.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-09-11T06:12:19.190 INFO:teuthology.orchestra.run.smithi112.stdout:Get:68 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.1.1-298-gb53b6d3e-1jammy [22.8 MB] 2024-09-11T06:12:19.239 INFO:teuthology.orchestra.run.smithi112.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-09-11T06:12:19.241 INFO:teuthology.orchestra.run.smithi112.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.2 [108 kB] 2024-09-11T06:12:19.243 INFO:teuthology.orchestra.run.smithi112.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-11T06:12:19.244 INFO:teuthology.orchestra.run.smithi112.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-09-11T06:12:19.246 INFO:teuthology.orchestra.run.smithi112.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-09-11T06:12:19.282 INFO:teuthology.orchestra.run.smithi112.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-09-11T06:12:19.285 INFO:teuthology.orchestra.run.smithi112.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-09-11T06:12:19.286 INFO:teuthology.orchestra.run.smithi112.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-09-11T06:12:19.336 INFO:teuthology.orchestra.run.smithi006.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-09-11T06:12:19.337 INFO:teuthology.orchestra.run.smithi006.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.2 [108 kB] 2024-09-11T06:12:19.340 INFO:teuthology.orchestra.run.smithi006.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-11T06:12:19.340 INFO:teuthology.orchestra.run.smithi006.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-09-11T06:12:19.343 INFO:teuthology.orchestra.run.smithi006.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-09-11T06:12:19.344 INFO:teuthology.orchestra.run.smithi112.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-09-11T06:12:19.344 INFO:teuthology.orchestra.run.smithi112.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-09-11T06:12:19.377 INFO:teuthology.orchestra.run.smithi006.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-09-11T06:12:19.380 INFO:teuthology.orchestra.run.smithi006.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-09-11T06:12:19.380 INFO:teuthology.orchestra.run.smithi006.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-09-11T06:12:19.407 INFO:teuthology.orchestra.run.smithi112.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-09-11T06:12:19.407 INFO:teuthology.orchestra.run.smithi112.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-09-11T06:12:19.408 INFO:teuthology.orchestra.run.smithi112.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-09-11T06:12:19.410 INFO:teuthology.orchestra.run.smithi112.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-09-11T06:12:19.410 INFO:teuthology.orchestra.run.smithi112.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-09-11T06:12:19.411 INFO:teuthology.orchestra.run.smithi112.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-09-11T06:12:19.418 INFO:teuthology.orchestra.run.smithi112.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-09-11T06:12:19.439 INFO:teuthology.orchestra.run.smithi006.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-09-11T06:12:19.439 INFO:teuthology.orchestra.run.smithi006.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-09-11T06:12:19.488 INFO:teuthology.orchestra.run.smithi112.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-09-11T06:12:19.491 INFO:teuthology.orchestra.run.smithi112.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-09-11T06:12:19.493 INFO:teuthology.orchestra.run.smithi112.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-09-11T06:12:19.502 INFO:teuthology.orchestra.run.smithi006.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-09-11T06:12:19.526 INFO:teuthology.orchestra.run.smithi006.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-09-11T06:12:19.527 INFO:teuthology.orchestra.run.smithi006.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-09-11T06:12:19.569 INFO:teuthology.orchestra.run.smithi112.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-09-11T06:12:19.576 INFO:teuthology.orchestra.run.smithi112.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-09-11T06:12:19.579 INFO:teuthology.orchestra.run.smithi112.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-09-11T06:12:19.580 INFO:teuthology.orchestra.run.smithi112.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.2 [473 kB] 2024-09-11T06:12:19.586 INFO:teuthology.orchestra.run.smithi006.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-09-11T06:12:19.586 INFO:teuthology.orchestra.run.smithi006.stdout:Get:77 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.1.1-298-gb53b6d3e-1jammy [5,145 kB] 2024-09-11T06:12:19.587 INFO:teuthology.orchestra.run.smithi006.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-09-11T06:12:19.588 INFO:teuthology.orchestra.run.smithi006.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-09-11T06:12:19.591 INFO:teuthology.orchestra.run.smithi112.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-09-11T06:12:19.593 INFO:teuthology.orchestra.run.smithi112.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-09-11T06:12:19.597 INFO:teuthology.orchestra.run.smithi006.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-09-11T06:12:19.601 INFO:teuthology.orchestra.run.smithi112.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-09-11T06:12:19.601 INFO:teuthology.orchestra.run.smithi006.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-09-11T06:12:19.605 INFO:teuthology.orchestra.run.smithi006.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-09-11T06:12:19.606 INFO:teuthology.orchestra.run.smithi006.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-09-11T06:12:19.651 INFO:teuthology.orchestra.run.smithi112.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-11T06:12:19.652 INFO:teuthology.orchestra.run.smithi112.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-09-11T06:12:19.653 INFO:teuthology.orchestra.run.smithi112.stdout:Get:98 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-09-11T06:12:19.665 INFO:teuthology.orchestra.run.smithi006.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-09-11T06:12:19.674 INFO:teuthology.orchestra.run.smithi006.stdout:Get:85 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.1.1-298-gb53b6d3e-1jammy [245 kB] 2024-09-11T06:12:19.678 INFO:teuthology.orchestra.run.smithi006.stdout:Get:86 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.1.1-298-gb53b6d3e-1jammy [124 kB] 2024-09-11T06:12:19.681 INFO:teuthology.orchestra.run.smithi006.stdout:Get:87 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.1.1-298-gb53b6d3e-1jammy [1,555 kB] 2024-09-11T06:12:19.696 INFO:teuthology.orchestra.run.smithi006.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-09-11T06:12:19.699 INFO:teuthology.orchestra.run.smithi006.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-09-11T06:12:19.700 INFO:teuthology.orchestra.run.smithi006.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.2 [473 kB] 2024-09-11T06:12:19.711 INFO:teuthology.orchestra.run.smithi006.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-09-11T06:12:19.713 INFO:teuthology.orchestra.run.smithi006.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-09-11T06:12:19.732 INFO:teuthology.orchestra.run.smithi112.stdout:Get:99 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-09-11T06:12:19.732 INFO:teuthology.orchestra.run.smithi006.stdout:Get:93 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.1.1-298-gb53b6d3e-1jammy [6,143 kB] 2024-09-11T06:12:19.734 INFO:teuthology.orchestra.run.smithi112.stdout:Get:100 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-09-11T06:12:19.751 INFO:teuthology.orchestra.run.smithi006.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-09-11T06:12:19.752 INFO:teuthology.orchestra.run.smithi006.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-11T06:12:19.753 INFO:teuthology.orchestra.run.smithi006.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-09-11T06:12:19.754 INFO:teuthology.orchestra.run.smithi112.stdout:Get:101 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-09-11T06:12:19.754 INFO:teuthology.orchestra.run.smithi006.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-09-11T06:12:19.755 INFO:teuthology.orchestra.run.smithi112.stdout:Get:102 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-09-11T06:12:19.755 INFO:teuthology.orchestra.run.smithi112.stdout:Get:103 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-09-11T06:12:19.761 INFO:teuthology.orchestra.run.smithi112.stdout:Get:104 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-09-11T06:12:19.763 INFO:teuthology.orchestra.run.smithi112.stdout:Get:105 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-09-11T06:12:19.828 INFO:teuthology.orchestra.run.smithi006.stdout:Get:98 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-09-11T06:12:19.830 INFO:teuthology.orchestra.run.smithi006.stdout:Get:99 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-09-11T06:12:19.847 INFO:teuthology.orchestra.run.smithi006.stdout:Get:100 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-09-11T06:12:19.848 INFO:teuthology.orchestra.run.smithi006.stdout:Get:101 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-09-11T06:12:19.849 INFO:teuthology.orchestra.run.smithi006.stdout:Get:102 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-09-11T06:12:19.910 INFO:teuthology.orchestra.run.smithi006.stdout:Get:103 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-09-11T06:12:19.911 INFO:teuthology.orchestra.run.smithi006.stdout:Get:104 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-09-11T06:12:19.939 INFO:teuthology.orchestra.run.smithi006.stdout:Get:105 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.1.1-298-gb53b6d3e-1jammy [22.8 MB] 2024-09-11T06:12:19.995 INFO:teuthology.orchestra.run.smithi112.stdout:Get:106 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.1.1-298-gb53b6d3e-1jammy [14.1 kB] 2024-09-11T06:12:19.995 INFO:teuthology.orchestra.run.smithi112.stdout:Get:107 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.1.1-298-gb53b6d3e-1jammy [877 kB] 2024-09-11T06:12:20.013 INFO:teuthology.orchestra.run.smithi112.stdout:Get:108 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.1.1-298-gb53b6d3e-1jammy [2,462 kB] 2024-09-11T06:12:20.054 INFO:teuthology.orchestra.run.smithi112.stdout:Get:109 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.1.1-298-gb53b6d3e-1jammy [794 kB] 2024-09-11T06:12:20.072 INFO:teuthology.orchestra.run.smithi112.stdout:Get:110 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.1.1-298-gb53b6d3e-1jammy [152 kB] 2024-09-11T06:12:20.074 INFO:teuthology.orchestra.run.smithi112.stdout:Get:111 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.1.1-298-gb53b6d3e-1jammy [2,284 kB] 2024-09-11T06:12:20.141 INFO:teuthology.orchestra.run.smithi112.stdout:Get:112 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.1.1-298-gb53b6d3e-1jammy [8,625 kB] 2024-09-11T06:12:20.429 INFO:teuthology.orchestra.run.smithi112.stdout:Get:113 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.1.1-298-gb53b6d3e-1jammy [14.2 kB] 2024-09-11T06:12:20.429 INFO:teuthology.orchestra.run.smithi112.stdout:Get:114 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.1.1-298-gb53b6d3e-1jammy [51.4 MB] 2024-09-11T06:12:20.769 INFO:teuthology.orchestra.run.smithi006.stdout:Get:106 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.1.1-298-gb53b6d3e-1jammy [14.1 kB] 2024-09-11T06:12:20.770 INFO:teuthology.orchestra.run.smithi006.stdout:Get:107 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.1.1-298-gb53b6d3e-1jammy [877 kB] 2024-09-11T06:12:20.788 INFO:teuthology.orchestra.run.smithi006.stdout:Get:108 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.1.1-298-gb53b6d3e-1jammy [2,462 kB] 2024-09-11T06:12:20.831 INFO:teuthology.orchestra.run.smithi006.stdout:Get:109 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.1.1-298-gb53b6d3e-1jammy [794 kB] 2024-09-11T06:12:20.841 INFO:teuthology.orchestra.run.smithi006.stdout:Get:110 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.1.1-298-gb53b6d3e-1jammy [152 kB] 2024-09-11T06:12:20.844 INFO:teuthology.orchestra.run.smithi006.stdout:Get:111 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.1.1-298-gb53b6d3e-1jammy [2,284 kB] 2024-09-11T06:12:20.927 INFO:teuthology.orchestra.run.smithi006.stdout:Get:112 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.1.1-298-gb53b6d3e-1jammy [8,625 kB] 2024-09-11T06:12:21.215 INFO:teuthology.orchestra.run.smithi006.stdout:Get:113 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.1.1-298-gb53b6d3e-1jammy [14.2 kB] 2024-09-11T06:12:21.216 INFO:teuthology.orchestra.run.smithi006.stdout:Get:114 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.1.1-298-gb53b6d3e-1jammy [51.4 MB] 2024-09-11T06:12:22.241 INFO:teuthology.orchestra.run.smithi112.stdout:Get:115 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.1.1-298-gb53b6d3e-1jammy [130 kB] 2024-09-11T06:12:22.244 INFO:teuthology.orchestra.run.smithi112.stdout:Get:116 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.1.1-298-gb53b6d3e-1jammy [39.0 kB] 2024-09-11T06:12:22.245 INFO:teuthology.orchestra.run.smithi112.stdout:Get:117 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.1.1-298-gb53b6d3e-1jammy [13.6 MB] 2024-09-11T06:12:22.589 INFO:teuthology.orchestra.run.smithi112.stdout:Get:118 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.1.1-298-gb53b6d3e-1jammy [91.5 kB] 2024-09-11T06:12:22.591 INFO:teuthology.orchestra.run.smithi112.stdout:Get:119 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 rbd-mirror amd64 19.1.1-298-gb53b6d3e-1jammy [3,140 kB] 2024-09-11T06:12:22.995 INFO:teuthology.orchestra.run.smithi006.stdout:Get:115 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.1.1-298-gb53b6d3e-1jammy [130 kB] 2024-09-11T06:12:22.997 INFO:teuthology.orchestra.run.smithi006.stdout:Get:116 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.1.1-298-gb53b6d3e-1jammy [39.0 kB] 2024-09-11T06:12:22.999 INFO:teuthology.orchestra.run.smithi006.stdout:Get:117 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.1.1-298-gb53b6d3e-1jammy [13.6 MB] 2024-09-11T06:12:23.189 INFO:teuthology.orchestra.run.smithi006.stdout:Get:118 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.1.1-298-gb53b6d3e-1jammy [91.5 kB] 2024-09-11T06:12:23.191 INFO:teuthology.orchestra.run.smithi006.stdout:Get:119 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 rbd-mirror amd64 19.1.1-298-gb53b6d3e-1jammy [3,140 kB] 2024-09-11T06:12:23.258 INFO:teuthology.orchestra.run.smithi112.stdout:Fetched 184 MB in 5s (35.5 MB/s) 2024-09-11T06:12:23.425 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-09-11T06:12:23.526 INFO:teuthology.orchestra.run.smithi112.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165988 files and directories currently installed.) 2024-09-11T06:12:23.531 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-09-11T06:12:23.815 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-11T06:12:23.860 INFO:teuthology.orchestra.run.smithi006.stdout:Fetched 184 MB in 6s (32.6 MB/s) 2024-09-11T06:12:23.995 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-09-11T06:12:24.096 INFO:teuthology.orchestra.run.smithi006.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165988 files and directories currently installed.) 2024-09-11T06:12:24.102 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-09-11T06:12:24.168 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-11T06:12:24.169 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-11T06:12:24.195 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-09-11T06:12:24.235 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-11T06:12:24.547 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-11T06:12:24.573 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-09-11T06:12:24.580 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-11T06:12:24.607 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-09-11T06:12:24.613 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-11T06:12:24.646 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-11T06:12:24.984 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-11T06:12:24.991 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-11T06:12:25.010 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-11T06:12:25.018 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-09-11T06:12:25.058 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-11T06:12:25.059 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-11T06:12:25.462 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-11T06:12:25.486 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-11T06:12:25.488 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-11T06:12:25.514 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-11T06:12:25.528 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-11T06:12:25.561 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-11T06:12:25.857 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-11T06:12:25.882 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-11T06:12:25.915 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-11T06:12:25.981 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-11T06:12:26.008 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-11T06:12:26.048 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-11T06:12:26.209 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-09-11T06:12:26.224 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-09-11T06:12:26.267 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-11T06:12:26.409 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-11T06:12:26.436 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-11T06:12:26.484 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-11T06:12:26.595 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package librados2. 2024-09-11T06:12:26.622 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../007-librados2_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T06:12:26.662 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking librados2 (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:12:26.820 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-09-11T06:12:26.836 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-09-11T06:12:26.870 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-11T06:12:27.173 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package librbd1. 2024-09-11T06:12:27.198 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package librados2. 2024-09-11T06:12:27.199 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../008-librbd1_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T06:12:27.225 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../007-librados2_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T06:12:27.240 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking librbd1 (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:12:27.265 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking librados2 (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:12:27.642 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package libcephfs2. 2024-09-11T06:12:27.657 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../009-libcephfs2_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T06:12:27.692 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking libcephfs2 (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:12:27.717 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package librbd1. 2024-09-11T06:12:27.744 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../008-librbd1_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T06:12:27.784 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking librbd1 (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:12:27.961 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-rados. 2024-09-11T06:12:27.987 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../010-python3-rados_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T06:12:28.028 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-rados (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:12:28.228 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package libcephfs2. 2024-09-11T06:12:28.255 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../009-libcephfs2_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T06:12:28.295 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking libcephfs2 (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:12:28.356 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-11T06:12:28.382 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.1.1-298-gb53b6d3e-1jammy_all.deb ... 2024-09-11T06:12:28.431 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-ceph-argparse (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:12:28.564 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-rados. 2024-09-11T06:12:28.580 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../010-python3-rados_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T06:12:28.614 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-rados (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:12:28.875 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-cephfs. 2024-09-11T06:12:28.902 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../012-python3-cephfs_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T06:12:28.917 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-11T06:12:28.933 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-cephfs (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:12:28.943 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.1.1-298-gb53b6d3e-1jammy_all.deb ... 2024-09-11T06:12:28.983 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-ceph-argparse (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:12:29.228 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-11T06:12:29.236 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-cephfs. 2024-09-11T06:12:29.254 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../013-python3-ceph-common_19.1.1-298-gb53b6d3e-1jammy_all.deb ... 2024-09-11T06:12:29.262 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../012-python3-cephfs_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T06:12:29.294 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-ceph-common (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:12:29.302 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-cephfs (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:12:29.613 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-11T06:12:29.614 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-11T06:12:29.640 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../013-python3-ceph-common_19.1.1-298-gb53b6d3e-1jammy_all.deb ... 2024-09-11T06:12:29.640 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-09-11T06:12:29.680 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-ceph-common (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:12:29.681 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-11T06:12:29.983 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-prettytable. 2024-09-11T06:12:30.010 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-09-11T06:12:30.017 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-11T06:12:30.043 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-09-11T06:12:30.050 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-09-11T06:12:30.083 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-11T06:12:30.319 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-rbd. 2024-09-11T06:12:30.334 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../016-python3-rbd_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T06:12:30.361 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-prettytable. 2024-09-11T06:12:30.369 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-rbd (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:12:30.388 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-09-11T06:12:30.419 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-09-11T06:12:30.688 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-rbd. 2024-09-11T06:12:30.697 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-11T06:12:30.704 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../016-python3-rbd_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T06:12:30.724 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-09-11T06:12:30.747 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-rbd (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:12:30.763 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-11T06:12:31.050 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-09-11T06:12:31.076 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-09-11T06:12:31.091 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-11T06:12:31.116 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-09-11T06:12:31.118 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-09-11T06:12:31.158 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-11T06:12:31.427 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-09-11T06:12:31.443 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-09-11T06:12:31.454 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-09-11T06:12:31.470 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-09-11T06:12:31.494 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-11T06:12:31.510 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-09-11T06:12:31.780 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package lua5.1. 2024-09-11T06:12:31.788 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-09-11T06:12:31.796 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-09-11T06:12:31.804 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-09-11T06:12:31.830 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-09-11T06:12:31.838 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-11T06:12:32.142 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package lua-any. 2024-09-11T06:12:32.166 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package lua5.1. 2024-09-11T06:12:32.168 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-09-11T06:12:32.193 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-09-11T06:12:32.208 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking lua-any (27ubuntu1) ... 2024-09-11T06:12:32.233 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-09-11T06:12:32.461 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package zip. 2024-09-11T06:12:32.487 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-09-11T06:12:32.536 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking zip (3.0-12build2) ... 2024-09-11T06:12:32.553 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package lua-any. 2024-09-11T06:12:32.579 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-09-11T06:12:32.619 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking lua-any (27ubuntu1) ... 2024-09-11T06:12:32.846 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package zip. 2024-09-11T06:12:32.847 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package unzip. 2024-09-11T06:12:32.862 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-09-11T06:12:32.874 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-09-11T06:12:32.896 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking zip (3.0-12build2) ... 2024-09-11T06:12:32.914 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-09-11T06:12:33.217 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package unzip. 2024-09-11T06:12:33.244 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-09-11T06:12:33.268 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package luarocks. 2024-09-11T06:12:33.283 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-09-11T06:12:33.294 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-09-11T06:12:33.334 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-09-11T06:12:33.679 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package luarocks. 2024-09-11T06:12:33.705 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-09-11T06:12:33.745 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-09-11T06:12:33.751 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package librgw2. 2024-09-11T06:12:33.778 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../025-librgw2_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T06:12:33.817 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking librgw2 (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:12:34.603 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-rgw. 2024-09-11T06:12:34.619 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../026-python3-rgw_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T06:12:34.661 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-rgw (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:12:34.794 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package librgw2. 2024-09-11T06:12:34.821 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../025-librgw2_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T06:12:34.861 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking librgw2 (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:12:35.006 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package liboath0:amd64. 2024-09-11T06:12:35.033 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-09-11T06:12:35.072 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-09-11T06:12:35.342 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package libradosstriper1. 2024-09-11T06:12:35.369 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../028-libradosstriper1_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T06:12:35.371 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-rgw. 2024-09-11T06:12:35.387 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../026-python3-rgw_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T06:12:35.400 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking libradosstriper1 (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:12:35.421 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-rgw (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:12:35.758 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package liboath0:amd64. 2024-09-11T06:12:35.784 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-09-11T06:12:35.824 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-09-11T06:12:35.854 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package ceph-common. 2024-09-11T06:12:35.881 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../029-ceph-common_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T06:12:35.920 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking ceph-common (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:12:36.135 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package libradosstriper1. 2024-09-11T06:12:36.162 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../028-libradosstriper1_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T06:12:36.202 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking libradosstriper1 (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:12:36.589 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package ceph-common. 2024-09-11T06:12:36.615 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../029-ceph-common_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T06:12:36.655 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking ceph-common (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:12:37.401 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package ceph-base. 2024-09-11T06:12:37.427 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../030-ceph-base_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T06:12:37.477 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking ceph-base (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:12:37.996 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-09-11T06:12:38.023 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-09-11T06:12:38.055 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-09-11T06:12:38.227 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package ceph-base. 2024-09-11T06:12:38.254 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../030-ceph-base_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T06:12:38.303 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking ceph-base (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:12:38.366 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-cheroot. 2024-09-11T06:12:38.392 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-09-11T06:12:38.433 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-11T06:12:38.752 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-09-11T06:12:38.778 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-09-11T06:12:38.819 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-09-11T06:12:38.915 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-09-11T06:12:38.942 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-09-11T06:12:38.981 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-09-11T06:12:39.113 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-jaraco.text. 2024-09-11T06:12:39.139 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-09-11T06:12:39.180 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-09-11T06:12:39.284 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-cheroot. 2024-09-11T06:12:39.311 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-09-11T06:12:39.351 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-11T06:12:39.424 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-09-11T06:12:39.440 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-09-11T06:12:39.474 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-09-11T06:12:39.637 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-09-11T06:12:39.664 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-09-11T06:12:39.695 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-09-11T06:12:39.777 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-tempora. 2024-09-11T06:12:39.803 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-09-11T06:12:39.844 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-09-11T06:12:39.981 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-jaraco.text. 2024-09-11T06:12:40.008 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-09-11T06:12:40.048 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-09-11T06:12:40.163 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-portend. 2024-09-11T06:12:40.189 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-09-11T06:12:40.230 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-09-11T06:12:40.351 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-09-11T06:12:40.378 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-09-11T06:12:40.417 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-09-11T06:12:40.508 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-09-11T06:12:40.534 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-09-11T06:12:40.574 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-09-11T06:12:40.721 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-tempora. 2024-09-11T06:12:40.747 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-09-11T06:12:40.787 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-09-11T06:12:40.877 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-11T06:12:40.903 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-09-11T06:12:40.944 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-09-11T06:12:41.039 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-portend. 2024-09-11T06:12:41.066 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-09-11T06:12:41.098 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-09-11T06:12:41.255 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-natsort. 2024-09-11T06:12:41.282 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-09-11T06:12:41.314 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-09-11T06:12:41.409 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-09-11T06:12:41.436 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-09-11T06:12:41.475 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-09-11T06:12:41.625 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-logutils. 2024-09-11T06:12:41.651 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-09-11T06:12:41.692 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-09-11T06:12:41.795 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-11T06:12:41.822 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-09-11T06:12:41.870 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-09-11T06:12:41.995 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-11T06:12:42.021 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-09-11T06:12:42.061 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-09-11T06:12:42.224 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-natsort. 2024-09-11T06:12:42.251 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-09-11T06:12:42.290 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-09-11T06:12:42.364 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-mako. 2024-09-11T06:12:42.390 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-09-11T06:12:42.430 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-11T06:12:42.585 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-logutils. 2024-09-11T06:12:42.612 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-09-11T06:12:42.651 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-09-11T06:12:42.742 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-11T06:12:42.768 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-11T06:12:42.800 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-11T06:12:42.896 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-11T06:12:42.911 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-09-11T06:12:42.954 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-09-11T06:12:43.078 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-11T06:12:43.105 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-09-11T06:12:43.145 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-09-11T06:12:43.257 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-mako. 2024-09-11T06:12:43.284 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-09-11T06:12:43.323 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-11T06:12:43.447 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-webob. 2024-09-11T06:12:43.474 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1ubuntu0.1_all.deb ... 2024-09-11T06:12:43.514 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-11T06:12:43.660 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-11T06:12:43.687 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-11T06:12:43.726 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-11T06:12:43.851 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-waitress. 2024-09-11T06:12:43.877 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-09-11T06:12:43.945 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-11T06:12:44.029 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-11T06:12:44.056 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-09-11T06:12:44.096 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-09-11T06:12:44.262 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-tempita. 2024-09-11T06:12:44.288 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-09-11T06:12:44.328 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-09-11T06:12:44.395 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-webob. 2024-09-11T06:12:44.423 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1ubuntu0.1_all.deb ... 2024-09-11T06:12:44.573 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-11T06:12:44.589 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-paste. 2024-09-11T06:12:44.616 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-09-11T06:12:44.648 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-09-11T06:12:44.901 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-waitress. 2024-09-11T06:12:44.929 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-09-11T06:12:44.943 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-11T06:12:44.958 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-09-11T06:12:44.971 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-11T06:12:45.001 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-09-11T06:12:45.279 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-tempita. 2024-09-11T06:12:45.304 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-11T06:12:45.306 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-09-11T06:12:45.330 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-09-11T06:12:45.346 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-09-11T06:12:45.370 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-09-11T06:12:45.665 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-paste. 2024-09-11T06:12:45.682 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-webtest. 2024-09-11T06:12:45.692 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-09-11T06:12:45.709 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-09-11T06:12:45.732 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-09-11T06:12:45.748 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-09-11T06:12:46.052 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-11T06:12:46.059 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-pecan. 2024-09-11T06:12:46.079 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-09-11T06:12:46.086 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-09-11T06:12:46.118 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-09-11T06:12:46.126 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-09-11T06:12:46.421 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-11T06:12:46.448 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-09-11T06:12:46.487 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-09-11T06:12:46.497 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-certifi. 2024-09-11T06:12:46.524 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-09-11T06:12:46.564 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-09-11T06:12:46.815 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-webtest. 2024-09-11T06:12:46.842 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-09-11T06:12:46.882 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-09-11T06:12:46.900 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-idna. 2024-09-11T06:12:46.928 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../055-python3-idna_3.3-1ubuntu0.1_all.deb ... 2024-09-11T06:12:46.967 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-idna (3.3-1ubuntu0.1) ... 2024-09-11T06:12:47.143 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-pecan. 2024-09-11T06:12:47.171 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-09-11T06:12:47.201 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-09-11T06:12:47.287 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-urllib3. 2024-09-11T06:12:47.313 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-09-11T06:12:47.353 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-11T06:12:47.531 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-certifi. 2024-09-11T06:12:47.558 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-09-11T06:12:47.589 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-09-11T06:12:47.681 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-requests. 2024-09-11T06:12:47.708 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-09-11T06:12:47.748 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-11T06:12:47.926 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-idna. 2024-09-11T06:12:47.942 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../055-python3-idna_3.3-1ubuntu0.1_all.deb ... 2024-09-11T06:12:47.984 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-idna (3.3-1ubuntu0.1) ... 2024-09-11T06:12:48.068 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-11T06:12:48.095 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.2_all.deb ... 2024-09-11T06:12:48.134 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-11T06:12:48.295 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-urllib3. 2024-09-11T06:12:48.322 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-09-11T06:12:48.361 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-11T06:12:48.445 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-11T06:12:48.461 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.1.1-298-gb53b6d3e-1jammy_all.deb ... 2024-09-11T06:12:48.504 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking ceph-mgr-modules-core (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:12:48.690 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-requests. 2024-09-11T06:12:48.717 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-09-11T06:12:48.756 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-11T06:12:48.850 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-11T06:12:48.878 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T06:12:48.908 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking libsqlite3-mod-ceph (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:12:49.036 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-11T06:12:49.063 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.2_all.deb ... 2024-09-11T06:12:49.094 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-11T06:12:49.228 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-11T06:12:49.255 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-09-11T06:12:49.294 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-09-11T06:12:49.397 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-11T06:12:49.413 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.1.1-298-gb53b6d3e-1jammy_all.deb ... 2024-09-11T06:12:49.455 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking ceph-mgr-modules-core (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:12:49.597 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-openssl. 2024-09-11T06:12:49.624 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-09-11T06:12:49.664 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-09-11T06:12:50.226 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package ceph-mgr. 2024-09-11T06:12:50.253 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../063-ceph-mgr_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T06:12:50.292 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking ceph-mgr (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:12:50.720 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-11T06:12:50.747 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T06:12:50.786 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking libsqlite3-mod-ceph (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:12:51.106 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-11T06:12:51.132 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-09-11T06:12:51.172 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-09-11T06:12:51.362 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package ceph-mon. 2024-09-11T06:12:51.390 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../064-ceph-mon_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T06:12:51.475 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-openssl. 2024-09-11T06:12:51.502 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-09-11T06:12:51.541 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-09-11T06:12:51.545 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking ceph-mon (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:12:51.869 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package ceph-mgr. 2024-09-11T06:12:51.896 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../063-ceph-mgr_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T06:12:51.936 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking ceph-mgr (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:12:52.249 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package ceph-osd. 2024-09-11T06:12:52.276 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../065-ceph-osd_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T06:12:52.314 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package ceph-mon. 2024-09-11T06:12:52.324 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking ceph-osd (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:12:52.341 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../064-ceph-mon_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T06:12:52.380 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking ceph-mon (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:12:52.984 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package ceph-osd. 2024-09-11T06:12:53.011 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../065-ceph-osd_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T06:12:53.050 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking ceph-osd (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:12:53.360 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package ceph. 2024-09-11T06:12:53.376 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../066-ceph_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T06:12:53.410 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking ceph (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:12:53.738 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package ceph-fuse. 2024-09-11T06:12:53.765 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../067-ceph-fuse_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T06:12:53.804 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking ceph-fuse (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:12:54.183 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package ceph-mds. 2024-09-11T06:12:54.210 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../068-ceph-mds_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T06:12:54.220 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package ceph. 2024-09-11T06:12:54.236 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../066-ceph_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T06:12:54.249 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking ceph-mds (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:12:54.278 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking ceph (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:12:54.531 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package ceph-fuse. 2024-09-11T06:12:54.547 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../067-ceph-fuse_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T06:12:54.581 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking ceph-fuse (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:12:54.627 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package cephadm. 2024-09-11T06:12:54.654 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../069-cephadm_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T06:12:54.694 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking cephadm (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:12:54.951 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package ceph-mds. 2024-09-11T06:12:54.955 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-asyncssh. 2024-09-11T06:12:54.978 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../068-ceph-mds_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T06:12:54.981 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-09-11T06:12:55.017 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking ceph-mds (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:12:55.021 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-09-11T06:12:55.366 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python-babel-localedata. 2024-09-11T06:12:55.381 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-09-11T06:12:55.420 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package cephadm. 2024-09-11T06:12:55.424 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-11T06:12:55.447 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../069-cephadm_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T06:12:55.486 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking cephadm (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:12:55.781 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-asyncssh. 2024-09-11T06:12:55.797 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-09-11T06:12:55.831 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-09-11T06:12:56.192 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python-babel-localedata. 2024-09-11T06:12:56.208 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-09-11T06:12:56.250 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-11T06:12:56.546 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-babel. 2024-09-11T06:12:56.573 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-09-11T06:12:56.613 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-09-11T06:12:56.966 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-jinja2. 2024-09-11T06:12:56.992 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.2_all.deb ... 2024-09-11T06:12:57.032 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-11T06:12:57.272 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-babel. 2024-09-11T06:12:57.299 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-09-11T06:12:57.338 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-09-11T06:12:57.385 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-11T06:12:57.412 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.1.1-298-gb53b6d3e-1jammy_all.deb ... 2024-09-11T06:12:57.452 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking ceph-mgr-cephadm (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:12:57.641 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-jinja2. 2024-09-11T06:12:57.668 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.2_all.deb ... 2024-09-11T06:12:57.707 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-11T06:12:57.805 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-11T06:12:57.832 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-09-11T06:12:57.872 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-11T06:12:58.010 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-11T06:12:58.026 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.1.1-298-gb53b6d3e-1jammy_all.deb ... 2024-09-11T06:12:58.069 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking ceph-mgr-cephadm (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:12:58.141 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-routes. 2024-09-11T06:12:58.168 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-09-11T06:12:58.208 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-09-11T06:12:58.355 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-11T06:12:58.371 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-09-11T06:12:58.413 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-11T06:12:58.511 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-11T06:12:58.538 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.1.1-298-gb53b6d3e-1jammy_all.deb ... 2024-09-11T06:12:58.577 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking ceph-mgr-dashboard (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:12:58.691 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-routes. 2024-09-11T06:12:58.719 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-09-11T06:12:58.758 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-09-11T06:12:59.052 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-11T06:12:59.080 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.1.1-298-gb53b6d3e-1jammy_all.deb ... 2024-09-11T06:12:59.119 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking ceph-mgr-dashboard (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:12:59.816 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-09-11T06:12:59.831 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-09-11T06:12:59.866 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-11T06:13:00.396 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-joblib. 2024-09-11T06:13:00.424 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-09-11T06:13:00.471 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-09-11T06:13:00.532 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-09-11T06:13:00.549 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-09-11T06:13:00.590 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-11T06:13:00.850 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-09-11T06:13:00.877 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-09-11T06:13:00.916 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-09-11T06:13:01.120 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-joblib. 2024-09-11T06:13:01.146 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-09-11T06:13:01.185 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-09-11T06:13:01.227 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-sklearn. 2024-09-11T06:13:01.255 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-09-11T06:13:01.294 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-11T06:13:01.505 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-09-11T06:13:01.522 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-09-11T06:13:01.555 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-09-11T06:13:01.850 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-sklearn. 2024-09-11T06:13:01.877 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-09-11T06:13:01.916 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-11T06:13:02.036 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-11T06:13:02.063 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.1.1-298-gb53b6d3e-1jammy_all.deb ... 2024-09-11T06:13:02.196 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking ceph-mgr-diskprediction-local (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:13:02.641 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-11T06:13:02.658 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.1.1-298-gb53b6d3e-1jammy_all.deb ... 2024-09-11T06:13:02.785 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking ceph-mgr-diskprediction-local (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:13:03.216 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-cachetools. 2024-09-11T06:13:03.243 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-09-11T06:13:03.283 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-09-11T06:13:03.594 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-11T06:13:03.621 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-09-11T06:13:03.661 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-09-11T06:13:03.747 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-cachetools. 2024-09-11T06:13:03.774 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-09-11T06:13:03.813 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-09-11T06:13:04.006 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-11T06:13:04.033 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-09-11T06:13:04.072 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-09-11T06:13:04.116 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-11T06:13:04.144 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-09-11T06:13:04.182 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-09-11T06:13:04.384 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-rsa. 2024-09-11T06:13:04.411 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-09-11T06:13:04.450 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-rsa (4.8-1) ... 2024-09-11T06:13:04.502 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-11T06:13:04.530 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-09-11T06:13:04.569 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-09-11T06:13:04.737 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-google-auth. 2024-09-11T06:13:04.764 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-09-11T06:13:04.795 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-09-11T06:13:04.880 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-rsa. 2024-09-11T06:13:04.908 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-09-11T06:13:04.947 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-rsa (4.8-1) ... 2024-09-11T06:13:05.115 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-11T06:13:05.142 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-09-11T06:13:05.181 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-11T06:13:05.244 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-google-auth. 2024-09-11T06:13:05.270 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-09-11T06:13:05.308 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-09-11T06:13:05.626 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-websocket. 2024-09-11T06:13:05.629 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-11T06:13:05.654 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-09-11T06:13:05.656 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-09-11T06:13:05.695 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-11T06:13:05.768 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-09-11T06:13:06.007 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-websocket. 2024-09-11T06:13:06.034 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-09-11T06:13:06.063 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-11T06:13:06.073 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-09-11T06:13:06.090 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-09-11T06:13:06.121 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-11T06:13:06.393 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-11T06:13:06.420 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-09-11T06:13:06.459 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-11T06:13:06.835 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-11T06:13:06.863 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.1.1-298-gb53b6d3e-1jammy_all.deb ... 2024-09-11T06:13:06.901 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking ceph-mgr-k8sevents (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:13:07.190 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-11T06:13:07.206 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.1.1-298-gb53b6d3e-1jammy_all.deb ... 2024-09-11T06:13:07.221 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-11T06:13:07.248 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking ceph-mgr-k8sevents (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:13:07.248 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-09-11T06:13:07.287 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-11T06:13:07.549 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-11T06:13:07.576 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-09-11T06:13:07.615 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-11T06:13:07.617 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-11T06:13:07.645 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-09-11T06:13:07.684 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-11T06:13:07.885 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package jq. 2024-09-11T06:13:07.912 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-09-11T06:13:07.951 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-09-11T06:13:07.987 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-11T06:13:08.015 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-09-11T06:13:08.045 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-11T06:13:08.187 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package socat. 2024-09-11T06:13:08.203 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-09-11T06:13:08.245 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-09-11T06:13:08.306 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package jq. 2024-09-11T06:13:08.322 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-09-11T06:13:08.364 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-09-11T06:13:08.557 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package xmlstarlet. 2024-09-11T06:13:08.584 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-09-11T06:13:08.600 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package socat. 2024-09-11T06:13:08.616 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-09-11T06:13:08.624 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-09-11T06:13:08.678 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-09-11T06:13:08.995 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package xmlstarlet. 2024-09-11T06:13:08.995 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package ceph-test. 2024-09-11T06:13:09.011 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-09-11T06:13:09.022 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../097-ceph-test_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T06:13:09.061 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking ceph-test (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:13:09.062 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-09-11T06:13:09.432 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package ceph-test. 2024-09-11T06:13:09.460 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../097-ceph-test_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T06:13:09.499 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking ceph-test (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:13:11.357 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package ceph-volume. 2024-09-11T06:13:11.373 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../098-ceph-volume_19.1.1-298-gb53b6d3e-1jammy_all.deb ... 2024-09-11T06:13:11.407 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking ceph-volume (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:13:11.710 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-11T06:13:11.726 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../099-libcephfs-dev_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T06:13:11.769 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package ceph-volume. 2024-09-11T06:13:11.777 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking libcephfs-dev (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:13:11.800 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../098-ceph-volume_19.1.1-298-gb53b6d3e-1jammy_all.deb ... 2024-09-11T06:13:11.836 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking ceph-volume (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:13:12.088 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package lua-socket:amd64. 2024-09-11T06:13:12.116 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-09-11T06:13:12.146 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-11T06:13:12.148 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-11T06:13:12.165 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../099-libcephfs-dev_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T06:13:12.206 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking libcephfs-dev (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:13:12.468 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package lua-sec:amd64. 2024-09-11T06:13:12.495 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-09-11T06:13:12.534 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-09-11T06:13:12.542 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package lua-socket:amd64. 2024-09-11T06:13:12.570 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-09-11T06:13:12.609 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-11T06:13:12.913 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package nvme-cli. 2024-09-11T06:13:12.930 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package lua-sec:amd64. 2024-09-11T06:13:12.941 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.2_amd64.deb ... 2024-09-11T06:13:12.958 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-09-11T06:13:12.980 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking nvme-cli (1.16-3ubuntu0.2) ... 2024-09-11T06:13:12.997 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-09-11T06:13:13.376 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package nvme-cli. 2024-09-11T06:13:13.404 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.2_amd64.deb ... 2024-09-11T06:13:13.418 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package pkg-config. 2024-09-11T06:13:13.442 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking nvme-cli (1.16-3ubuntu0.2) ... 2024-09-11T06:13:13.446 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-09-11T06:13:13.493 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-09-11T06:13:13.738 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-09-11T06:13:13.754 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-09-11T06:13:13.787 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-09-11T06:13:13.864 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package pkg-config. 2024-09-11T06:13:13.892 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-09-11T06:13:13.930 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-09-11T06:13:14.191 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-09-11T06:13:14.209 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-09-11T06:13:14.215 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-iniconfig. 2024-09-11T06:13:14.231 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-09-11T06:13:14.252 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-09-11T06:13:14.265 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-09-11T06:13:14.535 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-pastescript. 2024-09-11T06:13:14.562 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-09-11T06:13:14.601 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-11T06:13:14.686 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-iniconfig. 2024-09-11T06:13:14.714 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-09-11T06:13:14.752 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-09-11T06:13:14.922 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-pluggy. 2024-09-11T06:13:14.949 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-09-11T06:13:14.988 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-09-11T06:13:15.022 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-pastescript. 2024-09-11T06:13:15.050 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-09-11T06:13:15.080 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-11T06:13:15.299 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-psutil. 2024-09-11T06:13:15.327 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-09-11T06:13:15.365 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-09-11T06:13:15.383 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-pluggy. 2024-09-11T06:13:15.412 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-09-11T06:13:15.450 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-09-11T06:13:15.677 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-py. 2024-09-11T06:13:15.704 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-09-11T06:13:15.743 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-py (1.10.0-1) ... 2024-09-11T06:13:16.028 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-psutil. 2024-09-11T06:13:16.056 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-09-11T06:13:16.072 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-pygments. 2024-09-11T06:13:16.094 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-09-11T06:13:16.099 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-09-11T06:13:16.138 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-09-11T06:13:16.456 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-py. 2024-09-11T06:13:16.484 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-09-11T06:13:16.522 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-py (1.10.0-1) ... 2024-09-11T06:13:16.601 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-11T06:13:16.628 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-09-11T06:13:16.667 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-09-11T06:13:16.833 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-pygments. 2024-09-11T06:13:16.850 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-09-11T06:13:16.892 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-09-11T06:13:16.945 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-toml. 2024-09-11T06:13:16.973 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-09-11T06:13:17.003 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-09-11T06:13:17.289 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-pytest. 2024-09-11T06:13:17.317 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-09-11T06:13:17.321 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-11T06:13:17.349 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-09-11T06:13:17.356 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-09-11T06:13:17.387 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-09-11T06:13:17.667 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-simplejson. 2024-09-11T06:13:17.683 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-09-11T06:13:17.707 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-toml. 2024-09-11T06:13:17.717 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-09-11T06:13:17.735 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-09-11T06:13:17.773 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-09-11T06:13:17.945 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-11T06:13:17.958 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-09-11T06:13:17.986 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-09-11T06:13:18.077 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-pytest. 2024-09-11T06:13:18.104 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-09-11T06:13:18.142 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-09-11T06:13:18.505 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package python3-simplejson. 2024-09-11T06:13:18.533 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-09-11T06:13:18.571 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-09-11T06:13:18.733 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package radosgw. 2024-09-11T06:13:18.760 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../116-radosgw_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T06:13:18.799 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking radosgw (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:13:18.840 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-11T06:13:18.868 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-09-11T06:13:18.906 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-09-11T06:13:19.585 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package rbd-fuse. 2024-09-11T06:13:19.601 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../117-rbd-fuse_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T06:13:19.643 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking rbd-fuse (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:13:19.652 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package radosgw. 2024-09-11T06:13:19.681 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../116-radosgw_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T06:13:19.718 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking radosgw (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:13:19.972 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package rbd-mirror. 2024-09-11T06:13:20.000 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../118-rbd-mirror_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T06:13:20.038 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking rbd-mirror (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:13:20.517 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-09-11T06:13:20.663 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package rbd-fuse. 2024-09-11T06:13:20.691 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../117-rbd-fuse_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T06:13:20.730 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking rbd-fuse (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:13:20.777 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-11T06:13:20.911 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up nvme-cli (1.16-3ubuntu0.2) ... 2024-09-11T06:13:21.100 INFO:teuthology.orchestra.run.smithi006.stdout:Selecting previously unselected package rbd-mirror. 2024-09-11T06:13:21.128 INFO:teuthology.orchestra.run.smithi006.stdout:Preparing to unpack .../118-rbd-mirror_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T06:13:21.166 INFO:teuthology.orchestra.run.smithi006.stdout:Unpacking rbd-mirror (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:13:21.177 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-09-11T06:13:21.513 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-09-11T06:13:21.621 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-09-11T06:13:21.928 INFO:teuthology.orchestra.run.smithi112.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-09-11T06:13:21.938 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-11T06:13:21.988 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up cephadm (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:13:22.055 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up nvme-cli (1.16-3ubuntu0.2) ... 2024-09-11T06:13:22.303 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-09-11T06:13:22.496 INFO:teuthology.orchestra.run.smithi112.stdout:Adding system user cephadm....done 2024-09-11T06:13:22.648 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-09-11T06:13:22.652 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-11T06:13:22.937 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-09-11T06:13:23.054 INFO:teuthology.orchestra.run.smithi006.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-09-11T06:13:23.107 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up cephadm (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:13:23.188 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-09-11T06:13:23.314 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-09-11T06:13:23.539 INFO:teuthology.orchestra.run.smithi006.stdout:Adding system user cephadm....done 2024-09-11T06:13:23.591 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-11T06:13:23.671 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-11T06:13:23.850 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-09-11T06:13:23.948 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-09-11T06:13:23.976 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-py (1.10.0-1) ... 2024-09-11T06:13:24.182 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-09-11T06:13:24.277 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-09-11T06:13:24.291 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-09-11T06:13:24.534 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-11T06:13:24.647 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-09-11T06:13:24.819 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-09-11T06:13:24.897 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-09-11T06:13:24.945 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-py (1.10.0-1) ... 2024-09-11T06:13:25.149 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-09-11T06:13:25.263 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-09-11T06:13:25.392 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-09-11T06:13:25.632 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-09-11T06:13:25.643 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-09-11T06:13:25.883 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-09-11T06:13:25.905 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-09-11T06:13:26.137 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-ceph-argparse (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:13:26.159 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-09-11T06:13:26.394 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-09-11T06:13:26.413 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-11T06:13:26.539 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-11T06:13:26.653 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-09-11T06:13:26.665 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-09-11T06:13:26.816 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-09-11T06:13:26.930 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-09-11T06:13:27.075 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-09-11T06:13:27.181 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-ceph-argparse (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:13:27.462 INFO:teuthology.orchestra.run.smithi112.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-09-11T06:13:27.489 INFO:teuthology.orchestra.run.smithi112.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-09-11T06:13:27.515 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-11T06:13:27.554 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-11T06:13:27.641 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-11T06:13:27.671 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-09-11T06:13:27.742 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-09-11T06:13:27.843 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-09-11T06:13:28.014 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-11T06:13:28.119 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-09-11T06:13:28.123 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-09-11T06:13:28.391 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-09-11T06:13:28.480 INFO:teuthology.orchestra.run.smithi006.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-09-11T06:13:28.509 INFO:teuthology.orchestra.run.smithi006.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-09-11T06:13:28.572 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-11T06:13:28.651 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-09-11T06:13:28.698 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-09-11T06:13:28.969 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up zip (3.0-12build2) ... 2024-09-11T06:13:29.074 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-11T06:13:29.095 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-09-11T06:13:29.200 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-09-11T06:13:29.468 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-09-11T06:13:29.727 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-09-11T06:13:29.855 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-09-11T06:13:30.037 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up zip (3.0-12build2) ... 2024-09-11T06:13:30.098 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-09-11T06:13:30.163 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-09-11T06:13:30.224 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-09-11T06:13:30.350 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-11T06:13:30.643 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-09-11T06:13:30.857 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-09-11T06:13:31.019 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-09-11T06:13:31.124 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-09-11T06:13:31.262 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-09-11T06:13:31.533 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-09-11T06:13:31.630 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-11T06:13:31.676 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-11T06:13:31.923 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-11T06:13:32.010 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-09-11T06:13:32.291 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-09-11T06:13:32.436 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-09-11T06:13:32.542 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-09-11T06:13:32.668 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-ceph-common (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:13:32.704 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-09-11T06:13:32.969 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-idna (3.3-1ubuntu0.1) ... 2024-09-11T06:13:33.081 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-11T06:13:33.287 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-11T06:13:33.374 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-11T06:13:33.717 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-09-11T06:13:33.968 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-09-11T06:13:34.085 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-ceph-common (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:13:34.395 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-idna (3.3-1ubuntu0.1) ... 2024-09-11T06:13:34.493 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-09-11T06:13:34.721 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-11T06:13:34.753 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-11T06:13:34.862 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-toml (0.10.2-1) ... 2024-09-11T06:13:35.097 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-11T06:13:35.214 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-09-11T06:13:35.332 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-11T06:13:35.641 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-09-11T06:13:35.935 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-09-11T06:13:36.093 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-09-11T06:13:36.195 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-11T06:13:36.323 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-toml (0.10.2-1) ... 2024-09-11T06:13:36.344 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-11T06:13:36.460 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-09-11T06:13:36.582 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-11T06:13:36.683 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-09-11T06:13:36.784 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-11T06:13:37.086 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-09-11T06:13:37.155 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-09-11T06:13:37.345 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-09-11T06:13:37.389 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-09-11T06:13:37.596 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-11T06:13:37.648 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-09-11T06:13:37.722 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-09-11T06:13:37.900 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-11T06:13:38.041 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-09-11T06:13:38.151 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-09-11T06:13:38.300 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-09-11T06:13:38.410 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-11T06:13:38.528 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-09-11T06:13:38.559 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-09-11T06:13:38.802 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-11T06:13:38.804 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-11T06:13:38.922 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-11T06:13:39.062 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-09-11T06:13:39.206 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-11T06:13:39.329 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-11T06:13:39.480 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-09-11T06:13:39.516 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-09-11T06:13:39.775 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-11T06:13:39.790 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-11T06:13:39.901 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-09-11T06:13:39.908 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-11T06:13:40.046 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-09-11T06:13:40.133 INFO:teuthology.orchestra.run.smithi112.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-09-11T06:13:40.200 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-11T06:13:40.379 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-11T06:13:40.480 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-09-11T06:13:40.493 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-09-11T06:13:40.753 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-11T06:13:40.873 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-09-11T06:13:40.879 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-09-11T06:13:41.005 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-09-11T06:13:41.094 INFO:teuthology.orchestra.run.smithi006.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-09-11T06:13:41.141 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up lua-any (27ubuntu1) ... 2024-09-11T06:13:41.267 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-portend (3.0.0-1) ... 2024-09-11T06:13:41.331 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-11T06:13:41.457 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-09-11T06:13:41.534 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-11T06:13:41.660 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-11T06:13:41.851 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-09-11T06:13:41.986 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-09-11T06:13:42.093 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up lua-any (27ubuntu1) ... 2024-09-11T06:13:42.211 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-portend (3.0.0-1) ... 2024-09-11T06:13:42.288 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-11T06:13:42.470 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-11T06:13:42.588 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-11T06:13:42.589 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-09-11T06:13:42.715 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-rsa (4.8-1) ... 2024-09-11T06:13:42.906 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-09-11T06:13:43.008 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-09-11T06:13:43.215 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-11T06:13:43.309 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-11T06:13:43.533 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-09-11T06:13:43.634 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-rsa (4.8-1) ... 2024-09-11T06:13:43.679 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-09-11T06:13:43.894 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-09-11T06:13:44.064 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-11T06:13:44.170 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-11T06:13:44.340 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-09-11T06:13:44.523 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-09-11T06:13:44.676 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-11T06:13:44.802 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up librados2 (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:13:44.917 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-11T06:13:44.937 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up libsqlite3-mod-ceph (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:13:45.038 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-09-11T06:13:45.202 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-09-11T06:13:45.349 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up libcephfs2 (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:13:45.475 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up libradosstriper1 (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:13:45.537 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-11T06:13:45.601 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-09-11T06:13:45.663 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up librados2 (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:13:45.789 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up libsqlite3-mod-ceph (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:13:45.886 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up librbd1 (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:13:45.924 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-09-11T06:13:45.998 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up ceph-mgr-modules-core (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:13:46.124 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up ceph-fuse (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:13:46.227 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up libcephfs2 (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:13:46.336 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-11T06:13:46.337 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-11T06:13:46.353 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up libradosstriper1 (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:13:46.479 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-09-11T06:13:46.756 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up librbd1 (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:13:46.858 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up libcephfs-dev (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:13:46.890 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up ceph-mgr-modules-core (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:13:46.985 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-rados (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:13:47.016 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up ceph-fuse (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:13:47.111 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up librgw2 (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:13:47.220 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-11T06:13:47.278 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-11T06:13:47.278 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-11T06:13:47.709 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up libcephfs-dev (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:13:47.810 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-rados (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:13:47.919 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up librgw2 (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:13:48.045 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-11T06:13:48.349 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-rbd (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:13:48.491 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up rbd-fuse (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:13:48.617 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-rgw (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:13:48.743 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-cephfs (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:13:48.869 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up ceph-common (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:13:49.316 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-rbd (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:13:49.354 INFO:teuthology.orchestra.run.smithi112.stdout:Adding group ceph....done 2024-09-11T06:13:49.459 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up rbd-fuse (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:13:49.585 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-rgw (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:13:49.711 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up python3-cephfs (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:13:49.738 INFO:teuthology.orchestra.run.smithi112.stdout:Adding system user ceph....done 2024-09-11T06:13:49.837 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up ceph-common (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:13:49.873 INFO:teuthology.orchestra.run.smithi112.stdout:Setting system user ceph properties....done 2024-09-11T06:13:49.893 INFO:teuthology.orchestra.run.smithi112.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-11T06:13:50.027 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-11T06:13:50.339 INFO:teuthology.orchestra.run.smithi006.stdout:Adding group ceph....done 2024-09-11T06:13:50.347 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-11T06:13:50.666 INFO:teuthology.orchestra.run.smithi006.stdout:Adding system user ceph....done 2024-09-11T06:13:50.787 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up ceph-test (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:13:50.833 INFO:teuthology.orchestra.run.smithi006.stdout:Setting system user ceph properties....done 2024-09-11T06:13:50.845 INFO:teuthology.orchestra.run.smithi006.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-11T06:13:50.888 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up radosgw (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:13:50.985 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-11T06:13:51.317 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-11T06:13:51.339 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-11T06:13:51.339 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-11T06:13:51.771 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up ceph-test (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:13:51.815 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up ceph-base (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:13:51.872 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up radosgw (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:13:52.168 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-11T06:13:52.368 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-11T06:13:52.368 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-11T06:13:52.659 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up ceph-mds (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:13:52.842 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up ceph-base (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:13:52.882 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-11T06:13:52.883 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-11T06:13:53.191 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-11T06:13:53.377 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up rbd-mirror (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:13:53.590 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target -> /lib/systemd/system/ceph-rbd-mirror.target. 2024-09-11T06:13:53.590 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target -> /lib/systemd/system/ceph-rbd-mirror.target. 2024-09-11T06:13:53.686 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up ceph-mds (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:13:53.918 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-11T06:13:53.919 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-11T06:13:54.078 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up ceph-mgr (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:13:54.278 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-11T06:13:54.278 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-11T06:13:54.438 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up rbd-mirror (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:13:54.680 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target -> /lib/systemd/system/ceph-rbd-mirror.target. 2024-09-11T06:13:54.680 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target -> /lib/systemd/system/ceph-rbd-mirror.target. 2024-09-11T06:13:54.746 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up ceph-osd (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:13:55.003 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-11T06:13:55.003 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-11T06:13:55.172 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up ceph-mgr (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:13:55.465 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up ceph-mgr-k8sevents (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:13:55.465 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-11T06:13:55.465 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-11T06:13:55.591 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up ceph-mgr-diskprediction-local (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:13:55.742 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up ceph-mon (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:13:55.953 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-11T06:13:55.953 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-11T06:13:55.958 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up ceph-osd (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:13:56.242 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-11T06:13:56.243 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-11T06:13:56.393 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up ceph-mgr-cephadm (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:13:56.544 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up ceph (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:13:56.691 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up ceph-mgr-dashboard (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:13:56.744 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up ceph-mgr-k8sevents (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:13:56.846 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up ceph-volume (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:13:56.869 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up ceph-mgr-diskprediction-local (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:13:57.020 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up ceph-mon (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:13:57.199 INFO:teuthology.orchestra.run.smithi112.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-09-11T06:13:57.308 INFO:teuthology.orchestra.run.smithi112.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-09-11T06:13:57.595 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-11T06:13:57.595 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-11T06:13:57.605 INFO:teuthology.orchestra.run.smithi112.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-09-11T06:13:58.081 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up ceph-mgr-cephadm (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:13:58.224 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up ceph (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:13:58.350 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up ceph-mgr-dashboard (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:13:58.501 INFO:teuthology.orchestra.run.smithi006.stdout:Setting up ceph-volume (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T06:13:58.732 INFO:teuthology.orchestra.run.smithi112.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-09-11T06:13:58.879 INFO:teuthology.orchestra.run.smithi006.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-09-11T06:13:58.988 INFO:teuthology.orchestra.run.smithi006.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-09-11T06:13:59.115 INFO:teuthology.orchestra.run.smithi006.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-09-11T06:13:59.372 INFO:teuthology.orchestra.run.smithi112.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-11T06:13:59.377 DEBUG:teuthology.parallel:result is None 2024-09-11T06:13:59.865 INFO:teuthology.orchestra.run.smithi006.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-09-11T06:14:00.526 INFO:teuthology.orchestra.run.smithi006.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-11T06:14:00.531 DEBUG:teuthology.parallel:result is None 2024-09-11T06:14:00.532 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc 2024-09-11T06:14:00.680 DEBUG:teuthology.orchestra.run.smithi006:> dpkg-query -W -f '${Version}' ceph 2024-09-11T06:14:00.700 INFO:teuthology.orchestra.run.smithi006.stdout:19.1.1-298-gb53b6d3e-1jammy 2024-09-11T06:14:00.701 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-298-gb53b6d3e-1jammy 2024-09-11T06:14:00.701 INFO:teuthology.task.install:The correct ceph version 19.1.1-298-gb53b6d3e-1jammy is installed. 2024-09-11T06:14:00.702 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc 2024-09-11T06:14:00.842 DEBUG:teuthology.orchestra.run.smithi112:> dpkg-query -W -f '${Version}' ceph 2024-09-11T06:14:00.862 INFO:teuthology.orchestra.run.smithi112.stdout:19.1.1-298-gb53b6d3e-1jammy 2024-09-11T06:14:00.862 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-298-gb53b6d3e-1jammy 2024-09-11T06:14:00.862 INFO:teuthology.task.install:The correct ceph version 19.1.1-298-gb53b6d3e-1jammy is installed. 2024-09-11T06:14:00.863 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-11T06:14:00.864 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-09-11T06:14:00.864 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-11T06:14:00.878 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-11T06:14:00.878 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-11T06:14:00.919 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-11T06:14:00.919 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-09-11T06:14:00.919 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/usr/bin/daemon-helper 2024-09-11T06:14:00.934 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-11T06:14:00.988 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-11T06:14:00.988 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/usr/bin/daemon-helper 2024-09-11T06:14:01.001 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-11T06:14:01.054 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-11T06:14:01.055 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-09-11T06:14:01.055 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-11T06:14:01.068 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-11T06:14:01.124 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-11T06:14:01.124 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-11T06:14:01.137 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-11T06:14:01.190 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-11T06:14:01.190 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-09-11T06:14:01.190 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/usr/bin/stdin-killer 2024-09-11T06:14:01.203 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-11T06:14:01.256 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-11T06:14:01.256 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/usr/bin/stdin-killer 2024-09-11T06:14:01.268 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-11T06:14:01.321 INFO:teuthology.run_tasks:Running task ceph... 2024-09-11T06:14:01.415 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-11T06:14:01.416 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 777 /var/log/ceph 2024-09-11T06:14:01.419 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod 777 /var/log/ceph 2024-09-11T06:14:01.432 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-11T06:14:01.433 DEBUG:teuthology.orchestra.run.smithi006:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-11T06:14:01.478 DEBUG:teuthology.orchestra.run.smithi112:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-11T06:14:01.490 INFO:tasks.ceph:Creating extra log directories... 2024-09-11T06:14:01.490 DEBUG:teuthology.orchestra.run.smithi006:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-11T06:14:01.534 DEBUG:teuthology.orchestra.run.smithi112:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-11T06:14:01.560 INFO:tasks.ceph:Creating ceph cluster cluster1... 2024-09-11T06:14:01.560 INFO:tasks.ceph:config {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'cluster1', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-11T06:14:01.561 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-09-10_21:16:03-rbd-squid-distro-default-smithi/7900605', 'branch': 'squid', 'description': 'rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-zlib supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-stress-workunit-min-compat-client-octopus}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7900605', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'meta': [{'desc': 'run two ceph clusters and install rbd-mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': '2 ceph clusters with 1 mon and 3 osds each'}, {'desc': 'run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon'}], 'name': 'teuthology-2024-09-10_21:16:03-rbd-squid-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'squid'}, 'ceph': {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'squid', 'sha1': 'daaf78de334647fc0c800f4ac495cc9475a26bae'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 100, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'], ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6']], 'seed': 6175, 'sha1': 'b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc', 'sleep_before_teardown': 0, 'subset': '88/128', 'suite': 'rbd', 'suite_branch': 'squid', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_daaf78de334647fc0c800f4ac495cc9475a26bae/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': 'daaf78de334647fc0c800f4ac495cc9475a26bae', 'targets': {'smithi006.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBO0WRxR505R28vAPOaiTE3OCExO5dRsDtnACcGHqGLEups3p4DMCG5xRwBug/cQh6QcewlLrbWsCJohOpKXaaQg=', 'smithi112.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJfTs0r1xAi00gNyDf/PIuEnNrqSpWYDR6gneJW9eH9P26G5v230dQe7xH8BPEZImnabz26XbGX7EdknHM19Ris='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': 'b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc'}}, {'ceph': {'cluster': 'cluster1', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc'}}, {'ceph': {'cluster': 'cluster2'}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror.4 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.5 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.6 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.4 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.5 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.6 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'workunit': {'clients': {'cluster1.client.mirror': ['rbd/rbd_mirror_stress.sh']}, 'env': {'CEPH_ARGS': '', 'RBD_MIRROR_MIN_COMPAT_CLIENT': 'octopus', 'RBD_MIRROR_USE_EXISTING_CLUSTER': '1'}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-10_21:16:03', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-11T06:14:01.561 DEBUG:teuthology.orchestra.run.smithi006:> install -d -m0755 -- /home/ubuntu/cephtest/cluster1.data 2024-09-11T06:14:01.590 DEBUG:teuthology.orchestra.run.smithi112:> install -d -m0755 -- /home/ubuntu/cephtest/cluster1.data 2024-09-11T06:14:01.596 DEBUG:teuthology.orchestra.run.smithi006:> sudo install -d -m0777 -- /var/run/ceph 2024-09-11T06:14:01.638 DEBUG:teuthology.orchestra.run.smithi112:> sudo install -d -m0777 -- /var/run/ceph 2024-09-11T06:14:01.655 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-09-11T06:14:01.655 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/scratch_devs of=/dev/stdout 2024-09-11T06:14:01.701 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-11T06:14:01.701 DEBUG:teuthology.orchestra.run.smithi006:> stat /dev/vg_nvme/lv_1 2024-09-11T06:14:01.749 INFO:teuthology.orchestra.run.smithi006.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-11T06:14:01.749 INFO:teuthology.orchestra.run.smithi006.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T06:14:01.750 INFO:teuthology.orchestra.run.smithi006.stdout:Device: 5h/5d Inode: 890 Links: 1 2024-09-11T06:14:01.750 INFO:teuthology.orchestra.run.smithi006.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T06:14:01.750 INFO:teuthology.orchestra.run.smithi006.stdout:Access: 2024-09-11 06:08:30.500882393 +0000 2024-09-11T06:14:01.750 INFO:teuthology.orchestra.run.smithi006.stdout:Modify: 2024-09-11 06:08:30.240884715 +0000 2024-09-11T06:14:01.750 INFO:teuthology.orchestra.run.smithi006.stdout:Change: 2024-09-11 06:08:30.240884715 +0000 2024-09-11T06:14:01.750 INFO:teuthology.orchestra.run.smithi006.stdout: Birth: - 2024-09-11T06:14:01.750 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-11T06:14:01.808 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records in 2024-09-11T06:14:01.808 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records out 2024-09-11T06:14:01.808 INFO:teuthology.orchestra.run.smithi006.stderr:512 bytes copied, 0.000448886 s, 1.1 MB/s 2024-09-11T06:14:01.810 DEBUG:teuthology.orchestra.run.smithi006:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-11T06:14:01.864 DEBUG:teuthology.orchestra.run.smithi006:> stat /dev/vg_nvme/lv_2 2024-09-11T06:14:01.914 INFO:teuthology.orchestra.run.smithi006.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-11T06:14:01.914 INFO:teuthology.orchestra.run.smithi006.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T06:14:01.914 INFO:teuthology.orchestra.run.smithi006.stdout:Device: 5h/5d Inode: 905 Links: 1 2024-09-11T06:14:01.914 INFO:teuthology.orchestra.run.smithi006.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T06:14:01.914 INFO:teuthology.orchestra.run.smithi006.stdout:Access: 2024-09-11 06:08:30.984878070 +0000 2024-09-11T06:14:01.914 INFO:teuthology.orchestra.run.smithi006.stdout:Modify: 2024-09-11 06:08:30.740880249 +0000 2024-09-11T06:14:01.915 INFO:teuthology.orchestra.run.smithi006.stdout:Change: 2024-09-11 06:08:30.740880249 +0000 2024-09-11T06:14:01.915 INFO:teuthology.orchestra.run.smithi006.stdout: Birth: - 2024-09-11T06:14:01.915 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-11T06:14:01.972 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records in 2024-09-11T06:14:01.972 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records out 2024-09-11T06:14:01.972 INFO:teuthology.orchestra.run.smithi006.stderr:512 bytes copied, 0.000430031 s, 1.2 MB/s 2024-09-11T06:14:01.974 DEBUG:teuthology.orchestra.run.smithi006:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-11T06:14:02.023 DEBUG:teuthology.orchestra.run.smithi006:> stat /dev/vg_nvme/lv_3 2024-09-11T06:14:02.069 INFO:teuthology.orchestra.run.smithi006.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-11T06:14:02.069 INFO:teuthology.orchestra.run.smithi006.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T06:14:02.070 INFO:teuthology.orchestra.run.smithi006.stdout:Device: 5h/5d Inode: 918 Links: 1 2024-09-11T06:14:02.070 INFO:teuthology.orchestra.run.smithi006.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T06:14:02.070 INFO:teuthology.orchestra.run.smithi006.stdout:Access: 2024-09-11 06:08:31.460873818 +0000 2024-09-11T06:14:02.070 INFO:teuthology.orchestra.run.smithi006.stdout:Modify: 2024-09-11 06:08:31.216875998 +0000 2024-09-11T06:14:02.070 INFO:teuthology.orchestra.run.smithi006.stdout:Change: 2024-09-11 06:08:31.216875998 +0000 2024-09-11T06:14:02.070 INFO:teuthology.orchestra.run.smithi006.stdout: Birth: - 2024-09-11T06:14:02.070 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-11T06:14:02.124 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records in 2024-09-11T06:14:02.124 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records out 2024-09-11T06:14:02.124 INFO:teuthology.orchestra.run.smithi006.stderr:512 bytes copied, 0.000347093 s, 1.5 MB/s 2024-09-11T06:14:02.125 DEBUG:teuthology.orchestra.run.smithi006:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-11T06:14:02.178 DEBUG:teuthology.orchestra.run.smithi006:> stat /dev/vg_nvme/lv_4 2024-09-11T06:14:02.230 INFO:teuthology.orchestra.run.smithi006.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-11T06:14:02.230 INFO:teuthology.orchestra.run.smithi006.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T06:14:02.230 INFO:teuthology.orchestra.run.smithi006.stdout:Device: 5h/5d Inode: 941 Links: 1 2024-09-11T06:14:02.230 INFO:teuthology.orchestra.run.smithi006.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T06:14:02.230 INFO:teuthology.orchestra.run.smithi006.stdout:Access: 2024-09-11 06:08:31.940869530 +0000 2024-09-11T06:14:02.230 INFO:teuthology.orchestra.run.smithi006.stdout:Modify: 2024-09-11 06:08:31.684871817 +0000 2024-09-11T06:14:02.230 INFO:teuthology.orchestra.run.smithi006.stdout:Change: 2024-09-11 06:08:31.684871817 +0000 2024-09-11T06:14:02.230 INFO:teuthology.orchestra.run.smithi006.stdout: Birth: - 2024-09-11T06:14:02.230 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-11T06:14:02.288 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records in 2024-09-11T06:14:02.288 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records out 2024-09-11T06:14:02.288 INFO:teuthology.orchestra.run.smithi006.stderr:512 bytes copied, 0.000368472 s, 1.4 MB/s 2024-09-11T06:14:02.289 DEBUG:teuthology.orchestra.run.smithi006:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-11T06:14:02.338 INFO:tasks.ceph:osd dev map: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-11T06:14:02.338 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi006.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-11T06:14:02.339 INFO:tasks.ceph:Generating config... 2024-09-11T06:14:02.340 INFO:tasks.ceph:[client] debug rbd = 20 2024-09-11T06:14:02.340 INFO:tasks.ceph:[client] debug rbd_mirror = 15 2024-09-11T06:14:02.340 INFO:tasks.ceph:[client] log to stderr = False 2024-09-11T06:14:02.340 INFO:tasks.ceph:[client] rbd default features = 125 2024-09-11T06:14:02.340 INFO:tasks.ceph:[client.mirror.0] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-11T06:14:02.340 INFO:tasks.ceph:[client.mirror.0] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-11T06:14:02.340 INFO:tasks.ceph:[client.mirror.1] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-11T06:14:02.340 INFO:tasks.ceph:[client.mirror.1] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-11T06:14:02.341 INFO:tasks.ceph:[client.mirror.2] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-11T06:14:02.341 INFO:tasks.ceph:[client.mirror.2] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-11T06:14:02.341 INFO:tasks.ceph:[client.mirror.3] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-11T06:14:02.341 INFO:tasks.ceph:[client.mirror.3] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-11T06:14:02.341 INFO:tasks.ceph:[client.mirror.4] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-11T06:14:02.341 INFO:tasks.ceph:[client.mirror.4] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-11T06:14:02.341 INFO:tasks.ceph:[client.mirror.5] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-11T06:14:02.341 INFO:tasks.ceph:[client.mirror.5] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-11T06:14:02.341 INFO:tasks.ceph:[client.mirror.6] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-11T06:14:02.341 INFO:tasks.ceph:[client.mirror.6] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-11T06:14:02.341 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-09-11T06:14:02.341 INFO:tasks.ceph:[global] mon warn on pool no app = False 2024-09-11T06:14:02.342 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-09-11T06:14:02.342 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-11T06:14:02.342 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-11T06:14:02.342 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-11T06:14:02.342 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-11T06:14:02.342 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-11T06:14:02.342 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-11T06:14:02.342 INFO:tasks.ceph:[osd] bluestore compression algorithm = zlib 2024-09-11T06:14:02.342 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-09-11T06:14:02.342 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-11T06:14:02.342 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-09-11T06:14:02.342 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-09-11T06:14:02.342 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-11T06:14:02.342 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-11T06:14:02.343 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-09-11T06:14:02.343 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-11T06:14:02.343 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-11T06:14:02.343 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-11T06:14:02.343 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-11T06:14:02.343 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-11T06:14:02.343 INFO:tasks.ceph:Setting up cluster1.mon.a... 2024-09-11T06:14:02.343 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/cluster1.keyring 2024-09-11T06:14:02.415 INFO:teuthology.orchestra.run.smithi006.stdout:creating /etc/ceph/cluster1.keyring 2024-09-11T06:14:02.424 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/cluster1.keyring 2024-09-11T06:14:02.505 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2024-09-11T06:14:02.561 DEBUG:tasks.ceph:Ceph mon addresses: [('cluster1.mon.a', '172.21.15.6')] 2024-09-11T06:14:02.562 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.6', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125}, 'mon.a': {}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}} 2024-09-11T06:14:02.563 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-09-11T06:14:02.563 DEBUG:teuthology.orchestra.run.smithi006:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-11T06:14:02.609 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.6 --print /home/ubuntu/cephtest/cluster1.monmap 2024-09-11T06:14:02.690 INFO:teuthology.orchestra.run.smithi006.stdout:monmaptool: monmap file /home/ubuntu/cephtest/cluster1.monmap 2024-09-11T06:14:02.690 INFO:teuthology.orchestra.run.smithi006.stdout:monmaptool: generated fsid c4197d0c-e10d-4a46-b3cc-95a1d7af9683 2024-09-11T06:14:02.690 INFO:teuthology.orchestra.run.smithi006.stdout:setting min_mon_release = quincy 2024-09-11T06:14:02.690 INFO:teuthology.orchestra.run.smithi006.stdout:epoch 0 2024-09-11T06:14:02.690 INFO:teuthology.orchestra.run.smithi006.stdout:fsid c4197d0c-e10d-4a46-b3cc-95a1d7af9683 2024-09-11T06:14:02.691 INFO:teuthology.orchestra.run.smithi006.stdout:last_changed 2024-09-11T06:14:02.690549+0000 2024-09-11T06:14:02.691 INFO:teuthology.orchestra.run.smithi006.stdout:created 2024-09-11T06:14:02.690549+0000 2024-09-11T06:14:02.691 INFO:teuthology.orchestra.run.smithi006.stdout:min_mon_release 17 (quincy) 2024-09-11T06:14:02.691 INFO:teuthology.orchestra.run.smithi006.stdout:election_strategy: 1 2024-09-11T06:14:02.691 INFO:teuthology.orchestra.run.smithi006.stdout:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2024-09-11T06:14:02.691 INFO:teuthology.orchestra.run.smithi006.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/cluster1.monmap (1 monitors) 2024-09-11T06:14:02.692 DEBUG:teuthology.orchestra.run.smithi006:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-11T06:14:02.741 INFO:tasks.ceph:Writing /etc/ceph/cluster1.conf for FSID c4197d0c-e10d-4a46-b3cc-95a1d7af9683... 2024-09-11T06:14:02.742 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster1.conf && sudo chmod 0644 /etc/ceph/cluster1.conf > /dev/null 2024-09-11T06:14:02.786 DEBUG:teuthology.orchestra.run.smithi112:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster1.conf && sudo chmod 0644 /etc/ceph/cluster1.conf > /dev/null 2024-09-11T06:14:02.814 INFO:teuthology.orchestra.run.smithi006.stdout:[global] 2024-09-11T06:14:02.814 INFO:teuthology.orchestra.run.smithi006.stdout: chdir = "" 2024-09-11T06:14:02.814 INFO:teuthology.orchestra.run.smithi006.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-11T06:14:02.814 INFO:teuthology.orchestra.run.smithi006.stdout: auth supported = cephx 2024-09-11T06:14:02.814 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-11T06:14:02.815 INFO:teuthology.orchestra.run.smithi006.stdout: filestore xattr use omap = true 2024-09-11T06:14:02.815 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-11T06:14:02.815 INFO:teuthology.orchestra.run.smithi006.stdout: mon clock drift allowed = 1.000 2024-09-11T06:14:02.815 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-11T06:14:02.815 INFO:teuthology.orchestra.run.smithi006.stdout: osd crush chooseleaf type = 0 2024-09-11T06:14:02.815 INFO:teuthology.orchestra.run.smithi006.stdout: auth debug = true 2024-09-11T06:14:02.815 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-11T06:14:02.815 INFO:teuthology.orchestra.run.smithi006.stdout: ms die on old message = true 2024-09-11T06:14:02.815 INFO:teuthology.orchestra.run.smithi006.stdout: ms die on bug = true 2024-09-11T06:14:02.815 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-11T06:14:02.815 INFO:teuthology.orchestra.run.smithi006.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-11T06:14:02.815 INFO:teuthology.orchestra.run.smithi006.stdout: mon pg warn max object skew = 0 2024-09-11T06:14:02.815 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-11T06:14:02.815 INFO:teuthology.orchestra.run.smithi006.stdout: # disable pg_autoscaler by default for new pools 2024-09-11T06:14:02.816 INFO:teuthology.orchestra.run.smithi006.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-11T06:14:02.816 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-11T06:14:02.816 INFO:teuthology.orchestra.run.smithi006.stdout: osd pool default size = 2 2024-09-11T06:14:02.816 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-11T06:14:02.816 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd allow primary affinity = true 2024-09-11T06:14:02.816 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd allow pg remap = true 2024-09-11T06:14:02.816 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on legacy crush tunables = false 2024-09-11T06:14:02.816 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on crush straw calc version zero = false 2024-09-11T06:14:02.816 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on no sortbitwise = false 2024-09-11T06:14:02.816 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on osd down out interval zero = false 2024-09-11T06:14:02.816 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on too few osds = false 2024-09-11T06:14:02.816 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-11T06:14:02.817 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-11T06:14:02.817 INFO:teuthology.orchestra.run.smithi006.stdout: mon_allow_pool_size_one = true 2024-09-11T06:14:02.817 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-11T06:14:02.817 INFO:teuthology.orchestra.run.smithi006.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-11T06:14:02.817 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-11T06:14:02.817 INFO:teuthology.orchestra.run.smithi006.stdout: osd default data pool replay window = 5 2024-09-11T06:14:02.817 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-11T06:14:02.817 INFO:teuthology.orchestra.run.smithi006.stdout: mon allow pool delete = true 2024-09-11T06:14:02.817 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-11T06:14:02.817 INFO:teuthology.orchestra.run.smithi006.stdout: mon cluster log file level = debug 2024-09-11T06:14:02.817 INFO:teuthology.orchestra.run.smithi006.stdout: debug asserts on shutdown = true 2024-09-11T06:14:02.817 INFO:teuthology.orchestra.run.smithi006.stdout: mon health detail to clog = false 2024-09-11T06:14:02.817 INFO:teuthology.orchestra.run.smithi006.stdout: mon host = 172.21.15.6 2024-09-11T06:14:02.818 INFO:teuthology.orchestra.run.smithi006.stdout: mon client directed command retry = 5 2024-09-11T06:14:02.818 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on pool no app = False 2024-09-11T06:14:02.818 INFO:teuthology.orchestra.run.smithi006.stdout: ms inject socket failures = 5000 2024-09-11T06:14:02.818 INFO:teuthology.orchestra.run.smithi006.stdout: fsid = c4197d0c-e10d-4a46-b3cc-95a1d7af9683 2024-09-11T06:14:02.818 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-11T06:14:02.818 INFO:teuthology.orchestra.run.smithi006.stdout:[osd] 2024-09-11T06:14:02.818 INFO:teuthology.orchestra.run.smithi006.stdout: osd journal size = 100 2024-09-11T06:14:02.818 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-11T06:14:02.818 INFO:teuthology.orchestra.run.smithi006.stdout: osd scrub load threshold = 5.0 2024-09-11T06:14:02.818 INFO:teuthology.orchestra.run.smithi006.stdout: osd scrub max interval = 600 2024-09-11T06:14:02.818 INFO:teuthology.orchestra.run.smithi006.stdout: osd mclock profile = high_recovery_ops 2024-09-11T06:14:02.818 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-11T06:14:02.818 INFO:teuthology.orchestra.run.smithi006.stdout: osd recover clone overlap = true 2024-09-11T06:14:02.819 INFO:teuthology.orchestra.run.smithi006.stdout: osd recovery max chunk = 1048576 2024-09-11T06:14:02.819 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-11T06:14:02.819 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug shutdown = true 2024-09-11T06:14:02.819 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug op order = true 2024-09-11T06:14:02.819 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug verify stray on activate = true 2024-09-11T06:14:02.819 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug trim objects = true 2024-09-11T06:14:02.819 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-11T06:14:02.819 INFO:teuthology.orchestra.run.smithi006.stdout: osd open classes on start = true 2024-09-11T06:14:02.819 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug pg log writeout = true 2024-09-11T06:14:02.819 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-11T06:14:02.819 INFO:teuthology.orchestra.run.smithi006.stdout: osd deep scrub update digest min age = 30 2024-09-11T06:14:02.819 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-11T06:14:02.819 INFO:teuthology.orchestra.run.smithi006.stdout: osd map max advance = 10 2024-09-11T06:14:02.820 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-11T06:14:02.820 INFO:teuthology.orchestra.run.smithi006.stdout: journal zero on create = true 2024-09-11T06:14:02.820 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-11T06:14:02.820 INFO:teuthology.orchestra.run.smithi006.stdout: filestore ondisk finisher threads = 3 2024-09-11T06:14:02.820 INFO:teuthology.orchestra.run.smithi006.stdout: filestore apply finisher threads = 3 2024-09-11T06:14:02.820 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-11T06:14:02.820 INFO:teuthology.orchestra.run.smithi006.stdout: bdev debug aio = true 2024-09-11T06:14:02.820 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug misdirected ops = true 2024-09-11T06:14:02.820 INFO:teuthology.orchestra.run.smithi006.stdout: bluestore block size = 96636764160 2024-09-11T06:14:02.820 INFO:teuthology.orchestra.run.smithi006.stdout: bluestore compression algorithm = zlib 2024-09-11T06:14:02.820 INFO:teuthology.orchestra.run.smithi006.stdout: bluestore compression mode = aggressive 2024-09-11T06:14:02.820 INFO:teuthology.orchestra.run.smithi006.stdout: bluestore fsck on mount = True 2024-09-11T06:14:02.820 INFO:teuthology.orchestra.run.smithi006.stdout: debug bluefs = 1/20 2024-09-11T06:14:02.820 INFO:teuthology.orchestra.run.smithi006.stdout: debug bluestore = 1/20 2024-09-11T06:14:02.821 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-09-11T06:14:02.821 INFO:teuthology.orchestra.run.smithi006.stdout: debug osd = 20 2024-09-11T06:14:02.821 INFO:teuthology.orchestra.run.smithi006.stdout: debug rocksdb = 4/10 2024-09-11T06:14:02.821 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd backfillfull_ratio = 0.85 2024-09-11T06:14:02.821 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd full ratio = 0.9 2024-09-11T06:14:02.821 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd nearfull ratio = 0.8 2024-09-11T06:14:02.821 INFO:teuthology.orchestra.run.smithi006.stdout: osd failsafe full ratio = 0.95 2024-09-11T06:14:02.821 INFO:teuthology.orchestra.run.smithi006.stdout: osd objectstore = bluestore 2024-09-11T06:14:02.821 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-11T06:14:02.821 INFO:teuthology.orchestra.run.smithi006.stdout:[mgr] 2024-09-11T06:14:02.821 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-09-11T06:14:02.821 INFO:teuthology.orchestra.run.smithi006.stdout: debug mgr = 20 2024-09-11T06:14:02.822 INFO:teuthology.orchestra.run.smithi006.stdout: debug mon = 20 2024-09-11T06:14:02.822 INFO:teuthology.orchestra.run.smithi006.stdout: debug auth = 20 2024-09-11T06:14:02.822 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min pgs per osd = 4 2024-09-11T06:14:02.822 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min bytes per osd = 10 2024-09-11T06:14:02.822 INFO:teuthology.orchestra.run.smithi006.stdout: mgr/telemetry/nag = false 2024-09-11T06:14:02.822 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-11T06:14:02.822 INFO:teuthology.orchestra.run.smithi006.stdout:[mon] 2024-09-11T06:14:02.822 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-09-11T06:14:02.822 INFO:teuthology.orchestra.run.smithi006.stdout: debug mon = 20 2024-09-11T06:14:02.822 INFO:teuthology.orchestra.run.smithi006.stdout: debug paxos = 20 2024-09-11T06:14:02.822 INFO:teuthology.orchestra.run.smithi006.stdout: debug auth = 20 2024-09-11T06:14:02.823 INFO:teuthology.orchestra.run.smithi006.stdout: mon data avail warn = 5 2024-09-11T06:14:02.823 INFO:teuthology.orchestra.run.smithi006.stdout: mon mgr mkfs grace = 240 2024-09-11T06:14:02.823 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min pgs per osd = 4 2024-09-11T06:14:02.823 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd reporter subtree level = osd 2024-09-11T06:14:02.823 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd prime pg temp = true 2024-09-11T06:14:02.823 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min bytes per osd = 10 2024-09-11T06:14:02.823 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-11T06:14:02.823 INFO:teuthology.orchestra.run.smithi006.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-11T06:14:02.823 INFO:teuthology.orchestra.run.smithi006.stdout: auth mon ticket ttl = 660 # 11m 2024-09-11T06:14:02.824 INFO:teuthology.orchestra.run.smithi006.stdout: auth service ticket ttl = 240 # 4m 2024-09-11T06:14:02.824 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-11T06:14:02.824 INFO:teuthology.orchestra.run.smithi006.stdout: # don't complain about insecure global_id in the test suite 2024-09-11T06:14:02.824 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-11T06:14:02.824 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-11T06:14:02.824 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-11T06:14:02.824 INFO:teuthology.orchestra.run.smithi006.stdout: # 1m isn't quite enough 2024-09-11T06:14:02.824 INFO:teuthology.orchestra.run.smithi006.stdout: mon_down_mkfs_grace = 2m 2024-09-11T06:14:02.824 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-11T06:14:02.824 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_filestore_osds = false 2024-09-11T06:14:02.824 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-11T06:14:02.824 INFO:teuthology.orchestra.run.smithi006.stdout:[client] 2024-09-11T06:14:02.824 INFO:teuthology.orchestra.run.smithi006.stdout: rgw cache enabled = true 2024-09-11T06:14:02.824 INFO:teuthology.orchestra.run.smithi006.stdout: rgw enable ops log = true 2024-09-11T06:14:02.825 INFO:teuthology.orchestra.run.smithi006.stdout: rgw enable usage log = true 2024-09-11T06:14:02.825 INFO:teuthology.orchestra.run.smithi006.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-11T06:14:02.825 INFO:teuthology.orchestra.run.smithi006.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-11T06:14:02.825 INFO:teuthology.orchestra.run.smithi006.stdout: debug rbd = 20 2024-09-11T06:14:02.825 INFO:teuthology.orchestra.run.smithi006.stdout: debug rbd_mirror = 15 2024-09-11T06:14:02.825 INFO:teuthology.orchestra.run.smithi006.stdout: log to stderr = False 2024-09-11T06:14:02.825 INFO:teuthology.orchestra.run.smithi006.stdout: rbd default features = 125 2024-09-11T06:14:02.825 INFO:teuthology.orchestra.run.smithi006.stdout:[mon.a] 2024-09-11T06:14:02.825 INFO:teuthology.orchestra.run.smithi006.stdout:[client.mirror.0] 2024-09-11T06:14:02.825 INFO:teuthology.orchestra.run.smithi006.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-11T06:14:02.826 INFO:teuthology.orchestra.run.smithi006.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-11T06:14:02.826 INFO:teuthology.orchestra.run.smithi006.stdout:[client.mirror.1] 2024-09-11T06:14:02.826 INFO:teuthology.orchestra.run.smithi006.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-11T06:14:02.826 INFO:teuthology.orchestra.run.smithi006.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-11T06:14:02.826 INFO:teuthology.orchestra.run.smithi006.stdout:[client.mirror.2] 2024-09-11T06:14:02.826 INFO:teuthology.orchestra.run.smithi006.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-11T06:14:02.826 INFO:teuthology.orchestra.run.smithi006.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-11T06:14:02.826 INFO:teuthology.orchestra.run.smithi006.stdout:[client.mirror.3] 2024-09-11T06:14:02.826 INFO:teuthology.orchestra.run.smithi006.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-11T06:14:02.826 INFO:teuthology.orchestra.run.smithi006.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-11T06:14:02.826 INFO:teuthology.orchestra.run.smithi006.stdout:[client.mirror.4] 2024-09-11T06:14:02.826 INFO:teuthology.orchestra.run.smithi006.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-11T06:14:02.826 INFO:teuthology.orchestra.run.smithi006.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-11T06:14:02.826 INFO:teuthology.orchestra.run.smithi006.stdout:[client.mirror.5] 2024-09-11T06:14:02.826 INFO:teuthology.orchestra.run.smithi006.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-11T06:14:02.827 INFO:teuthology.orchestra.run.smithi006.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-11T06:14:02.827 INFO:teuthology.orchestra.run.smithi006.stdout:[client.mirror.6] 2024-09-11T06:14:02.827 INFO:teuthology.orchestra.run.smithi006.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-11T06:14:02.827 INFO:teuthology.orchestra.run.smithi006.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-11T06:14:02.828 INFO:teuthology.orchestra.run.smithi112.stdout:[global] 2024-09-11T06:14:02.828 INFO:teuthology.orchestra.run.smithi112.stdout: chdir = "" 2024-09-11T06:14:02.828 INFO:teuthology.orchestra.run.smithi112.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-11T06:14:02.828 INFO:teuthology.orchestra.run.smithi112.stdout: auth supported = cephx 2024-09-11T06:14:02.828 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-11T06:14:02.828 INFO:teuthology.orchestra.run.smithi112.stdout: filestore xattr use omap = true 2024-09-11T06:14:02.828 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-11T06:14:02.829 INFO:teuthology.orchestra.run.smithi112.stdout: mon clock drift allowed = 1.000 2024-09-11T06:14:02.829 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-11T06:14:02.829 INFO:teuthology.orchestra.run.smithi112.stdout: osd crush chooseleaf type = 0 2024-09-11T06:14:02.829 INFO:teuthology.orchestra.run.smithi112.stdout: auth debug = true 2024-09-11T06:14:02.829 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-11T06:14:02.829 INFO:teuthology.orchestra.run.smithi112.stdout: ms die on old message = true 2024-09-11T06:14:02.829 INFO:teuthology.orchestra.run.smithi112.stdout: ms die on bug = true 2024-09-11T06:14:02.829 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-11T06:14:02.829 INFO:teuthology.orchestra.run.smithi112.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-11T06:14:02.829 INFO:teuthology.orchestra.run.smithi112.stdout: mon pg warn max object skew = 0 2024-09-11T06:14:02.829 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-11T06:14:02.829 INFO:teuthology.orchestra.run.smithi112.stdout: # disable pg_autoscaler by default for new pools 2024-09-11T06:14:02.829 INFO:teuthology.orchestra.run.smithi112.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-11T06:14:02.829 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-11T06:14:02.830 INFO:teuthology.orchestra.run.smithi112.stdout: osd pool default size = 2 2024-09-11T06:14:02.830 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-11T06:14:02.830 INFO:teuthology.orchestra.run.smithi112.stdout: mon osd allow primary affinity = true 2024-09-11T06:14:02.830 INFO:teuthology.orchestra.run.smithi112.stdout: mon osd allow pg remap = true 2024-09-11T06:14:02.830 INFO:teuthology.orchestra.run.smithi112.stdout: mon warn on legacy crush tunables = false 2024-09-11T06:14:02.830 INFO:teuthology.orchestra.run.smithi112.stdout: mon warn on crush straw calc version zero = false 2024-09-11T06:14:02.830 INFO:teuthology.orchestra.run.smithi112.stdout: mon warn on no sortbitwise = false 2024-09-11T06:14:02.830 INFO:teuthology.orchestra.run.smithi112.stdout: mon warn on osd down out interval zero = false 2024-09-11T06:14:02.830 INFO:teuthology.orchestra.run.smithi112.stdout: mon warn on too few osds = false 2024-09-11T06:14:02.830 INFO:teuthology.orchestra.run.smithi112.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-11T06:14:02.830 INFO:teuthology.orchestra.run.smithi112.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-11T06:14:02.830 INFO:teuthology.orchestra.run.smithi112.stdout: mon_allow_pool_size_one = true 2024-09-11T06:14:02.830 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-11T06:14:02.830 INFO:teuthology.orchestra.run.smithi112.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-11T06:14:02.830 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-11T06:14:02.831 INFO:teuthology.orchestra.run.smithi112.stdout: osd default data pool replay window = 5 2024-09-11T06:14:02.831 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-11T06:14:02.831 INFO:teuthology.orchestra.run.smithi112.stdout: mon allow pool delete = true 2024-09-11T06:14:02.831 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-11T06:14:02.831 INFO:teuthology.orchestra.run.smithi112.stdout: mon cluster log file level = debug 2024-09-11T06:14:02.831 INFO:teuthology.orchestra.run.smithi112.stdout: debug asserts on shutdown = true 2024-09-11T06:14:02.831 INFO:teuthology.orchestra.run.smithi112.stdout: mon health detail to clog = false 2024-09-11T06:14:02.831 INFO:teuthology.orchestra.run.smithi112.stdout: mon host = 172.21.15.6 2024-09-11T06:14:02.831 INFO:teuthology.orchestra.run.smithi112.stdout: mon client directed command retry = 5 2024-09-11T06:14:02.831 INFO:teuthology.orchestra.run.smithi112.stdout: mon warn on pool no app = False 2024-09-11T06:14:02.831 INFO:teuthology.orchestra.run.smithi112.stdout: ms inject socket failures = 5000 2024-09-11T06:14:02.831 INFO:teuthology.orchestra.run.smithi112.stdout: fsid = c4197d0c-e10d-4a46-b3cc-95a1d7af9683 2024-09-11T06:14:02.831 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-11T06:14:02.831 INFO:teuthology.orchestra.run.smithi112.stdout:[osd] 2024-09-11T06:14:02.831 INFO:teuthology.orchestra.run.smithi112.stdout: osd journal size = 100 2024-09-11T06:14:02.832 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-11T06:14:02.832 INFO:teuthology.orchestra.run.smithi112.stdout: osd scrub load threshold = 5.0 2024-09-11T06:14:02.832 INFO:teuthology.orchestra.run.smithi112.stdout: osd scrub max interval = 600 2024-09-11T06:14:02.832 INFO:teuthology.orchestra.run.smithi112.stdout: osd mclock profile = high_recovery_ops 2024-09-11T06:14:02.832 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-11T06:14:02.832 INFO:teuthology.orchestra.run.smithi112.stdout: osd recover clone overlap = true 2024-09-11T06:14:02.832 INFO:teuthology.orchestra.run.smithi112.stdout: osd recovery max chunk = 1048576 2024-09-11T06:14:02.832 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-11T06:14:02.832 INFO:teuthology.orchestra.run.smithi112.stdout: osd debug shutdown = true 2024-09-11T06:14:02.832 INFO:teuthology.orchestra.run.smithi112.stdout: osd debug op order = true 2024-09-11T06:14:02.832 INFO:teuthology.orchestra.run.smithi112.stdout: osd debug verify stray on activate = true 2024-09-11T06:14:02.832 INFO:teuthology.orchestra.run.smithi112.stdout: osd debug trim objects = true 2024-09-11T06:14:02.833 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-11T06:14:02.833 INFO:teuthology.orchestra.run.smithi112.stdout: osd open classes on start = true 2024-09-11T06:14:02.833 INFO:teuthology.orchestra.run.smithi112.stdout: osd debug pg log writeout = true 2024-09-11T06:14:02.833 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-11T06:14:02.833 INFO:teuthology.orchestra.run.smithi112.stdout: osd deep scrub update digest min age = 30 2024-09-11T06:14:02.833 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-11T06:14:02.833 INFO:teuthology.orchestra.run.smithi112.stdout: osd map max advance = 10 2024-09-11T06:14:02.833 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-11T06:14:02.833 INFO:teuthology.orchestra.run.smithi112.stdout: journal zero on create = true 2024-09-11T06:14:02.833 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-11T06:14:02.833 INFO:teuthology.orchestra.run.smithi112.stdout: filestore ondisk finisher threads = 3 2024-09-11T06:14:02.833 INFO:teuthology.orchestra.run.smithi112.stdout: filestore apply finisher threads = 3 2024-09-11T06:14:02.833 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-11T06:14:02.833 INFO:teuthology.orchestra.run.smithi112.stdout: bdev debug aio = true 2024-09-11T06:14:02.834 INFO:teuthology.orchestra.run.smithi112.stdout: osd debug misdirected ops = true 2024-09-11T06:14:02.834 INFO:teuthology.orchestra.run.smithi112.stdout: bluestore block size = 96636764160 2024-09-11T06:14:02.834 INFO:teuthology.orchestra.run.smithi112.stdout: bluestore compression algorithm = zlib 2024-09-11T06:14:02.834 INFO:teuthology.orchestra.run.smithi112.stdout: bluestore compression mode = aggressive 2024-09-11T06:14:02.834 INFO:teuthology.orchestra.run.smithi112.stdout: bluestore fsck on mount = True 2024-09-11T06:14:02.834 INFO:teuthology.orchestra.run.smithi112.stdout: debug bluefs = 1/20 2024-09-11T06:14:02.834 INFO:teuthology.orchestra.run.smithi112.stdout: debug bluestore = 1/20 2024-09-11T06:14:02.834 INFO:teuthology.orchestra.run.smithi112.stdout: debug ms = 1 2024-09-11T06:14:02.834 INFO:teuthology.orchestra.run.smithi112.stdout: debug osd = 20 2024-09-11T06:14:02.834 INFO:teuthology.orchestra.run.smithi112.stdout: debug rocksdb = 4/10 2024-09-11T06:14:02.834 INFO:teuthology.orchestra.run.smithi112.stdout: mon osd backfillfull_ratio = 0.85 2024-09-11T06:14:02.834 INFO:teuthology.orchestra.run.smithi112.stdout: mon osd full ratio = 0.9 2024-09-11T06:14:02.834 INFO:teuthology.orchestra.run.smithi112.stdout: mon osd nearfull ratio = 0.8 2024-09-11T06:14:02.834 INFO:teuthology.orchestra.run.smithi112.stdout: osd failsafe full ratio = 0.95 2024-09-11T06:14:02.835 INFO:teuthology.orchestra.run.smithi112.stdout: osd objectstore = bluestore 2024-09-11T06:14:02.835 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-11T06:14:02.835 INFO:teuthology.orchestra.run.smithi112.stdout:[mgr] 2024-09-11T06:14:02.835 INFO:teuthology.orchestra.run.smithi112.stdout: debug ms = 1 2024-09-11T06:14:02.835 INFO:teuthology.orchestra.run.smithi112.stdout: debug mgr = 20 2024-09-11T06:14:02.835 INFO:teuthology.orchestra.run.smithi112.stdout: debug mon = 20 2024-09-11T06:14:02.835 INFO:teuthology.orchestra.run.smithi112.stdout: debug auth = 20 2024-09-11T06:14:02.835 INFO:teuthology.orchestra.run.smithi112.stdout: mon reweight min pgs per osd = 4 2024-09-11T06:14:02.835 INFO:teuthology.orchestra.run.smithi112.stdout: mon reweight min bytes per osd = 10 2024-09-11T06:14:02.835 INFO:teuthology.orchestra.run.smithi112.stdout: mgr/telemetry/nag = false 2024-09-11T06:14:02.835 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-11T06:14:02.835 INFO:teuthology.orchestra.run.smithi112.stdout:[mon] 2024-09-11T06:14:02.835 INFO:teuthology.orchestra.run.smithi112.stdout: debug ms = 1 2024-09-11T06:14:02.835 INFO:teuthology.orchestra.run.smithi112.stdout: debug mon = 20 2024-09-11T06:14:02.836 INFO:teuthology.orchestra.run.smithi112.stdout: debug paxos = 20 2024-09-11T06:14:02.836 INFO:teuthology.orchestra.run.smithi112.stdout: debug auth = 20 2024-09-11T06:14:02.836 INFO:teuthology.orchestra.run.smithi112.stdout: mon data avail warn = 5 2024-09-11T06:14:02.836 INFO:teuthology.orchestra.run.smithi112.stdout: mon mgr mkfs grace = 240 2024-09-11T06:14:02.836 INFO:teuthology.orchestra.run.smithi112.stdout: mon reweight min pgs per osd = 4 2024-09-11T06:14:02.836 INFO:teuthology.orchestra.run.smithi112.stdout: mon osd reporter subtree level = osd 2024-09-11T06:14:02.836 INFO:teuthology.orchestra.run.smithi112.stdout: mon osd prime pg temp = true 2024-09-11T06:14:02.836 INFO:teuthology.orchestra.run.smithi112.stdout: mon reweight min bytes per osd = 10 2024-09-11T06:14:02.836 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-11T06:14:02.836 INFO:teuthology.orchestra.run.smithi112.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-11T06:14:02.836 INFO:teuthology.orchestra.run.smithi112.stdout: auth mon ticket ttl = 660 # 11m 2024-09-11T06:14:02.836 INFO:teuthology.orchestra.run.smithi112.stdout: auth service ticket ttl = 240 # 4m 2024-09-11T06:14:02.836 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-11T06:14:02.836 INFO:teuthology.orchestra.run.smithi112.stdout: # don't complain about insecure global_id in the test suite 2024-09-11T06:14:02.837 INFO:teuthology.orchestra.run.smithi112.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-11T06:14:02.837 INFO:teuthology.orchestra.run.smithi112.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-11T06:14:02.837 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-11T06:14:02.837 INFO:teuthology.orchestra.run.smithi112.stdout: # 1m isn't quite enough 2024-09-11T06:14:02.837 INFO:teuthology.orchestra.run.smithi112.stdout: mon_down_mkfs_grace = 2m 2024-09-11T06:14:02.837 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-11T06:14:02.837 INFO:teuthology.orchestra.run.smithi112.stdout: mon_warn_on_filestore_osds = false 2024-09-11T06:14:02.837 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-11T06:14:02.837 INFO:teuthology.orchestra.run.smithi112.stdout:[client] 2024-09-11T06:14:02.837 INFO:teuthology.orchestra.run.smithi112.stdout: rgw cache enabled = true 2024-09-11T06:14:02.837 INFO:teuthology.orchestra.run.smithi112.stdout: rgw enable ops log = true 2024-09-11T06:14:02.837 INFO:teuthology.orchestra.run.smithi112.stdout: rgw enable usage log = true 2024-09-11T06:14:02.837 INFO:teuthology.orchestra.run.smithi112.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-11T06:14:02.837 INFO:teuthology.orchestra.run.smithi112.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-11T06:14:02.837 INFO:teuthology.orchestra.run.smithi112.stdout: debug rbd = 20 2024-09-11T06:14:02.838 INFO:teuthology.orchestra.run.smithi112.stdout: debug rbd_mirror = 15 2024-09-11T06:14:02.838 INFO:teuthology.orchestra.run.smithi112.stdout: log to stderr = False 2024-09-11T06:14:02.838 INFO:teuthology.orchestra.run.smithi112.stdout: rbd default features = 125 2024-09-11T06:14:02.838 INFO:teuthology.orchestra.run.smithi112.stdout:[mon.a] 2024-09-11T06:14:02.838 INFO:teuthology.orchestra.run.smithi112.stdout:[client.mirror.0] 2024-09-11T06:14:02.838 INFO:teuthology.orchestra.run.smithi112.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-11T06:14:02.838 INFO:teuthology.orchestra.run.smithi112.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-11T06:14:02.838 INFO:teuthology.orchestra.run.smithi112.stdout:[client.mirror.1] 2024-09-11T06:14:02.838 INFO:teuthology.orchestra.run.smithi112.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-11T06:14:02.838 INFO:teuthology.orchestra.run.smithi112.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-11T06:14:02.838 INFO:teuthology.orchestra.run.smithi112.stdout:[client.mirror.2] 2024-09-11T06:14:02.838 INFO:teuthology.orchestra.run.smithi112.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-11T06:14:02.838 INFO:teuthology.orchestra.run.smithi112.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-11T06:14:02.838 INFO:teuthology.orchestra.run.smithi112.stdout:[client.mirror.3] 2024-09-11T06:14:02.838 INFO:teuthology.orchestra.run.smithi112.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-11T06:14:02.839 INFO:teuthology.orchestra.run.smithi112.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-11T06:14:02.839 INFO:teuthology.orchestra.run.smithi112.stdout:[client.mirror.4] 2024-09-11T06:14:02.839 INFO:teuthology.orchestra.run.smithi112.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-11T06:14:02.839 INFO:teuthology.orchestra.run.smithi112.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-11T06:14:02.839 INFO:teuthology.orchestra.run.smithi112.stdout:[client.mirror.5] 2024-09-11T06:14:02.839 INFO:teuthology.orchestra.run.smithi112.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-11T06:14:02.839 INFO:teuthology.orchestra.run.smithi112.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-11T06:14:02.839 INFO:teuthology.orchestra.run.smithi112.stdout:[client.mirror.6] 2024-09-11T06:14:02.839 INFO:teuthology.orchestra.run.smithi112.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-11T06:14:02.839 INFO:teuthology.orchestra.run.smithi112.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-11T06:14:02.839 INFO:tasks.ceph:Creating admin key on cluster1.mon.a... 2024-09-11T06:14:02.840 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/cluster1.keyring 2024-09-11T06:14:02.918 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-11T06:14:02.919 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-09-11T06:14:02.919 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/etc/ceph/cluster1.keyring of=/dev/stdout 2024-09-11T06:14:02.965 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-09-11T06:14:02.965 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/home/ubuntu/cephtest/cluster1.monmap of=/dev/stdout 2024-09-11T06:14:03.013 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi006.front.sepia.ceph.com 2024-09-11T06:14:03.013 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-09-11T06:14:03.013 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/etc/ceph/cluster1.keyring 2024-09-11T06:14:03.013 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2024-09-11T06:14:03.082 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-09-11T06:14:03.082 DEBUG:teuthology.orchestra.run.smithi006:> dd of=/home/ubuntu/cephtest/cluster1.monmap 2024-09-11T06:14:03.129 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi112.front.sepia.ceph.com 2024-09-11T06:14:03.130 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-11T06:14:03.130 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/etc/ceph/cluster1.keyring 2024-09-11T06:14:03.130 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2024-09-11T06:14:03.154 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-11T06:14:03.154 DEBUG:teuthology.orchestra.run.smithi112:> dd of=/home/ubuntu/cephtest/cluster1.monmap 2024-09-11T06:14:03.203 INFO:tasks.ceph:Setting up mon nodes... 2024-09-11T06:14:03.203 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-11T06:14:03.203 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /var/lib/ceph/mgr/cluster1-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/cluster1-x/keyring 2024-09-11T06:14:03.248 INFO:teuthology.orchestra.run.smithi006.stdout:creating /var/lib/ceph/mgr/cluster1-x/keyring 2024-09-11T06:14:03.251 INFO:tasks.ceph:Setting up mds nodes... 2024-09-11T06:14:03.252 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-11T06:14:03.252 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/cluster1.client.0.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.0.keyring 2024-09-11T06:14:03.286 INFO:teuthology.orchestra.run.smithi006.stdout:creating /etc/ceph/cluster1.client.0.keyring 2024-09-11T06:14:03.297 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror /etc/ceph/cluster1.client.mirror.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.keyring 2024-09-11T06:14:03.332 INFO:teuthology.orchestra.run.smithi112.stdout:creating /etc/ceph/cluster1.client.mirror.keyring 2024-09-11T06:14:03.353 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.0 /etc/ceph/cluster1.client.mirror.0.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.0.keyring 2024-09-11T06:14:03.429 INFO:teuthology.orchestra.run.smithi112.stdout:creating /etc/ceph/cluster1.client.mirror.0.keyring 2024-09-11T06:14:03.440 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.1 /etc/ceph/cluster1.client.mirror.1.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.1.keyring 2024-09-11T06:14:03.513 INFO:teuthology.orchestra.run.smithi112.stdout:creating /etc/ceph/cluster1.client.mirror.1.keyring 2024-09-11T06:14:03.523 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.2 /etc/ceph/cluster1.client.mirror.2.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.2.keyring 2024-09-11T06:14:03.598 INFO:teuthology.orchestra.run.smithi112.stdout:creating /etc/ceph/cluster1.client.mirror.2.keyring 2024-09-11T06:14:03.608 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.3 /etc/ceph/cluster1.client.mirror.3.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.3.keyring 2024-09-11T06:14:03.681 INFO:teuthology.orchestra.run.smithi112.stdout:creating /etc/ceph/cluster1.client.mirror.3.keyring 2024-09-11T06:14:03.692 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.4 /etc/ceph/cluster1.client.mirror.4.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.4.keyring 2024-09-11T06:14:03.764 INFO:teuthology.orchestra.run.smithi112.stdout:creating /etc/ceph/cluster1.client.mirror.4.keyring 2024-09-11T06:14:03.774 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.5 /etc/ceph/cluster1.client.mirror.5.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.5.keyring 2024-09-11T06:14:03.849 INFO:teuthology.orchestra.run.smithi112.stdout:creating /etc/ceph/cluster1.client.mirror.5.keyring 2024-09-11T06:14:03.859 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.6 /etc/ceph/cluster1.client.mirror.6.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.6.keyring 2024-09-11T06:14:03.932 INFO:teuthology.orchestra.run.smithi112.stdout:creating /etc/ceph/cluster1.client.mirror.6.keyring 2024-09-11T06:14:03.942 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-11T06:14:03.942 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi006.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-11T06:14:03.942 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /var/lib/ceph/osd/cluster1-0 2024-09-11T06:14:03.957 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-11T06:14:03.957 INFO:tasks.ceph:role: cluster1.osd.0 2024-09-11T06:14:03.957 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi006.front.sepia.ceph.com 2024-09-11T06:14:03.957 DEBUG:teuthology.orchestra.run.smithi006:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-11T06:14:04.022 INFO:teuthology.orchestra.run.smithi006.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-11T06:14:04.022 INFO:teuthology.orchestra.run.smithi006.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-11T06:14:04.022 INFO:teuthology.orchestra.run.smithi006.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-11T06:14:04.022 INFO:teuthology.orchestra.run.smithi006.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-11T06:14:04.022 INFO:teuthology.orchestra.run.smithi006.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-11T06:14:04.023 INFO:teuthology.orchestra.run.smithi006.stdout: = sunit=0 swidth=0 blks 2024-09-11T06:14:04.023 INFO:teuthology.orchestra.run.smithi006.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-11T06:14:04.023 INFO:teuthology.orchestra.run.smithi006.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-11T06:14:04.023 INFO:teuthology.orchestra.run.smithi006.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-11T06:14:04.023 INFO:teuthology.orchestra.run.smithi006.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-11T06:14:04.463 INFO:teuthology.orchestra.run.smithi006.stdout:Discarding blocks...Done. 2024-09-11T06:14:04.465 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi006.front.sepia.ceph.com -o noatime 2024-09-11T06:14:04.466 DEBUG:teuthology.orchestra.run.smithi006:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/cluster1-0 2024-09-11T06:14:04.559 DEBUG:teuthology.orchestra.run.smithi006:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-0 2024-09-11T06:14:04.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T06:14:04.569 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: /sbin/restorecon: command not found 2024-09-11T06:14:04.570 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /var/lib/ceph/osd/cluster1-1 2024-09-11T06:14:04.623 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-11T06:14:04.623 INFO:tasks.ceph:role: cluster1.osd.1 2024-09-11T06:14:04.624 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi006.front.sepia.ceph.com 2024-09-11T06:14:04.624 DEBUG:teuthology.orchestra.run.smithi006:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-11T06:14:04.685 INFO:teuthology.orchestra.run.smithi006.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-11T06:14:04.685 INFO:teuthology.orchestra.run.smithi006.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-11T06:14:04.685 INFO:teuthology.orchestra.run.smithi006.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-11T06:14:04.685 INFO:teuthology.orchestra.run.smithi006.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-11T06:14:04.685 INFO:teuthology.orchestra.run.smithi006.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-11T06:14:04.685 INFO:teuthology.orchestra.run.smithi006.stdout: = sunit=0 swidth=0 blks 2024-09-11T06:14:04.685 INFO:teuthology.orchestra.run.smithi006.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-11T06:14:04.685 INFO:teuthology.orchestra.run.smithi006.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-11T06:14:04.685 INFO:teuthology.orchestra.run.smithi006.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-11T06:14:04.685 INFO:teuthology.orchestra.run.smithi006.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-11T06:14:05.153 INFO:teuthology.orchestra.run.smithi006.stdout:Discarding blocks...Done. 2024-09-11T06:14:05.155 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi006.front.sepia.ceph.com -o noatime 2024-09-11T06:14:05.155 DEBUG:teuthology.orchestra.run.smithi006:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/cluster1-1 2024-09-11T06:14:05.302 DEBUG:teuthology.orchestra.run.smithi006:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-1 2024-09-11T06:14:05.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T06:14:05.313 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: /sbin/restorecon: command not found 2024-09-11T06:14:05.314 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /var/lib/ceph/osd/cluster1-2 2024-09-11T06:14:05.368 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-11T06:14:05.369 INFO:tasks.ceph:role: cluster1.osd.2 2024-09-11T06:14:05.369 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi006.front.sepia.ceph.com 2024-09-11T06:14:05.369 DEBUG:teuthology.orchestra.run.smithi006:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-11T06:14:05.433 INFO:teuthology.orchestra.run.smithi006.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-11T06:14:05.434 INFO:teuthology.orchestra.run.smithi006.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-11T06:14:05.434 INFO:teuthology.orchestra.run.smithi006.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-11T06:14:05.434 INFO:teuthology.orchestra.run.smithi006.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-11T06:14:05.434 INFO:teuthology.orchestra.run.smithi006.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-11T06:14:05.434 INFO:teuthology.orchestra.run.smithi006.stdout: = sunit=0 swidth=0 blks 2024-09-11T06:14:05.434 INFO:teuthology.orchestra.run.smithi006.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-11T06:14:05.434 INFO:teuthology.orchestra.run.smithi006.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-11T06:14:05.434 INFO:teuthology.orchestra.run.smithi006.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-11T06:14:05.434 INFO:teuthology.orchestra.run.smithi006.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-11T06:14:05.914 INFO:teuthology.orchestra.run.smithi006.stdout:Discarding blocks...Done. 2024-09-11T06:14:05.916 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi006.front.sepia.ceph.com -o noatime 2024-09-11T06:14:05.916 DEBUG:teuthology.orchestra.run.smithi006:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/cluster1-2 2024-09-11T06:14:06.041 DEBUG:teuthology.orchestra.run.smithi006:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-2 2024-09-11T06:14:06.052 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: /sbin/restorecon: command not found 2024-09-11T06:14:06.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T06:14:06.053 DEBUG:teuthology.orchestra.run.smithi006:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/cluster1.monmap 2024-09-11T06:14:06.170 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-11T06:14:06.165+0000 7f4aeb3687c0 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-0/keyring: can't open /var/lib/ceph/osd/cluster1-0/keyring: (2) No such file or directory 2024-09-11T06:14:06.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-11T06:14:06.165+0000 7f4aeb3687c0 -1 created new key in keyring /var/lib/ceph/osd/cluster1-0/keyring 2024-09-11T06:14:06.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-11T06:14:06.165+0000 7f4aeb3687c0 -1 bdev(0x55e895c9f800 /var/lib/ceph/osd/cluster1-0/block) open stat got: (1) Operation not permitted 2024-09-11T06:14:06.171 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-11T06:14:06.165+0000 7f4aeb3687c0 -1 bluestore(/var/lib/ceph/osd/cluster1-0) _read_fsid unparsable uuid 2024-09-11T06:14:09.356 DEBUG:teuthology.orchestra.run.smithi006:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-0 2024-09-11T06:14:09.370 DEBUG:teuthology.orchestra.run.smithi006:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/cluster1.monmap 2024-09-11T06:14:09.451 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-11T06:14:09.445+0000 7f9082c6f7c0 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-1/keyring: can't open /var/lib/ceph/osd/cluster1-1/keyring: (2) No such file or directory 2024-09-11T06:14:09.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-11T06:14:09.445+0000 7f9082c6f7c0 -1 created new key in keyring /var/lib/ceph/osd/cluster1-1/keyring 2024-09-11T06:14:09.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-11T06:14:09.445+0000 7f9082c6f7c0 -1 bdev(0x56544eeb7800 /var/lib/ceph/osd/cluster1-1/block) open stat got: (1) Operation not permitted 2024-09-11T06:14:09.452 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-11T06:14:09.445+0000 7f9082c6f7c0 -1 bluestore(/var/lib/ceph/osd/cluster1-1) _read_fsid unparsable uuid 2024-09-11T06:14:12.616 DEBUG:teuthology.orchestra.run.smithi006:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-1 2024-09-11T06:14:12.673 DEBUG:teuthology.orchestra.run.smithi006:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/cluster1.monmap 2024-09-11T06:14:12.757 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-11T06:14:12.753+0000 7f2b6d49b7c0 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-2/keyring: can't open /var/lib/ceph/osd/cluster1-2/keyring: (2) No such file or directory 2024-09-11T06:14:12.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-11T06:14:12.753+0000 7f2b6d49b7c0 -1 created new key in keyring /var/lib/ceph/osd/cluster1-2/keyring 2024-09-11T06:14:12.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-11T06:14:12.753+0000 7f2b6d49b7c0 -1 bdev(0x564abe9c9800 /var/lib/ceph/osd/cluster1-2/block) open stat got: (1) Operation not permitted 2024-09-11T06:14:12.758 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-11T06:14:12.753+0000 7f2b6d49b7c0 -1 bluestore(/var/lib/ceph/osd/cluster1-2) _read_fsid unparsable uuid 2024-09-11T06:14:15.920 DEBUG:teuthology.orchestra.run.smithi006:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-2 2024-09-11T06:14:15.973 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-11T06:14:15.973 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-09-11T06:14:15.974 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/mgr/cluster1-x/keyring of=/dev/stdout 2024-09-11T06:14:16.029 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-09-11T06:14:16.029 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/osd/cluster1-0/keyring of=/dev/stdout 2024-09-11T06:14:16.089 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-09-11T06:14:16.089 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/osd/cluster1-1/keyring of=/dev/stdout 2024-09-11T06:14:16.144 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-09-11T06:14:16.144 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/osd/cluster1-2/keyring of=/dev/stdout 2024-09-11T06:14:16.200 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-09-11T06:14:16.201 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/etc/ceph/cluster1.client.0.keyring of=/dev/stdout 2024-09-11T06:14:16.253 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-11T06:14:16.253 DEBUG:teuthology.orchestra.run.smithi112:> dd if=/etc/ceph/cluster1.client.mirror.keyring of=/dev/stdout 2024-09-11T06:14:16.259 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-11T06:14:16.259 DEBUG:teuthology.orchestra.run.smithi112:> dd if=/etc/ceph/cluster1.client.mirror.0.keyring of=/dev/stdout 2024-09-11T06:14:16.306 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-11T06:14:16.306 DEBUG:teuthology.orchestra.run.smithi112:> dd if=/etc/ceph/cluster1.client.mirror.1.keyring of=/dev/stdout 2024-09-11T06:14:16.354 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-11T06:14:16.354 DEBUG:teuthology.orchestra.run.smithi112:> dd if=/etc/ceph/cluster1.client.mirror.2.keyring of=/dev/stdout 2024-09-11T06:14:16.402 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-11T06:14:16.402 DEBUG:teuthology.orchestra.run.smithi112:> dd if=/etc/ceph/cluster1.client.mirror.3.keyring of=/dev/stdout 2024-09-11T06:14:16.450 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-11T06:14:16.450 DEBUG:teuthology.orchestra.run.smithi112:> dd if=/etc/ceph/cluster1.client.mirror.4.keyring of=/dev/stdout 2024-09-11T06:14:16.498 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-11T06:14:16.499 DEBUG:teuthology.orchestra.run.smithi112:> dd if=/etc/ceph/cluster1.client.mirror.5.keyring of=/dev/stdout 2024-09-11T06:14:16.546 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-11T06:14:16.546 DEBUG:teuthology.orchestra.run.smithi112:> dd if=/etc/ceph/cluster1.client.mirror.6.keyring of=/dev/stdout 2024-09-11T06:14:16.594 INFO:tasks.ceph:Adding keys to all mons... 2024-09-11T06:14:16.594 DEBUG:teuthology.orchestra.run.smithi006:> sudo tee -a /etc/ceph/cluster1.keyring 2024-09-11T06:14:16.607 INFO:teuthology.orchestra.run.smithi006.stdout:[mgr.x] 2024-09-11T06:14:16.607 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQArNeFmVRHbDhAA8FGkAfQk1PYAxu0Ad3jLnQ== 2024-09-11T06:14:16.607 INFO:teuthology.orchestra.run.smithi006.stdout:[osd.0] 2024-09-11T06:14:16.607 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQAuNeFmkxUqChAATII5337Ukha8b2p/KlBUKg== 2024-09-11T06:14:16.607 INFO:teuthology.orchestra.run.smithi006.stdout:[osd.1] 2024-09-11T06:14:16.607 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQAxNeFmQhLtGhAA8oslAEp3XZQUfbMdF1YI4g== 2024-09-11T06:14:16.607 INFO:teuthology.orchestra.run.smithi006.stdout:[osd.2] 2024-09-11T06:14:16.607 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQA0NeFmJxkpLRAAriFFyIfc6E+L4AeukMBqOA== 2024-09-11T06:14:16.608 INFO:teuthology.orchestra.run.smithi006.stdout:[client.0] 2024-09-11T06:14:16.608 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQArNeFmie4WERAAxrS/En32nXSmocJo/F3kOg== 2024-09-11T06:14:16.608 INFO:teuthology.orchestra.run.smithi006.stdout:[client.mirror] 2024-09-11T06:14:16.608 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQArNeFmmQjtExAA67y5TzmrVP0wFCygKy7oMQ== 2024-09-11T06:14:16.608 INFO:teuthology.orchestra.run.smithi006.stdout:[client.mirror.0] 2024-09-11T06:14:16.608 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQArNeFmmey1GRAAVP7NPZoRCOGiWipbcYzlQA== 2024-09-11T06:14:16.608 INFO:teuthology.orchestra.run.smithi006.stdout:[client.mirror.1] 2024-09-11T06:14:16.608 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQArNeFmhf68HhAAmFLjhruV41s1zYhJ0T2nRQ== 2024-09-11T06:14:16.608 INFO:teuthology.orchestra.run.smithi006.stdout:[client.mirror.2] 2024-09-11T06:14:16.608 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQArNeFmxw/NIxAAuJdPtgE0fCgzmSv4fNAQZg== 2024-09-11T06:14:16.608 INFO:teuthology.orchestra.run.smithi006.stdout:[client.mirror.3] 2024-09-11T06:14:16.608 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQArNeFmgyq2KBAAYUjwbCzNdqe+13lAXmcBqA== 2024-09-11T06:14:16.608 INFO:teuthology.orchestra.run.smithi006.stdout:[client.mirror.4] 2024-09-11T06:14:16.609 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQArNeFmlomwLRAAbIuKhdEPl7Ijik3Zc9pQ4g== 2024-09-11T06:14:16.609 INFO:teuthology.orchestra.run.smithi006.stdout:[client.mirror.5] 2024-09-11T06:14:16.609 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQArNeFmoanLMhAAQ4UCfddDDmGBiaoUWhgAlQ== 2024-09-11T06:14:16.609 INFO:teuthology.orchestra.run.smithi006.stdout:[client.mirror.6] 2024-09-11T06:14:16.609 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQArNeFmHRe+NxAALFoPACVZgDoZcmPtDHodMA== 2024-09-11T06:14:16.610 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-11T06:14:16.699 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T06:14:16.745 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T06:14:16.831 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T06:14:16.918 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-11T06:14:17.013 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-11T06:14:17.057 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-11T06:14:17.147 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-11T06:14:17.235 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-11T06:14:17.325 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-11T06:14:17.368 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.4 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-11T06:14:17.453 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.5 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-11T06:14:17.544 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.6 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-11T06:14:17.588 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-11T06:14:17.588 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /var/lib/ceph/mon/cluster1-a 2024-09-11T06:14:17.644 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster cluster1 --mkfs -i a --monmap /home/ubuntu/cephtest/cluster1.monmap --keyring /etc/ceph/cluster1.keyring 2024-09-11T06:14:17.737 DEBUG:teuthology.orchestra.run.smithi006:> sudo chown -R ceph:ceph /var/lib/ceph/mon/cluster1-a 2024-09-11T06:14:17.748 DEBUG:teuthology.orchestra.run.smithi006:> rm -- /home/ubuntu/cephtest/cluster1.monmap 2024-09-11T06:14:17.801 INFO:tasks.ceph:Starting mon daemons in cluster cluster1... 2024-09-11T06:14:17.801 INFO:tasks.ceph.cluster1.mon.a:Restarting daemon 2024-09-11T06:14:17.801 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster cluster1 -i a 2024-09-11T06:14:17.846 INFO:tasks.ceph.cluster1.mon.a:Started 2024-09-11T06:14:17.846 INFO:tasks.ceph:Starting mgr daemons in cluster cluster1... 2024-09-11T06:14:17.847 INFO:tasks.ceph.cluster1.mgr.x:Restarting daemon 2024-09-11T06:14:17.847 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster cluster1 -i x 2024-09-11T06:14:17.849 INFO:tasks.ceph.cluster1.mgr.x:Started 2024-09-11T06:14:17.849 INFO:tasks.ceph:Setting crush tunables to default 2024-09-11T06:14:17.849 DEBUG:teuthology.orchestra.run.smithi006:> sudo ceph --cluster cluster1 osd crush tunables default 2024-09-11T06:14:18.097 INFO:teuthology.orchestra.run.smithi006.stderr:adjusted tunables profile to default 2024-09-11T06:14:18.115 INFO:tasks.ceph:check_enable_crimson: False 2024-09-11T06:14:18.115 INFO:tasks.ceph:Starting osd daemons in cluster cluster1... 2024-09-11T06:14:18.115 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-09-11T06:14:18.115 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/osd/cluster1-0/fsid of=/dev/stdout 2024-09-11T06:14:18.126 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-09-11T06:14:18.127 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/osd/cluster1-1/fsid of=/dev/stdout 2024-09-11T06:14:18.182 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-09-11T06:14:18.182 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/osd/cluster1-2/fsid of=/dev/stdout 2024-09-11T06:14:18.217 INFO:tasks.ceph.cluster1.mgr.x.smithi006.stderr:2024-09-11T06:14:18.213+0000 7fa02f539180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-11T06:14:18.228 DEBUG:teuthology.orchestra.run.smithi006:> sudo ceph --cluster cluster1 osd new 9351a1cd-aa93-4d78-a30d-8a0d7a3285f8 0 2024-09-11T06:14:18.481 INFO:teuthology.orchestra.run.smithi006.stdout:0 2024-09-11T06:14:18.497 DEBUG:teuthology.orchestra.run.smithi006:> sudo ceph --cluster cluster1 osd new d43f49e2-ee49-43c1-8a34-5a967034a8ad 1 2024-09-11T06:14:18.691 INFO:teuthology.orchestra.run.smithi006.stdout:1 2024-09-11T06:14:18.708 DEBUG:teuthology.orchestra.run.smithi006:> sudo ceph --cluster cluster1 osd new fd7eafd1-7f01-4be9-8977-8a7fad1990da 2 2024-09-11T06:14:18.725 INFO:tasks.ceph.cluster1.mgr.x.smithi006.stderr:2024-09-11T06:14:18.721+0000 7fa02f539180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-11T06:14:18.911 INFO:teuthology.orchestra.run.smithi006.stdout:2 2024-09-11T06:14:18.920 INFO:tasks.ceph.cluster1.mgr.x.smithi006.stderr:2024-09-11T06:14:18.913+0000 7fa02f539180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-11T06:14:18.928 INFO:tasks.ceph.cluster1.osd.0:Restarting daemon 2024-09-11T06:14:18.928 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 0 2024-09-11T06:14:18.970 INFO:tasks.ceph.cluster1.osd.0:Started 2024-09-11T06:14:18.970 INFO:tasks.ceph.cluster1.osd.1:Restarting daemon 2024-09-11T06:14:18.970 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 1 2024-09-11T06:14:18.972 INFO:tasks.ceph.cluster1.osd.1:Started 2024-09-11T06:14:18.972 INFO:tasks.ceph.cluster1.osd.2:Restarting daemon 2024-09-11T06:14:18.972 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 2 2024-09-11T06:14:18.974 INFO:tasks.ceph.cluster1.osd.2:Started 2024-09-11T06:14:18.974 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-09-11T06:14:19.151 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-11T06:14:19.151 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":5,"fsid":"c4197d0c-e10d-4a46-b3cc-95a1d7af9683","created":"2024-09-11T06:14:17.992688+0000","modified":"2024-09-11T06:14:18.910683+0000","last_up_change":"0.000000","last_in_change":"2024-09-11T06:14:18.910683+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9351a1cd-aa93-4d78-a30d-8a0d7a3285f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d43f49e2-ee49-43c1-8a34-5a967034a8ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fd7eafd1-7f01-4be9-8977-8a7fad1990da","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T06:14:19.166 INFO:tasks.ceph.ceph_manager.cluster1:[] 2024-09-11T06:14:19.167 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-11T06:14:19.469 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2024-09-11T06:14:19.529 INFO:tasks.ceph.cluster1.mgr.x.smithi006.stderr:2024-09-11T06:14:19.525+0000 7fa02f539180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-11T06:14:19.587 INFO:tasks.ceph.cluster1.mgr.x.smithi006.stderr:2024-09-11T06:14:19.581+0000 7fa02f539180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-11T06:14:19.643 INFO:tasks.ceph.cluster1.mgr.x.smithi006.stderr:2024-09-11T06:14:19.637+0000 7fa02f539180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-11T06:14:19.654 INFO:teuthology.misc.health.smithi006.stdout: 2024-09-11T06:14:19.654 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":5,"fsid":"c4197d0c-e10d-4a46-b3cc-95a1d7af9683","created":"2024-09-11T06:14:17.992688+0000","modified":"2024-09-11T06:14:18.910683+0000","last_up_change":"0.000000","last_in_change":"2024-09-11T06:14:18.910683+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9351a1cd-aa93-4d78-a30d-8a0d7a3285f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d43f49e2-ee49-43c1-8a34-5a967034a8ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fd7eafd1-7f01-4be9-8977-8a7fad1990da","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T06:14:19.669 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-09-11T06:14:19.725 INFO:tasks.ceph.cluster1.mgr.x.smithi006.stderr:2024-09-11T06:14:19.721+0000 7fa02f539180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-11T06:14:19.780 INFO:tasks.ceph.cluster1.mgr.x.smithi006.stderr:2024-09-11T06:14:19.773+0000 7fa02f539180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-11T06:14:19.860 INFO:tasks.ceph.cluster1.mgr.x.smithi006.stderr:2024-09-11T06:14:19.853+0000 7fa02f539180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-11T06:14:19.915 INFO:tasks.ceph.cluster1.mgr.x.smithi006.stderr:2024-09-11T06:14:19.909+0000 7fa02f539180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-11T06:14:20.073 INFO:tasks.ceph.cluster1.mgr.x.smithi006.stderr:2024-09-11T06:14:20.069+0000 7fa02f539180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-11T06:14:20.122 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:14:20.117+0000 7fc420f8a7c0 -1 Falling back to public interface 2024-09-11T06:14:20.137 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:14:20.133+0000 7fb4638ca7c0 -1 Falling back to public interface 2024-09-11T06:14:20.138 INFO:tasks.ceph.cluster1.mgr.x.smithi006.stderr:2024-09-11T06:14:20.133+0000 7fa02f539180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-11T06:14:20.833 INFO:tasks.ceph.cluster1.mgr.x.smithi006.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-09-11T06:14:20.833 INFO:tasks.ceph.cluster1.mgr.x.smithi006.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-09-11T06:14:20.833 INFO:tasks.ceph.cluster1.mgr.x.smithi006.stderr: from numpy import show_config as show_numpy_config 2024-09-11T06:14:20.850 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:14:20.845+0000 7fb2dad8f7c0 -1 Falling back to public interface 2024-09-11T06:14:20.877 INFO:tasks.ceph.cluster1.mgr.x.smithi006.stderr:2024-09-11T06:14:20.873+0000 7fa02f539180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-11T06:14:21.020 INFO:tasks.ceph.cluster1.mgr.x.smithi006.stderr:2024-09-11T06:14:21.013+0000 7fa02f539180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-11T06:14:21.298 INFO:tasks.ceph.cluster1.mgr.x.smithi006.stderr:2024-09-11T06:14:21.293+0000 7fa02f539180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-11T06:14:21.365 INFO:tasks.ceph.cluster1.mgr.x.smithi006.stderr:2024-09-11T06:14:21.361+0000 7fa02f539180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-11T06:14:21.506 INFO:tasks.ceph.cluster1.mgr.x.smithi006.stderr:2024-09-11T06:14:21.501+0000 7fa02f539180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-11T06:14:22.252 INFO:tasks.ceph.cluster1.mgr.x.smithi006.stderr:2024-09-11T06:14:22.245+0000 7fa02f539180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-11T06:14:22.331 INFO:tasks.ceph.cluster1.mgr.x.smithi006.stderr:2024-09-11T06:14:22.325+0000 7fa02f539180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-11T06:14:22.522 INFO:tasks.ceph.cluster1.mgr.x.smithi006.stderr:2024-09-11T06:14:22.517+0000 7fa02f539180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-11T06:14:22.599 INFO:tasks.ceph.cluster1.mgr.x.smithi006.stderr:2024-09-11T06:14:22.593+0000 7fa02f539180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-11T06:14:22.905 INFO:tasks.ceph.cluster1.mgr.x.smithi006.stderr:2024-09-11T06:14:22.901+0000 7fa02f539180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-11T06:14:22.971 INFO:tasks.ceph.cluster1.mgr.x.smithi006.stderr:2024-09-11T06:14:22.965+0000 7fa02f539180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-11T06:14:23.997 INFO:tasks.ceph.cluster1.mgr.x.smithi006.stderr:2024-09-11T06:14:23.993+0000 7f9fd6070640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-09-11T06:14:24.303 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:14:24.297+0000 7fb4638ca7c0 -1 osd.0 0 log_to_monitors true 2024-09-11T06:14:24.533 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:14:24.529+0000 7fc420f8a7c0 -1 osd.1 0 log_to_monitors true 2024-09-11T06:14:25.254 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:14:25.249+0000 7fb2dad8f7c0 -1 osd.2 0 log_to_monitors true 2024-09-11T06:14:25.972 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2024-09-11T06:14:26.291 INFO:teuthology.misc.health.smithi006.stdout: 2024-09-11T06:14:26.291 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":7,"fsid":"c4197d0c-e10d-4a46-b3cc-95a1d7af9683","created":"2024-09-11T06:14:17.992688+0000","modified":"2024-09-11T06:14:25.998470+0000","last_up_change":"0.000000","last_in_change":"2024-09-11T06:14:18.910683+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9351a1cd-aa93-4d78-a30d-8a0d7a3285f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d43f49e2-ee49-43c1-8a34-5a967034a8ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fd7eafd1-7f01-4be9-8977-8a7fad1990da","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T06:14:26.310 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-09-11T06:14:29.023 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:14:29.017+0000 7fb45f873640 -1 osd.0 0 waiting for initial osdmap 2024-09-11T06:14:29.057 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:14:29.053+0000 7fb4576a8640 -1 osd.0 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T06:14:29.061 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:14:29.057+0000 7fc41cf33640 -1 osd.1 0 waiting for initial osdmap 2024-09-11T06:14:29.093 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:14:29.089+0000 7fc414d68640 -1 osd.1 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T06:14:30.058 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:14:30.053+0000 7fb2d6d38640 -1 osd.2 0 waiting for initial osdmap 2024-09-11T06:14:30.085 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:14:30.081+0000 7fb2ceb6d640 -1 osd.2 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T06:14:32.613 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2024-09-11T06:14:32.953 INFO:teuthology.misc.health.smithi006.stdout: 2024-09-11T06:14:32.953 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":11,"fsid":"c4197d0c-e10d-4a46-b3cc-95a1d7af9683","created":"2024-09-11T06:14:17.992688+0000","modified":"2024-09-11T06:14:32.068129+0000","last_up_change":"2024-09-11T06:14:31.066451+0000","last_in_change":"2024-09-11T06:14:18.910683+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-11T06:14:31.066308+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9351a1cd-aa93-4d78-a30d-8a0d7a3285f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":2281847065},{"type":"v1","addr":"172.21.15.6:6809","nonce":2281847065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":2281847065},{"type":"v1","addr":"172.21.15.6:6811","nonce":2281847065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":2281847065},{"type":"v1","addr":"172.21.15.6:6815","nonce":2281847065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":2281847065},{"type":"v1","addr":"172.21.15.6:6813","nonce":2281847065}]},"public_addr":"172.21.15.6:6809/2281847065","cluster_addr":"172.21.15.6:6811/2281847065","heartbeat_back_addr":"172.21.15.6:6815/2281847065","heartbeat_front_addr":"172.21.15.6:6813/2281847065","state":["exists","up"]},{"osd":1,"uuid":"d43f49e2-ee49-43c1-8a34-5a967034a8ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":3363216727},{"type":"v1","addr":"172.21.15.6:6801","nonce":3363216727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":3363216727},{"type":"v1","addr":"172.21.15.6:6803","nonce":3363216727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":3363216727},{"type":"v1","addr":"172.21.15.6:6807","nonce":3363216727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":3363216727},{"type":"v1","addr":"172.21.15.6:6805","nonce":3363216727}]},"public_addr":"172.21.15.6:6801/3363216727","cluster_addr":"172.21.15.6:6803/3363216727","heartbeat_back_addr":"172.21.15.6:6807/3363216727","heartbeat_front_addr":"172.21.15.6:6805/3363216727","state":["exists","up"]},{"osd":2,"uuid":"fd7eafd1-7f01-4be9-8977-8a7fad1990da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":3564543485},{"type":"v1","addr":"172.21.15.6:6817","nonce":3564543485}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":3564543485},{"type":"v1","addr":"172.21.15.6:6819","nonce":3564543485}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":3564543485},{"type":"v1","addr":"172.21.15.6:6823","nonce":3564543485}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":3564543485},{"type":"v1","addr":"172.21.15.6:6821","nonce":3564543485}]},"public_addr":"172.21.15.6:6817/3564543485","cluster_addr":"172.21.15.6:6819/3564543485","heartbeat_back_addr":"172.21.15.6:6823/3564543485","heartbeat_front_addr":"172.21.15.6:6821/3564543485","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T06:14:25.351853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T06:14:25.565559+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T06:14:26.221801+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T06:14:32.972 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-09-11T06:14:32.972 INFO:tasks.ceph:Creating RBD pool 2024-09-11T06:14:32.972 DEBUG:teuthology.orchestra.run.smithi006:> sudo ceph --cluster cluster1 osd pool create rbd 8 2024-09-11T06:14:34.087 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'rbd' created 2024-09-11T06:14:34.116 DEBUG:teuthology.orchestra.run.smithi006:> rbd --cluster cluster1 pool init rbd 2024-09-11T06:14:37.134 INFO:tasks.ceph:Starting mds daemons in cluster cluster1... 2024-09-11T06:14:37.134 INFO:tasks.ceph:Waiting until cluster1 daemons up and pgs clean... 2024-09-11T06:14:37.134 INFO:tasks.ceph.ceph_manager.cluster1:waiting for mgr available 2024-09-11T06:14:37.134 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 mgr dump --format=json 2024-09-11T06:14:37.135 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-11T06:14:37.513 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-11T06:14:37.530 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":5,"flags":0,"active_gid":4099,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":1083398066},{"type":"v1","addr":"172.21.15.6:6825","nonce":1083398066}]},"active_addr":"172.21.15.6:6825/1083398066","active_change":"2024-09-11T06:14:22.990165+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.6:0","nonce":3991997226}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.6:0","nonce":3356234235}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.6:0","nonce":1950614485}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.6:0","nonce":2227946910}]}]} 2024-09-11T06:14:37.532 INFO:tasks.ceph.ceph_manager.cluster1:mgr available! 2024-09-11T06:14:37.532 INFO:tasks.ceph.ceph_manager.cluster1:waiting for all up 2024-09-11T06:14:37.532 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-09-11T06:14:37.826 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-11T06:14:37.826 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":16,"fsid":"c4197d0c-e10d-4a46-b3cc-95a1d7af9683","created":"2024-09-11T06:14:17.992688+0000","modified":"2024-09-11T06:14:37.102430+0000","last_up_change":"2024-09-11T06:14:31.066451+0000","last_in_change":"2024-09-11T06:14:18.910683+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-11T06:14:31.066308+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-11T06:14:33.321843+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9351a1cd-aa93-4d78-a30d-8a0d7a3285f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":2281847065},{"type":"v1","addr":"172.21.15.6:6809","nonce":2281847065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":2281847065},{"type":"v1","addr":"172.21.15.6:6811","nonce":2281847065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":2281847065},{"type":"v1","addr":"172.21.15.6:6815","nonce":2281847065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":2281847065},{"type":"v1","addr":"172.21.15.6:6813","nonce":2281847065}]},"public_addr":"172.21.15.6:6809/2281847065","cluster_addr":"172.21.15.6:6811/2281847065","heartbeat_back_addr":"172.21.15.6:6815/2281847065","heartbeat_front_addr":"172.21.15.6:6813/2281847065","state":["exists","up"]},{"osd":1,"uuid":"d43f49e2-ee49-43c1-8a34-5a967034a8ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":3363216727},{"type":"v1","addr":"172.21.15.6:6801","nonce":3363216727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":3363216727},{"type":"v1","addr":"172.21.15.6:6803","nonce":3363216727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":3363216727},{"type":"v1","addr":"172.21.15.6:6807","nonce":3363216727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":3363216727},{"type":"v1","addr":"172.21.15.6:6805","nonce":3363216727}]},"public_addr":"172.21.15.6:6801/3363216727","cluster_addr":"172.21.15.6:6803/3363216727","heartbeat_back_addr":"172.21.15.6:6807/3363216727","heartbeat_front_addr":"172.21.15.6:6805/3363216727","state":["exists","up"]},{"osd":2,"uuid":"fd7eafd1-7f01-4be9-8977-8a7fad1990da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":3564543485},{"type":"v1","addr":"172.21.15.6:6817","nonce":3564543485}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":3564543485},{"type":"v1","addr":"172.21.15.6:6819","nonce":3564543485}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":3564543485},{"type":"v1","addr":"172.21.15.6:6823","nonce":3564543485}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":3564543485},{"type":"v1","addr":"172.21.15.6:6821","nonce":3564543485}]},"public_addr":"172.21.15.6:6817/3564543485","cluster_addr":"172.21.15.6:6819/3564543485","heartbeat_back_addr":"172.21.15.6:6823/3564543485","heartbeat_front_addr":"172.21.15.6:6821/3564543485","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T06:14:25.351853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T06:14:25.565559+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T06:14:26.221801+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T06:14:37.844 INFO:tasks.ceph.ceph_manager.cluster1:all up! 2024-09-11T06:14:37.844 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-09-11T06:14:38.129 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-11T06:14:38.129 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":16,"fsid":"c4197d0c-e10d-4a46-b3cc-95a1d7af9683","created":"2024-09-11T06:14:17.992688+0000","modified":"2024-09-11T06:14:37.102430+0000","last_up_change":"2024-09-11T06:14:31.066451+0000","last_in_change":"2024-09-11T06:14:18.910683+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-11T06:14:31.066308+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-11T06:14:33.321843+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9351a1cd-aa93-4d78-a30d-8a0d7a3285f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":2281847065},{"type":"v1","addr":"172.21.15.6:6809","nonce":2281847065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":2281847065},{"type":"v1","addr":"172.21.15.6:6811","nonce":2281847065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":2281847065},{"type":"v1","addr":"172.21.15.6:6815","nonce":2281847065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":2281847065},{"type":"v1","addr":"172.21.15.6:6813","nonce":2281847065}]},"public_addr":"172.21.15.6:6809/2281847065","cluster_addr":"172.21.15.6:6811/2281847065","heartbeat_back_addr":"172.21.15.6:6815/2281847065","heartbeat_front_addr":"172.21.15.6:6813/2281847065","state":["exists","up"]},{"osd":1,"uuid":"d43f49e2-ee49-43c1-8a34-5a967034a8ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":3363216727},{"type":"v1","addr":"172.21.15.6:6801","nonce":3363216727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":3363216727},{"type":"v1","addr":"172.21.15.6:6803","nonce":3363216727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":3363216727},{"type":"v1","addr":"172.21.15.6:6807","nonce":3363216727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":3363216727},{"type":"v1","addr":"172.21.15.6:6805","nonce":3363216727}]},"public_addr":"172.21.15.6:6801/3363216727","cluster_addr":"172.21.15.6:6803/3363216727","heartbeat_back_addr":"172.21.15.6:6807/3363216727","heartbeat_front_addr":"172.21.15.6:6805/3363216727","state":["exists","up"]},{"osd":2,"uuid":"fd7eafd1-7f01-4be9-8977-8a7fad1990da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":3564543485},{"type":"v1","addr":"172.21.15.6:6817","nonce":3564543485}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":3564543485},{"type":"v1","addr":"172.21.15.6:6819","nonce":3564543485}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":3564543485},{"type":"v1","addr":"172.21.15.6:6823","nonce":3564543485}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":3564543485},{"type":"v1","addr":"172.21.15.6:6821","nonce":3564543485}]},"public_addr":"172.21.15.6:6817/3564543485","cluster_addr":"172.21.15.6:6819/3564543485","heartbeat_back_addr":"172.21.15.6:6823/3564543485","heartbeat_front_addr":"172.21.15.6:6821/3564543485","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T06:14:25.351853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T06:14:25.565559+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T06:14:26.221801+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T06:14:38.146 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.0 flush_pg_stats 2024-09-11T06:14:38.146 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.1 flush_pg_stats 2024-09-11T06:14:38.146 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.2 flush_pg_stats 2024-09-11T06:14:38.302 INFO:teuthology.orchestra.run.smithi006.stdout:42949672966 2024-09-11T06:14:38.303 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.2 2024-09-11T06:14:38.304 INFO:teuthology.orchestra.run.smithi006.stdout:38654705670 2024-09-11T06:14:38.304 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0 2024-09-11T06:14:38.318 INFO:teuthology.orchestra.run.smithi006.stdout:38654705672 2024-09-11T06:14:38.318 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.1 2024-09-11T06:14:38.597 INFO:teuthology.orchestra.run.smithi006.stdout:42949672964 2024-09-11T06:14:38.614 INFO:tasks.ceph.ceph_manager.cluster1:need seq 42949672966 got 42949672964 for osd.2 2024-09-11T06:14:38.649 INFO:teuthology.orchestra.run.smithi006.stdout:38654705669 2024-09-11T06:14:38.651 INFO:teuthology.orchestra.run.smithi006.stdout:38654705671 2024-09-11T06:14:38.668 INFO:tasks.ceph.ceph_manager.cluster1:need seq 38654705670 got 38654705669 for osd.0 2024-09-11T06:14:38.670 INFO:tasks.ceph.ceph_manager.cluster1:need seq 38654705672 got 38654705671 for osd.1 2024-09-11T06:14:39.615 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.2 2024-09-11T06:14:39.669 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0 2024-09-11T06:14:39.671 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.1 2024-09-11T06:14:39.960 INFO:teuthology.orchestra.run.smithi006.stdout:38654705670 2024-09-11T06:14:39.972 INFO:teuthology.orchestra.run.smithi006.stdout:42949672966 2024-09-11T06:14:39.977 INFO:tasks.ceph.ceph_manager.cluster1:need seq 38654705670 got 38654705670 for osd.0 2024-09-11T06:14:39.977 DEBUG:teuthology.parallel:result is None 2024-09-11T06:14:39.989 INFO:tasks.ceph.ceph_manager.cluster1:need seq 42949672966 got 42949672966 for osd.2 2024-09-11T06:14:39.990 DEBUG:teuthology.parallel:result is None 2024-09-11T06:14:40.003 INFO:teuthology.orchestra.run.smithi006.stdout:38654705672 2024-09-11T06:14:40.020 INFO:tasks.ceph.ceph_manager.cluster1:need seq 38654705672 got 38654705672 for osd.1 2024-09-11T06:14:40.020 DEBUG:teuthology.parallel:result is None 2024-09-11T06:14:40.020 INFO:tasks.ceph.ceph_manager.cluster1:waiting for clean 2024-09-11T06:14:40.020 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-11T06:14:40.350 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-11T06:14:40.351 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-09-11T06:14:40.368 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2024-09-11T06:14:38.996800+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":81608,"kb_used_data":520,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283033912,"statfs":{"total":289910292480,"available":289826725888,"internally_reserved":0,"allocated":532480,"data_stored":1297971,"data_compressed":7456,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4785,"internal_metadata":82963791},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.900222"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T06:14:37.124609+0000","last_change":"2024-09-11T06:14:37.124927+0000","last_active":"2024-09-11T06:14:37.124609+0000","last_peered":"2024-09-11T06:14:37.124609+0000","last_clean":"2024-09-11T06:14:37.124609+0000","last_became_active":"2024-09-11T06:14:35.103646+0000","last_became_peered":"2024-09-11T06:14:35.103646+0000","last_unstale":"2024-09-11T06:14:37.124609+0000","last_undegraded":"2024-09-11T06:14:37.124609+0000","last_fullsized":"2024-09-11T06:14:37.124609+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:14:34.076871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:14:34.076871+0000","last_clean_scrub_stamp":"2024-09-11T06:14:34.076871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:17:56.558226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00180295,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T06:14:37.124317+0000","last_change":"2024-09-11T06:14:37.124816+0000","last_active":"2024-09-11T06:14:37.124317+0000","last_peered":"2024-09-11T06:14:37.124317+0000","last_clean":"2024-09-11T06:14:37.124317+0000","last_became_active":"2024-09-11T06:14:35.103523+0000","last_became_peered":"2024-09-11T06:14:35.103523+0000","last_unstale":"2024-09-11T06:14:37.124317+0000","last_undegraded":"2024-09-11T06:14:37.124317+0000","last_fullsized":"2024-09-11T06:14:37.124317+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:14:34.076871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:14:34.076871+0000","last_clean_scrub_stamp":"2024-09-11T06:14:34.076871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:49:06.803512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016837309999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T06:14:37.124952+0000","last_change":"2024-09-11T06:14:37.125460+0000","last_active":"2024-09-11T06:14:37.124952+0000","last_peered":"2024-09-11T06:14:37.124952+0000","last_clean":"2024-09-11T06:14:37.124952+0000","last_became_active":"2024-09-11T06:14:35.103464+0000","last_became_peered":"2024-09-11T06:14:35.103464+0000","last_unstale":"2024-09-11T06:14:37.124952+0000","last_undegraded":"2024-09-11T06:14:37.124952+0000","last_fullsized":"2024-09-11T06:14:37.124952+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:14:34.076871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:14:34.076871+0000","last_clean_scrub_stamp":"2024-09-11T06:14:34.076871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T17:30:58.380858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022336330000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T06:14:37.123479+0000","last_change":"2024-09-11T06:14:37.124096+0000","last_active":"2024-09-11T06:14:37.123479+0000","last_peered":"2024-09-11T06:14:37.123479+0000","last_clean":"2024-09-11T06:14:37.123479+0000","last_became_active":"2024-09-11T06:14:35.101427+0000","last_became_peered":"2024-09-11T06:14:35.101427+0000","last_unstale":"2024-09-11T06:14:37.123479+0000","last_undegraded":"2024-09-11T06:14:37.123479+0000","last_fullsized":"2024-09-11T06:14:37.123479+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:14:34.076871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:14:34.076871+0000","last_clean_scrub_stamp":"2024-09-11T06:14:34.076871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:37:23.166823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001085391,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"16'2","reported_seq":21,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T06:14:37.115786+0000","last_change":"2024-09-11T06:14:37.115786+0000","last_active":"2024-09-11T06:14:37.115786+0000","last_peered":"2024-09-11T06:14:37.115786+0000","last_clean":"2024-09-11T06:14:37.115786+0000","last_became_active":"2024-09-11T06:14:35.105259+0000","last_became_peered":"2024-09-11T06:14:35.105259+0000","last_unstale":"2024-09-11T06:14:37.115786+0000","last_undegraded":"2024-09-11T06:14:37.115786+0000","last_fullsized":"2024-09-11T06:14:37.115786+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:14:34.076871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:14:34.076871+0000","last_clean_scrub_stamp":"2024-09-11T06:14:34.076871+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:33:55.339306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092280599999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T06:14:37.342578+0000","last_change":"2024-09-11T06:14:37.343308+0000","last_active":"2024-09-11T06:14:37.342578+0000","last_peered":"2024-09-11T06:14:37.342578+0000","last_clean":"2024-09-11T06:14:37.342578+0000","last_became_active":"2024-09-11T06:14:35.104976+0000","last_became_peered":"2024-09-11T06:14:35.104976+0000","last_unstale":"2024-09-11T06:14:37.342578+0000","last_undegraded":"2024-09-11T06:14:37.342578+0000","last_fullsized":"2024-09-11T06:14:37.342578+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:14:34.076871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:14:34.076871+0000","last_clean_scrub_stamp":"2024-09-11T06:14:34.076871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:33:14.682684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001060088,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T06:14:37.342705+0000","last_change":"2024-09-11T06:14:37.343210+0000","last_active":"2024-09-11T06:14:37.342705+0000","last_peered":"2024-09-11T06:14:37.342705+0000","last_clean":"2024-09-11T06:14:37.342705+0000","last_became_active":"2024-09-11T06:14:35.104951+0000","last_became_peered":"2024-09-11T06:14:35.104951+0000","last_unstale":"2024-09-11T06:14:37.342705+0000","last_undegraded":"2024-09-11T06:14:37.342705+0000","last_fullsized":"2024-09-11T06:14:37.342705+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:14:34.076871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:14:34.076871+0000","last_clean_scrub_stamp":"2024-09-11T06:14:34.076871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:18:43.513138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087051999999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"14'1","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T06:14:37.123580+0000","last_change":"2024-09-11T06:14:37.124468+0000","last_active":"2024-09-11T06:14:37.123580+0000","last_peered":"2024-09-11T06:14:37.123580+0000","last_clean":"2024-09-11T06:14:37.123580+0000","last_became_active":"2024-09-11T06:14:35.099575+0000","last_became_peered":"2024-09-11T06:14:35.099575+0000","last_unstale":"2024-09-11T06:14:37.123580+0000","last_undegraded":"2024-09-11T06:14:37.123580+0000","last_fullsized":"2024-09-11T06:14:37.123580+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:14:34.076871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:14:34.076871+0000","last_clean_scrub_stamp":"2024-09-11T06:14:34.076871+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:34:43.189610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010604620000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"11'97","reported_seq":147,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T06:14:37.122658+0000","last_change":"2024-09-11T06:14:32.080441+0000","last_active":"2024-09-11T06:14:37.122658+0000","last_peered":"2024-09-11T06:14:37.122658+0000","last_clean":"2024-09-11T06:14:37.122658+0000","last_became_active":"2024-09-11T06:14:32.077706+0000","last_became_peered":"2024-09-11T06:14:32.077706+0000","last_unstale":"2024-09-11T06:14:37.122658+0000","last_undegraded":"2024-09-11T06:14:37.122658+0000","last_fullsized":"2024-09-11T06:14:37.122658+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:14:31.066451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:14:31.066451+0000","last_clean_scrub_stamp":"2024-09-11T06:14:31.066451+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:50:25.120392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7456,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949672966,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27160,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344680,"statfs":{"total":96636764160,"available":96608952320,"internally_reserved":0,"allocated":131072,"data_stored":36691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705672,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":200704,"data_stored":630640,"data_compressed":3728,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705670,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":200704,"data_stored":630640,"data_compressed":3728,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3728,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3728,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T06:14:40.369 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-11T06:14:40.657 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-11T06:14:40.657 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-09-11T06:14:40.674 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2024-09-11T06:14:38.996800+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":81608,"kb_used_data":520,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283033912,"statfs":{"total":289910292480,"available":289826725888,"internally_reserved":0,"allocated":532480,"data_stored":1297971,"data_compressed":7456,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4785,"internal_metadata":82963791},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.900222"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T06:14:37.124609+0000","last_change":"2024-09-11T06:14:37.124927+0000","last_active":"2024-09-11T06:14:37.124609+0000","last_peered":"2024-09-11T06:14:37.124609+0000","last_clean":"2024-09-11T06:14:37.124609+0000","last_became_active":"2024-09-11T06:14:35.103646+0000","last_became_peered":"2024-09-11T06:14:35.103646+0000","last_unstale":"2024-09-11T06:14:37.124609+0000","last_undegraded":"2024-09-11T06:14:37.124609+0000","last_fullsized":"2024-09-11T06:14:37.124609+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:14:34.076871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:14:34.076871+0000","last_clean_scrub_stamp":"2024-09-11T06:14:34.076871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:17:56.558226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00180295,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T06:14:37.124317+0000","last_change":"2024-09-11T06:14:37.124816+0000","last_active":"2024-09-11T06:14:37.124317+0000","last_peered":"2024-09-11T06:14:37.124317+0000","last_clean":"2024-09-11T06:14:37.124317+0000","last_became_active":"2024-09-11T06:14:35.103523+0000","last_became_peered":"2024-09-11T06:14:35.103523+0000","last_unstale":"2024-09-11T06:14:37.124317+0000","last_undegraded":"2024-09-11T06:14:37.124317+0000","last_fullsized":"2024-09-11T06:14:37.124317+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:14:34.076871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:14:34.076871+0000","last_clean_scrub_stamp":"2024-09-11T06:14:34.076871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:49:06.803512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016837309999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T06:14:37.124952+0000","last_change":"2024-09-11T06:14:37.125460+0000","last_active":"2024-09-11T06:14:37.124952+0000","last_peered":"2024-09-11T06:14:37.124952+0000","last_clean":"2024-09-11T06:14:37.124952+0000","last_became_active":"2024-09-11T06:14:35.103464+0000","last_became_peered":"2024-09-11T06:14:35.103464+0000","last_unstale":"2024-09-11T06:14:37.124952+0000","last_undegraded":"2024-09-11T06:14:37.124952+0000","last_fullsized":"2024-09-11T06:14:37.124952+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:14:34.076871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:14:34.076871+0000","last_clean_scrub_stamp":"2024-09-11T06:14:34.076871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T17:30:58.380858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022336330000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T06:14:37.123479+0000","last_change":"2024-09-11T06:14:37.124096+0000","last_active":"2024-09-11T06:14:37.123479+0000","last_peered":"2024-09-11T06:14:37.123479+0000","last_clean":"2024-09-11T06:14:37.123479+0000","last_became_active":"2024-09-11T06:14:35.101427+0000","last_became_peered":"2024-09-11T06:14:35.101427+0000","last_unstale":"2024-09-11T06:14:37.123479+0000","last_undegraded":"2024-09-11T06:14:37.123479+0000","last_fullsized":"2024-09-11T06:14:37.123479+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:14:34.076871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:14:34.076871+0000","last_clean_scrub_stamp":"2024-09-11T06:14:34.076871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:37:23.166823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001085391,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"16'2","reported_seq":21,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T06:14:37.115786+0000","last_change":"2024-09-11T06:14:37.115786+0000","last_active":"2024-09-11T06:14:37.115786+0000","last_peered":"2024-09-11T06:14:37.115786+0000","last_clean":"2024-09-11T06:14:37.115786+0000","last_became_active":"2024-09-11T06:14:35.105259+0000","last_became_peered":"2024-09-11T06:14:35.105259+0000","last_unstale":"2024-09-11T06:14:37.115786+0000","last_undegraded":"2024-09-11T06:14:37.115786+0000","last_fullsized":"2024-09-11T06:14:37.115786+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:14:34.076871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:14:34.076871+0000","last_clean_scrub_stamp":"2024-09-11T06:14:34.076871+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:33:55.339306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092280599999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T06:14:37.342578+0000","last_change":"2024-09-11T06:14:37.343308+0000","last_active":"2024-09-11T06:14:37.342578+0000","last_peered":"2024-09-11T06:14:37.342578+0000","last_clean":"2024-09-11T06:14:37.342578+0000","last_became_active":"2024-09-11T06:14:35.104976+0000","last_became_peered":"2024-09-11T06:14:35.104976+0000","last_unstale":"2024-09-11T06:14:37.342578+0000","last_undegraded":"2024-09-11T06:14:37.342578+0000","last_fullsized":"2024-09-11T06:14:37.342578+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:14:34.076871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:14:34.076871+0000","last_clean_scrub_stamp":"2024-09-11T06:14:34.076871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:33:14.682684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001060088,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T06:14:37.342705+0000","last_change":"2024-09-11T06:14:37.343210+0000","last_active":"2024-09-11T06:14:37.342705+0000","last_peered":"2024-09-11T06:14:37.342705+0000","last_clean":"2024-09-11T06:14:37.342705+0000","last_became_active":"2024-09-11T06:14:35.104951+0000","last_became_peered":"2024-09-11T06:14:35.104951+0000","last_unstale":"2024-09-11T06:14:37.342705+0000","last_undegraded":"2024-09-11T06:14:37.342705+0000","last_fullsized":"2024-09-11T06:14:37.342705+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:14:34.076871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:14:34.076871+0000","last_clean_scrub_stamp":"2024-09-11T06:14:34.076871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:18:43.513138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087051999999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"14'1","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T06:14:37.123580+0000","last_change":"2024-09-11T06:14:37.124468+0000","last_active":"2024-09-11T06:14:37.123580+0000","last_peered":"2024-09-11T06:14:37.123580+0000","last_clean":"2024-09-11T06:14:37.123580+0000","last_became_active":"2024-09-11T06:14:35.099575+0000","last_became_peered":"2024-09-11T06:14:35.099575+0000","last_unstale":"2024-09-11T06:14:37.123580+0000","last_undegraded":"2024-09-11T06:14:37.123580+0000","last_fullsized":"2024-09-11T06:14:37.123580+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:14:34.076871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:14:34.076871+0000","last_clean_scrub_stamp":"2024-09-11T06:14:34.076871+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:34:43.189610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010604620000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"11'97","reported_seq":147,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-11T06:14:37.122658+0000","last_change":"2024-09-11T06:14:32.080441+0000","last_active":"2024-09-11T06:14:37.122658+0000","last_peered":"2024-09-11T06:14:37.122658+0000","last_clean":"2024-09-11T06:14:37.122658+0000","last_became_active":"2024-09-11T06:14:32.077706+0000","last_became_peered":"2024-09-11T06:14:32.077706+0000","last_unstale":"2024-09-11T06:14:37.122658+0000","last_undegraded":"2024-09-11T06:14:37.122658+0000","last_fullsized":"2024-09-11T06:14:37.122658+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:14:31.066451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:14:31.066451+0000","last_clean_scrub_stamp":"2024-09-11T06:14:31.066451+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:50:25.120392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7456,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949672966,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27160,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344680,"statfs":{"total":96636764160,"available":96608952320,"internally_reserved":0,"allocated":131072,"data_stored":36691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705672,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":200704,"data_stored":630640,"data_compressed":3728,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705670,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":200704,"data_stored":630640,"data_compressed":3728,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3728,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3728,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T06:14:40.674 INFO:tasks.ceph.ceph_manager.cluster1:clean! 2024-09-11T06:14:40.674 INFO:tasks.ceph:Waiting until ceph cluster cluster1 is healthy... 2024-09-11T06:14:40.675 INFO:tasks.ceph.ceph_manager.cluster1:wait_until_healthy 2024-09-11T06:14:40.675 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 health --format=json 2024-09-11T06:14:41.000 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-11T06:14:41.000 INFO:teuthology.orchestra.run.smithi006.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-11T06:14:41.020 INFO:tasks.ceph.ceph_manager.cluster1:wait_until_healthy done 2024-09-11T06:14:41.020 INFO:teuthology.run_tasks:Running task ceph... 2024-09-11T06:14:41.030 INFO:tasks.ceph:Creating ceph cluster cluster2... 2024-09-11T06:14:41.030 INFO:tasks.ceph:config {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'cluster2', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-11T06:14:41.030 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-09-10_21:16:03-rbd-squid-distro-default-smithi/7900605', 'branch': 'squid', 'description': 'rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-zlib supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-stress-workunit-min-compat-client-octopus}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7900605', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'meta': [{'desc': 'run two ceph clusters and install rbd-mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': '2 ceph clusters with 1 mon and 3 osds each'}, {'desc': 'run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon'}], 'name': 'teuthology-2024-09-10_21:16:03-rbd-squid-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'squid'}, 'ceph': {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'squid', 'sha1': 'daaf78de334647fc0c800f4ac495cc9475a26bae'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 100, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'], ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6']], 'seed': 6175, 'sha1': 'b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc', 'sleep_before_teardown': 0, 'subset': '88/128', 'suite': 'rbd', 'suite_branch': 'squid', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_daaf78de334647fc0c800f4ac495cc9475a26bae/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': 'daaf78de334647fc0c800f4ac495cc9475a26bae', 'targets': {'smithi006.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBO0WRxR505R28vAPOaiTE3OCExO5dRsDtnACcGHqGLEups3p4DMCG5xRwBug/cQh6QcewlLrbWsCJohOpKXaaQg=', 'smithi112.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJfTs0r1xAi00gNyDf/PIuEnNrqSpWYDR6gneJW9eH9P26G5v230dQe7xH8BPEZImnabz26XbGX7EdknHM19Ris='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': 'b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc'}}, {'ceph': {'cluster': 'cluster1', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc'}}, {'ceph': {'cluster': 'cluster2', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc'}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror.4 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.5 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.6 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.4 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.5 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.6 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'workunit': {'clients': {'cluster1.client.mirror': ['rbd/rbd_mirror_stress.sh']}, 'env': {'CEPH_ARGS': '', 'RBD_MIRROR_MIN_COMPAT_CLIENT': 'octopus', 'RBD_MIRROR_USE_EXISTING_CLUSTER': '1'}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-10_21:16:03', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-11T06:14:41.030 DEBUG:teuthology.orchestra.run.smithi006:> install -d -m0755 -- /home/ubuntu/cephtest/cluster2.data 2024-09-11T06:14:41.033 DEBUG:teuthology.orchestra.run.smithi112:> install -d -m0755 -- /home/ubuntu/cephtest/cluster2.data 2024-09-11T06:14:41.039 DEBUG:teuthology.orchestra.run.smithi006:> sudo install -d -m0777 -- /var/run/ceph 2024-09-11T06:14:41.082 DEBUG:teuthology.orchestra.run.smithi112:> sudo install -d -m0777 -- /var/run/ceph 2024-09-11T06:14:41.095 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-11T06:14:41.096 DEBUG:teuthology.orchestra.run.smithi112:> dd if=/scratch_devs of=/dev/stdout 2024-09-11T06:14:41.142 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-11T06:14:41.143 DEBUG:teuthology.orchestra.run.smithi112:> stat /dev/vg_nvme/lv_1 2024-09-11T06:14:41.190 INFO:teuthology.orchestra.run.smithi112.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-11T06:14:41.191 INFO:teuthology.orchestra.run.smithi112.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T06:14:41.191 INFO:teuthology.orchestra.run.smithi112.stdout:Device: 5h/5d Inode: 886 Links: 1 2024-09-11T06:14:41.191 INFO:teuthology.orchestra.run.smithi112.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T06:14:41.191 INFO:teuthology.orchestra.run.smithi112.stdout:Access: 2024-09-11 06:08:28.280454069 +0000 2024-09-11T06:14:41.191 INFO:teuthology.orchestra.run.smithi112.stdout:Modify: 2024-09-11 06:08:28.276454105 +0000 2024-09-11T06:14:41.191 INFO:teuthology.orchestra.run.smithi112.stdout:Change: 2024-09-11 06:08:28.276454105 +0000 2024-09-11T06:14:41.191 INFO:teuthology.orchestra.run.smithi112.stdout: Birth: - 2024-09-11T06:14:41.191 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-11T06:14:41.245 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records in 2024-09-11T06:14:41.245 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records out 2024-09-11T06:14:41.245 INFO:teuthology.orchestra.run.smithi112.stderr:512 bytes copied, 0.000374893 s, 1.4 MB/s 2024-09-11T06:14:41.246 DEBUG:teuthology.orchestra.run.smithi112:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-11T06:14:41.296 DEBUG:teuthology.orchestra.run.smithi112:> stat /dev/vg_nvme/lv_2 2024-09-11T06:14:41.342 INFO:teuthology.orchestra.run.smithi112.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-11T06:14:41.342 INFO:teuthology.orchestra.run.smithi112.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T06:14:41.342 INFO:teuthology.orchestra.run.smithi112.stdout:Device: 5h/5d Inode: 901 Links: 1 2024-09-11T06:14:41.342 INFO:teuthology.orchestra.run.smithi112.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T06:14:41.343 INFO:teuthology.orchestra.run.smithi112.stdout:Access: 2024-09-11 06:08:28.976447853 +0000 2024-09-11T06:14:41.343 INFO:teuthology.orchestra.run.smithi112.stdout:Modify: 2024-09-11 06:08:28.744449925 +0000 2024-09-11T06:14:41.343 INFO:teuthology.orchestra.run.smithi112.stdout:Change: 2024-09-11 06:08:28.744449925 +0000 2024-09-11T06:14:41.343 INFO:teuthology.orchestra.run.smithi112.stdout: Birth: - 2024-09-11T06:14:41.343 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-11T06:14:41.396 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records in 2024-09-11T06:14:41.397 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records out 2024-09-11T06:14:41.397 INFO:teuthology.orchestra.run.smithi112.stderr:512 bytes copied, 0.000445363 s, 1.1 MB/s 2024-09-11T06:14:41.398 DEBUG:teuthology.orchestra.run.smithi112:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-11T06:14:41.447 DEBUG:teuthology.orchestra.run.smithi112:> stat /dev/vg_nvme/lv_3 2024-09-11T06:14:41.494 INFO:teuthology.orchestra.run.smithi112.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-11T06:14:41.494 INFO:teuthology.orchestra.run.smithi112.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T06:14:41.494 INFO:teuthology.orchestra.run.smithi112.stdout:Device: 5h/5d Inode: 919 Links: 1 2024-09-11T06:14:41.494 INFO:teuthology.orchestra.run.smithi112.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T06:14:41.494 INFO:teuthology.orchestra.run.smithi112.stdout:Access: 2024-09-11 06:08:29.448443637 +0000 2024-09-11T06:14:41.494 INFO:teuthology.orchestra.run.smithi112.stdout:Modify: 2024-09-11 06:08:29.212445745 +0000 2024-09-11T06:14:41.494 INFO:teuthology.orchestra.run.smithi112.stdout:Change: 2024-09-11 06:08:29.212445745 +0000 2024-09-11T06:14:41.494 INFO:teuthology.orchestra.run.smithi112.stdout: Birth: - 2024-09-11T06:14:41.494 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-11T06:14:41.548 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records in 2024-09-11T06:14:41.549 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records out 2024-09-11T06:14:41.549 INFO:teuthology.orchestra.run.smithi112.stderr:512 bytes copied, 0.000309276 s, 1.7 MB/s 2024-09-11T06:14:41.550 DEBUG:teuthology.orchestra.run.smithi112:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-11T06:14:41.599 DEBUG:teuthology.orchestra.run.smithi112:> stat /dev/vg_nvme/lv_4 2024-09-11T06:14:41.646 INFO:teuthology.orchestra.run.smithi112.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-11T06:14:41.646 INFO:teuthology.orchestra.run.smithi112.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T06:14:41.646 INFO:teuthology.orchestra.run.smithi112.stdout:Device: 5h/5d Inode: 934 Links: 1 2024-09-11T06:14:41.646 INFO:teuthology.orchestra.run.smithi112.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T06:14:41.647 INFO:teuthology.orchestra.run.smithi112.stdout:Access: 2024-09-11 06:08:29.904439563 +0000 2024-09-11T06:14:41.647 INFO:teuthology.orchestra.run.smithi112.stdout:Modify: 2024-09-11 06:08:29.688441492 +0000 2024-09-11T06:14:41.647 INFO:teuthology.orchestra.run.smithi112.stdout:Change: 2024-09-11 06:08:29.688441492 +0000 2024-09-11T06:14:41.647 INFO:teuthology.orchestra.run.smithi112.stdout: Birth: - 2024-09-11T06:14:41.647 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-11T06:14:41.701 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records in 2024-09-11T06:14:41.701 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records out 2024-09-11T06:14:41.701 INFO:teuthology.orchestra.run.smithi112.stderr:512 bytes copied, 0.000353129 s, 1.4 MB/s 2024-09-11T06:14:41.702 DEBUG:teuthology.orchestra.run.smithi112:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-11T06:14:41.751 INFO:tasks.ceph:osd dev map: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-11T06:14:41.751 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi112.front.sepia.ceph.com'): {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-11T06:14:41.751 INFO:tasks.ceph:Generating config... 2024-09-11T06:14:41.753 INFO:tasks.ceph:[client] debug rbd = 20 2024-09-11T06:14:41.753 INFO:tasks.ceph:[client] debug rbd_mirror = 15 2024-09-11T06:14:41.753 INFO:tasks.ceph:[client] log to stderr = False 2024-09-11T06:14:41.753 INFO:tasks.ceph:[client] rbd default features = 125 2024-09-11T06:14:41.753 INFO:tasks.ceph:[client.mirror.0] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-11T06:14:41.753 INFO:tasks.ceph:[client.mirror.0] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-11T06:14:41.753 INFO:tasks.ceph:[client.mirror.1] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-11T06:14:41.753 INFO:tasks.ceph:[client.mirror.1] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-11T06:14:41.753 INFO:tasks.ceph:[client.mirror.2] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-11T06:14:41.753 INFO:tasks.ceph:[client.mirror.2] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-11T06:14:41.753 INFO:tasks.ceph:[client.mirror.3] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-11T06:14:41.753 INFO:tasks.ceph:[client.mirror.3] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-11T06:14:41.753 INFO:tasks.ceph:[client.mirror.4] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-11T06:14:41.754 INFO:tasks.ceph:[client.mirror.4] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-11T06:14:41.754 INFO:tasks.ceph:[client.mirror.5] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-11T06:14:41.754 INFO:tasks.ceph:[client.mirror.5] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-11T06:14:41.754 INFO:tasks.ceph:[client.mirror.6] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-11T06:14:41.754 INFO:tasks.ceph:[client.mirror.6] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-11T06:14:41.754 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-09-11T06:14:41.754 INFO:tasks.ceph:[global] mon warn on pool no app = False 2024-09-11T06:14:41.754 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-09-11T06:14:41.754 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-11T06:14:41.754 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-11T06:14:41.754 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-11T06:14:41.754 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-11T06:14:41.754 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-11T06:14:41.754 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-11T06:14:41.755 INFO:tasks.ceph:[osd] bluestore compression algorithm = zlib 2024-09-11T06:14:41.755 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-09-11T06:14:41.755 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-11T06:14:41.755 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-09-11T06:14:41.755 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-09-11T06:14:41.755 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-11T06:14:41.755 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-11T06:14:41.755 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-09-11T06:14:41.755 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-11T06:14:41.755 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-11T06:14:41.755 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-11T06:14:41.755 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-11T06:14:41.755 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-11T06:14:41.756 INFO:tasks.ceph:Setting up cluster2.mon.a... 2024-09-11T06:14:41.756 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/cluster2.keyring 2024-09-11T06:14:41.826 INFO:teuthology.orchestra.run.smithi112.stdout:creating /etc/ceph/cluster2.keyring 2024-09-11T06:14:41.829 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/cluster2.keyring 2024-09-11T06:14:41.910 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2024-09-11T06:14:41.969 DEBUG:tasks.ceph:Ceph mon addresses: [('cluster2.mon.a', '172.21.15.112')] 2024-09-11T06:14:41.970 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.112', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125}, 'mon.a': {}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}} 2024-09-11T06:14:41.970 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-11T06:14:41.970 DEBUG:teuthology.orchestra.run.smithi112:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-11T06:14:42.018 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.112 --print /home/ubuntu/cephtest/cluster2.monmap 2024-09-11T06:14:42.095 INFO:teuthology.orchestra.run.smithi112.stdout:monmaptool: monmap file /home/ubuntu/cephtest/cluster2.monmap 2024-09-11T06:14:42.096 INFO:teuthology.orchestra.run.smithi112.stdout:monmaptool: generated fsid f5a0f59c-ac24-45be-8681-35322ac680b8 2024-09-11T06:14:42.096 INFO:teuthology.orchestra.run.smithi112.stdout:setting min_mon_release = quincy 2024-09-11T06:14:42.096 INFO:teuthology.orchestra.run.smithi112.stdout:epoch 0 2024-09-11T06:14:42.096 INFO:teuthology.orchestra.run.smithi112.stdout:fsid f5a0f59c-ac24-45be-8681-35322ac680b8 2024-09-11T06:14:42.096 INFO:teuthology.orchestra.run.smithi112.stdout:last_changed 2024-09-11T06:14:42.098714+0000 2024-09-11T06:14:42.096 INFO:teuthology.orchestra.run.smithi112.stdout:created 2024-09-11T06:14:42.098714+0000 2024-09-11T06:14:42.096 INFO:teuthology.orchestra.run.smithi112.stdout:min_mon_release 17 (quincy) 2024-09-11T06:14:42.096 INFO:teuthology.orchestra.run.smithi112.stdout:election_strategy: 1 2024-09-11T06:14:42.096 INFO:teuthology.orchestra.run.smithi112.stdout:0: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.a 2024-09-11T06:14:42.096 INFO:teuthology.orchestra.run.smithi112.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/cluster2.monmap (1 monitors) 2024-09-11T06:14:42.097 DEBUG:teuthology.orchestra.run.smithi112:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-11T06:14:42.146 INFO:tasks.ceph:Writing /etc/ceph/cluster2.conf for FSID f5a0f59c-ac24-45be-8681-35322ac680b8... 2024-09-11T06:14:42.147 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster2.conf && sudo chmod 0644 /etc/ceph/cluster2.conf > /dev/null 2024-09-11T06:14:42.150 DEBUG:teuthology.orchestra.run.smithi112:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster2.conf && sudo chmod 0644 /etc/ceph/cluster2.conf > /dev/null 2024-09-11T06:14:42.192 INFO:teuthology.orchestra.run.smithi006.stdout:[global] 2024-09-11T06:14:42.192 INFO:teuthology.orchestra.run.smithi006.stdout: chdir = "" 2024-09-11T06:14:42.192 INFO:teuthology.orchestra.run.smithi006.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-11T06:14:42.192 INFO:teuthology.orchestra.run.smithi006.stdout: auth supported = cephx 2024-09-11T06:14:42.192 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-11T06:14:42.192 INFO:teuthology.orchestra.run.smithi006.stdout: filestore xattr use omap = true 2024-09-11T06:14:42.192 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-11T06:14:42.192 INFO:teuthology.orchestra.run.smithi006.stdout: mon clock drift allowed = 1.000 2024-09-11T06:14:42.192 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-11T06:14:42.193 INFO:teuthology.orchestra.run.smithi006.stdout: osd crush chooseleaf type = 0 2024-09-11T06:14:42.193 INFO:teuthology.orchestra.run.smithi006.stdout: auth debug = true 2024-09-11T06:14:42.193 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-11T06:14:42.193 INFO:teuthology.orchestra.run.smithi006.stdout: ms die on old message = true 2024-09-11T06:14:42.193 INFO:teuthology.orchestra.run.smithi006.stdout: ms die on bug = true 2024-09-11T06:14:42.193 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-11T06:14:42.193 INFO:teuthology.orchestra.run.smithi006.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-11T06:14:42.193 INFO:teuthology.orchestra.run.smithi006.stdout: mon pg warn max object skew = 0 2024-09-11T06:14:42.193 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-11T06:14:42.193 INFO:teuthology.orchestra.run.smithi006.stdout: # disable pg_autoscaler by default for new pools 2024-09-11T06:14:42.193 INFO:teuthology.orchestra.run.smithi006.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-11T06:14:42.193 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-11T06:14:42.193 INFO:teuthology.orchestra.run.smithi006.stdout: osd pool default size = 2 2024-09-11T06:14:42.193 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-11T06:14:42.194 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd allow primary affinity = true 2024-09-11T06:14:42.194 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd allow pg remap = true 2024-09-11T06:14:42.194 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on legacy crush tunables = false 2024-09-11T06:14:42.194 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on crush straw calc version zero = false 2024-09-11T06:14:42.194 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on no sortbitwise = false 2024-09-11T06:14:42.194 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on osd down out interval zero = false 2024-09-11T06:14:42.194 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on too few osds = false 2024-09-11T06:14:42.194 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-11T06:14:42.194 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-11T06:14:42.194 INFO:teuthology.orchestra.run.smithi006.stdout: mon_allow_pool_size_one = true 2024-09-11T06:14:42.194 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-11T06:14:42.194 INFO:teuthology.orchestra.run.smithi006.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-11T06:14:42.194 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-11T06:14:42.194 INFO:teuthology.orchestra.run.smithi006.stdout: osd default data pool replay window = 5 2024-09-11T06:14:42.194 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-11T06:14:42.195 INFO:teuthology.orchestra.run.smithi006.stdout: mon allow pool delete = true 2024-09-11T06:14:42.195 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-11T06:14:42.195 INFO:teuthology.orchestra.run.smithi006.stdout: mon cluster log file level = debug 2024-09-11T06:14:42.195 INFO:teuthology.orchestra.run.smithi006.stdout: debug asserts on shutdown = true 2024-09-11T06:14:42.195 INFO:teuthology.orchestra.run.smithi006.stdout: mon health detail to clog = false 2024-09-11T06:14:42.195 INFO:teuthology.orchestra.run.smithi006.stdout: mon host = 172.21.15.112 2024-09-11T06:14:42.195 INFO:teuthology.orchestra.run.smithi006.stdout: mon client directed command retry = 5 2024-09-11T06:14:42.195 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on pool no app = False 2024-09-11T06:14:42.195 INFO:teuthology.orchestra.run.smithi006.stdout: ms inject socket failures = 5000 2024-09-11T06:14:42.195 INFO:teuthology.orchestra.run.smithi006.stdout: fsid = f5a0f59c-ac24-45be-8681-35322ac680b8 2024-09-11T06:14:42.195 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-11T06:14:42.195 INFO:teuthology.orchestra.run.smithi006.stdout:[osd] 2024-09-11T06:14:42.195 INFO:teuthology.orchestra.run.smithi006.stdout: osd journal size = 100 2024-09-11T06:14:42.195 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-11T06:14:42.195 INFO:teuthology.orchestra.run.smithi006.stdout: osd scrub load threshold = 5.0 2024-09-11T06:14:42.195 INFO:teuthology.orchestra.run.smithi006.stdout: osd scrub max interval = 600 2024-09-11T06:14:42.196 INFO:teuthology.orchestra.run.smithi006.stdout: osd mclock profile = high_recovery_ops 2024-09-11T06:14:42.196 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-11T06:14:42.196 INFO:teuthology.orchestra.run.smithi006.stdout: osd recover clone overlap = true 2024-09-11T06:14:42.196 INFO:teuthology.orchestra.run.smithi006.stdout: osd recovery max chunk = 1048576 2024-09-11T06:14:42.196 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-11T06:14:42.196 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug shutdown = true 2024-09-11T06:14:42.196 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug op order = true 2024-09-11T06:14:42.196 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug verify stray on activate = true 2024-09-11T06:14:42.196 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug trim objects = true 2024-09-11T06:14:42.196 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-11T06:14:42.196 INFO:teuthology.orchestra.run.smithi006.stdout: osd open classes on start = true 2024-09-11T06:14:42.196 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug pg log writeout = true 2024-09-11T06:14:42.196 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-11T06:14:42.197 INFO:teuthology.orchestra.run.smithi006.stdout: osd deep scrub update digest min age = 30 2024-09-11T06:14:42.197 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-11T06:14:42.197 INFO:teuthology.orchestra.run.smithi006.stdout: osd map max advance = 10 2024-09-11T06:14:42.197 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-11T06:14:42.197 INFO:teuthology.orchestra.run.smithi006.stdout: journal zero on create = true 2024-09-11T06:14:42.197 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-11T06:14:42.197 INFO:teuthology.orchestra.run.smithi006.stdout: filestore ondisk finisher threads = 3 2024-09-11T06:14:42.197 INFO:teuthology.orchestra.run.smithi006.stdout: filestore apply finisher threads = 3 2024-09-11T06:14:42.197 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-11T06:14:42.197 INFO:teuthology.orchestra.run.smithi006.stdout: bdev debug aio = true 2024-09-11T06:14:42.197 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug misdirected ops = true 2024-09-11T06:14:42.197 INFO:teuthology.orchestra.run.smithi006.stdout: bluestore block size = 96636764160 2024-09-11T06:14:42.197 INFO:teuthology.orchestra.run.smithi006.stdout: bluestore compression algorithm = zlib 2024-09-11T06:14:42.198 INFO:teuthology.orchestra.run.smithi006.stdout: bluestore compression mode = aggressive 2024-09-11T06:14:42.198 INFO:teuthology.orchestra.run.smithi006.stdout: bluestore fsck on mount = True 2024-09-11T06:14:42.198 INFO:teuthology.orchestra.run.smithi006.stdout: debug bluefs = 1/20 2024-09-11T06:14:42.198 INFO:teuthology.orchestra.run.smithi006.stdout: debug bluestore = 1/20 2024-09-11T06:14:42.198 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-09-11T06:14:42.198 INFO:teuthology.orchestra.run.smithi006.stdout: debug osd = 20 2024-09-11T06:14:42.198 INFO:teuthology.orchestra.run.smithi006.stdout: debug rocksdb = 4/10 2024-09-11T06:14:42.198 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd backfillfull_ratio = 0.85 2024-09-11T06:14:42.198 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd full ratio = 0.9 2024-09-11T06:14:42.198 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd nearfull ratio = 0.8 2024-09-11T06:14:42.198 INFO:teuthology.orchestra.run.smithi006.stdout: osd failsafe full ratio = 0.95 2024-09-11T06:14:42.198 INFO:teuthology.orchestra.run.smithi006.stdout: osd objectstore = bluestore 2024-09-11T06:14:42.198 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-11T06:14:42.198 INFO:teuthology.orchestra.run.smithi006.stdout:[mgr] 2024-09-11T06:14:42.199 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-09-11T06:14:42.199 INFO:teuthology.orchestra.run.smithi006.stdout: debug mgr = 20 2024-09-11T06:14:42.199 INFO:teuthology.orchestra.run.smithi006.stdout: debug mon = 20 2024-09-11T06:14:42.199 INFO:teuthology.orchestra.run.smithi006.stdout: debug auth = 20 2024-09-11T06:14:42.199 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min pgs per osd = 4 2024-09-11T06:14:42.199 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min bytes per osd = 10 2024-09-11T06:14:42.199 INFO:teuthology.orchestra.run.smithi006.stdout: mgr/telemetry/nag = false 2024-09-11T06:14:42.199 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-11T06:14:42.199 INFO:teuthology.orchestra.run.smithi006.stdout:[mon] 2024-09-11T06:14:42.199 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-09-11T06:14:42.199 INFO:teuthology.orchestra.run.smithi006.stdout: debug mon = 20 2024-09-11T06:14:42.199 INFO:teuthology.orchestra.run.smithi006.stdout: debug paxos = 20 2024-09-11T06:14:42.199 INFO:teuthology.orchestra.run.smithi006.stdout: debug auth = 20 2024-09-11T06:14:42.199 INFO:teuthology.orchestra.run.smithi006.stdout: mon data avail warn = 5 2024-09-11T06:14:42.200 INFO:teuthology.orchestra.run.smithi006.stdout: mon mgr mkfs grace = 240 2024-09-11T06:14:42.200 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min pgs per osd = 4 2024-09-11T06:14:42.200 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd reporter subtree level = osd 2024-09-11T06:14:42.200 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd prime pg temp = true 2024-09-11T06:14:42.200 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min bytes per osd = 10 2024-09-11T06:14:42.200 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-11T06:14:42.200 INFO:teuthology.orchestra.run.smithi006.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-11T06:14:42.200 INFO:teuthology.orchestra.run.smithi006.stdout: auth mon ticket ttl = 660 # 11m 2024-09-11T06:14:42.200 INFO:teuthology.orchestra.run.smithi006.stdout: auth service ticket ttl = 240 # 4m 2024-09-11T06:14:42.200 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-11T06:14:42.200 INFO:teuthology.orchestra.run.smithi006.stdout: # don't complain about insecure global_id in the test suite 2024-09-11T06:14:42.200 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-11T06:14:42.200 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-11T06:14:42.201 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-11T06:14:42.201 INFO:teuthology.orchestra.run.smithi006.stdout: # 1m isn't quite enough 2024-09-11T06:14:42.201 INFO:teuthology.orchestra.run.smithi006.stdout: mon_down_mkfs_grace = 2m 2024-09-11T06:14:42.201 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-11T06:14:42.201 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_filestore_osds = false 2024-09-11T06:14:42.201 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-11T06:14:42.201 INFO:teuthology.orchestra.run.smithi006.stdout:[client] 2024-09-11T06:14:42.201 INFO:teuthology.orchestra.run.smithi006.stdout: rgw cache enabled = true 2024-09-11T06:14:42.201 INFO:teuthology.orchestra.run.smithi006.stdout: rgw enable ops log = true 2024-09-11T06:14:42.201 INFO:teuthology.orchestra.run.smithi006.stdout: rgw enable usage log = true 2024-09-11T06:14:42.201 INFO:teuthology.orchestra.run.smithi006.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-11T06:14:42.201 INFO:teuthology.orchestra.run.smithi006.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-11T06:14:42.201 INFO:teuthology.orchestra.run.smithi006.stdout: debug rbd = 20 2024-09-11T06:14:42.201 INFO:teuthology.orchestra.run.smithi006.stdout: debug rbd_mirror = 15 2024-09-11T06:14:42.201 INFO:teuthology.orchestra.run.smithi006.stdout: log to stderr = False 2024-09-11T06:14:42.202 INFO:teuthology.orchestra.run.smithi006.stdout: rbd default features = 125 2024-09-11T06:14:42.202 INFO:teuthology.orchestra.run.smithi006.stdout:[mon.a] 2024-09-11T06:14:42.202 INFO:teuthology.orchestra.run.smithi006.stdout:[client.mirror.0] 2024-09-11T06:14:42.202 INFO:teuthology.orchestra.run.smithi006.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-11T06:14:42.202 INFO:teuthology.orchestra.run.smithi006.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-11T06:14:42.202 INFO:teuthology.orchestra.run.smithi006.stdout:[client.mirror.1] 2024-09-11T06:14:42.202 INFO:teuthology.orchestra.run.smithi006.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-11T06:14:42.202 INFO:teuthology.orchestra.run.smithi006.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-11T06:14:42.202 INFO:teuthology.orchestra.run.smithi006.stdout:[client.mirror.2] 2024-09-11T06:14:42.202 INFO:teuthology.orchestra.run.smithi006.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-11T06:14:42.202 INFO:teuthology.orchestra.run.smithi006.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-11T06:14:42.202 INFO:teuthology.orchestra.run.smithi006.stdout:[client.mirror.3] 2024-09-11T06:14:42.202 INFO:teuthology.orchestra.run.smithi006.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-11T06:14:42.202 INFO:teuthology.orchestra.run.smithi006.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-11T06:14:42.202 INFO:teuthology.orchestra.run.smithi006.stdout:[client.mirror.4] 2024-09-11T06:14:42.203 INFO:teuthology.orchestra.run.smithi006.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-11T06:14:42.203 INFO:teuthology.orchestra.run.smithi006.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-11T06:14:42.203 INFO:teuthology.orchestra.run.smithi006.stdout:[client.mirror.5] 2024-09-11T06:14:42.203 INFO:teuthology.orchestra.run.smithi006.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-11T06:14:42.203 INFO:teuthology.orchestra.run.smithi006.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-11T06:14:42.203 INFO:teuthology.orchestra.run.smithi006.stdout:[client.mirror.6] 2024-09-11T06:14:42.203 INFO:teuthology.orchestra.run.smithi006.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-11T06:14:42.203 INFO:teuthology.orchestra.run.smithi006.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-11T06:14:42.218 INFO:teuthology.orchestra.run.smithi112.stdout:[global] 2024-09-11T06:14:42.218 INFO:teuthology.orchestra.run.smithi112.stdout: chdir = "" 2024-09-11T06:14:42.219 INFO:teuthology.orchestra.run.smithi112.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-11T06:14:42.219 INFO:teuthology.orchestra.run.smithi112.stdout: auth supported = cephx 2024-09-11T06:14:42.219 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-11T06:14:42.219 INFO:teuthology.orchestra.run.smithi112.stdout: filestore xattr use omap = true 2024-09-11T06:14:42.219 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-11T06:14:42.219 INFO:teuthology.orchestra.run.smithi112.stdout: mon clock drift allowed = 1.000 2024-09-11T06:14:42.219 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-11T06:14:42.219 INFO:teuthology.orchestra.run.smithi112.stdout: osd crush chooseleaf type = 0 2024-09-11T06:14:42.219 INFO:teuthology.orchestra.run.smithi112.stdout: auth debug = true 2024-09-11T06:14:42.219 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-11T06:14:42.219 INFO:teuthology.orchestra.run.smithi112.stdout: ms die on old message = true 2024-09-11T06:14:42.219 INFO:teuthology.orchestra.run.smithi112.stdout: ms die on bug = true 2024-09-11T06:14:42.219 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-11T06:14:42.219 INFO:teuthology.orchestra.run.smithi112.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-11T06:14:42.220 INFO:teuthology.orchestra.run.smithi112.stdout: mon pg warn max object skew = 0 2024-09-11T06:14:42.220 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-11T06:14:42.220 INFO:teuthology.orchestra.run.smithi112.stdout: # disable pg_autoscaler by default for new pools 2024-09-11T06:14:42.220 INFO:teuthology.orchestra.run.smithi112.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-11T06:14:42.220 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-11T06:14:42.220 INFO:teuthology.orchestra.run.smithi112.stdout: osd pool default size = 2 2024-09-11T06:14:42.220 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-11T06:14:42.220 INFO:teuthology.orchestra.run.smithi112.stdout: mon osd allow primary affinity = true 2024-09-11T06:14:42.220 INFO:teuthology.orchestra.run.smithi112.stdout: mon osd allow pg remap = true 2024-09-11T06:14:42.220 INFO:teuthology.orchestra.run.smithi112.stdout: mon warn on legacy crush tunables = false 2024-09-11T06:14:42.220 INFO:teuthology.orchestra.run.smithi112.stdout: mon warn on crush straw calc version zero = false 2024-09-11T06:14:42.220 INFO:teuthology.orchestra.run.smithi112.stdout: mon warn on no sortbitwise = false 2024-09-11T06:14:42.220 INFO:teuthology.orchestra.run.smithi112.stdout: mon warn on osd down out interval zero = false 2024-09-11T06:14:42.221 INFO:teuthology.orchestra.run.smithi112.stdout: mon warn on too few osds = false 2024-09-11T06:14:42.221 INFO:teuthology.orchestra.run.smithi112.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-11T06:14:42.221 INFO:teuthology.orchestra.run.smithi112.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-11T06:14:42.221 INFO:teuthology.orchestra.run.smithi112.stdout: mon_allow_pool_size_one = true 2024-09-11T06:14:42.221 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-11T06:14:42.221 INFO:teuthology.orchestra.run.smithi112.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-11T06:14:42.221 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-11T06:14:42.221 INFO:teuthology.orchestra.run.smithi112.stdout: osd default data pool replay window = 5 2024-09-11T06:14:42.221 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-11T06:14:42.221 INFO:teuthology.orchestra.run.smithi112.stdout: mon allow pool delete = true 2024-09-11T06:14:42.221 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-11T06:14:42.221 INFO:teuthology.orchestra.run.smithi112.stdout: mon cluster log file level = debug 2024-09-11T06:14:42.221 INFO:teuthology.orchestra.run.smithi112.stdout: debug asserts on shutdown = true 2024-09-11T06:14:42.221 INFO:teuthology.orchestra.run.smithi112.stdout: mon health detail to clog = false 2024-09-11T06:14:42.222 INFO:teuthology.orchestra.run.smithi112.stdout: mon host = 172.21.15.112 2024-09-11T06:14:42.222 INFO:teuthology.orchestra.run.smithi112.stdout: mon client directed command retry = 5 2024-09-11T06:14:42.222 INFO:teuthology.orchestra.run.smithi112.stdout: mon warn on pool no app = False 2024-09-11T06:14:42.222 INFO:teuthology.orchestra.run.smithi112.stdout: ms inject socket failures = 5000 2024-09-11T06:14:42.222 INFO:teuthology.orchestra.run.smithi112.stdout: fsid = f5a0f59c-ac24-45be-8681-35322ac680b8 2024-09-11T06:14:42.222 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-11T06:14:42.222 INFO:teuthology.orchestra.run.smithi112.stdout:[osd] 2024-09-11T06:14:42.222 INFO:teuthology.orchestra.run.smithi112.stdout: osd journal size = 100 2024-09-11T06:14:42.222 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-11T06:14:42.222 INFO:teuthology.orchestra.run.smithi112.stdout: osd scrub load threshold = 5.0 2024-09-11T06:14:42.222 INFO:teuthology.orchestra.run.smithi112.stdout: osd scrub max interval = 600 2024-09-11T06:14:42.222 INFO:teuthology.orchestra.run.smithi112.stdout: osd mclock profile = high_recovery_ops 2024-09-11T06:14:42.222 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-11T06:14:42.222 INFO:teuthology.orchestra.run.smithi112.stdout: osd recover clone overlap = true 2024-09-11T06:14:42.223 INFO:teuthology.orchestra.run.smithi112.stdout: osd recovery max chunk = 1048576 2024-09-11T06:14:42.223 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-11T06:14:42.223 INFO:teuthology.orchestra.run.smithi112.stdout: osd debug shutdown = true 2024-09-11T06:14:42.223 INFO:teuthology.orchestra.run.smithi112.stdout: osd debug op order = true 2024-09-11T06:14:42.223 INFO:teuthology.orchestra.run.smithi112.stdout: osd debug verify stray on activate = true 2024-09-11T06:14:42.223 INFO:teuthology.orchestra.run.smithi112.stdout: osd debug trim objects = true 2024-09-11T06:14:42.223 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-11T06:14:42.223 INFO:teuthology.orchestra.run.smithi112.stdout: osd open classes on start = true 2024-09-11T06:14:42.223 INFO:teuthology.orchestra.run.smithi112.stdout: osd debug pg log writeout = true 2024-09-11T06:14:42.223 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-11T06:14:42.223 INFO:teuthology.orchestra.run.smithi112.stdout: osd deep scrub update digest min age = 30 2024-09-11T06:14:42.223 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-11T06:14:42.223 INFO:teuthology.orchestra.run.smithi112.stdout: osd map max advance = 10 2024-09-11T06:14:42.223 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-11T06:14:42.224 INFO:teuthology.orchestra.run.smithi112.stdout: journal zero on create = true 2024-09-11T06:14:42.224 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-11T06:14:42.224 INFO:teuthology.orchestra.run.smithi112.stdout: filestore ondisk finisher threads = 3 2024-09-11T06:14:42.224 INFO:teuthology.orchestra.run.smithi112.stdout: filestore apply finisher threads = 3 2024-09-11T06:14:42.224 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-11T06:14:42.224 INFO:teuthology.orchestra.run.smithi112.stdout: bdev debug aio = true 2024-09-11T06:14:42.224 INFO:teuthology.orchestra.run.smithi112.stdout: osd debug misdirected ops = true 2024-09-11T06:14:42.224 INFO:teuthology.orchestra.run.smithi112.stdout: bluestore block size = 96636764160 2024-09-11T06:14:42.224 INFO:teuthology.orchestra.run.smithi112.stdout: bluestore compression algorithm = zlib 2024-09-11T06:14:42.224 INFO:teuthology.orchestra.run.smithi112.stdout: bluestore compression mode = aggressive 2024-09-11T06:14:42.224 INFO:teuthology.orchestra.run.smithi112.stdout: bluestore fsck on mount = True 2024-09-11T06:14:42.224 INFO:teuthology.orchestra.run.smithi112.stdout: debug bluefs = 1/20 2024-09-11T06:14:42.224 INFO:teuthology.orchestra.run.smithi112.stdout: debug bluestore = 1/20 2024-09-11T06:14:42.225 INFO:teuthology.orchestra.run.smithi112.stdout: debug ms = 1 2024-09-11T06:14:42.225 INFO:teuthology.orchestra.run.smithi112.stdout: debug osd = 20 2024-09-11T06:14:42.225 INFO:teuthology.orchestra.run.smithi112.stdout: debug rocksdb = 4/10 2024-09-11T06:14:42.225 INFO:teuthology.orchestra.run.smithi112.stdout: mon osd backfillfull_ratio = 0.85 2024-09-11T06:14:42.225 INFO:teuthology.orchestra.run.smithi112.stdout: mon osd full ratio = 0.9 2024-09-11T06:14:42.225 INFO:teuthology.orchestra.run.smithi112.stdout: mon osd nearfull ratio = 0.8 2024-09-11T06:14:42.225 INFO:teuthology.orchestra.run.smithi112.stdout: osd failsafe full ratio = 0.95 2024-09-11T06:14:42.225 INFO:teuthology.orchestra.run.smithi112.stdout: osd objectstore = bluestore 2024-09-11T06:14:42.225 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-11T06:14:42.225 INFO:teuthology.orchestra.run.smithi112.stdout:[mgr] 2024-09-11T06:14:42.225 INFO:teuthology.orchestra.run.smithi112.stdout: debug ms = 1 2024-09-11T06:14:42.225 INFO:teuthology.orchestra.run.smithi112.stdout: debug mgr = 20 2024-09-11T06:14:42.225 INFO:teuthology.orchestra.run.smithi112.stdout: debug mon = 20 2024-09-11T06:14:42.225 INFO:teuthology.orchestra.run.smithi112.stdout: debug auth = 20 2024-09-11T06:14:42.226 INFO:teuthology.orchestra.run.smithi112.stdout: mon reweight min pgs per osd = 4 2024-09-11T06:14:42.226 INFO:teuthology.orchestra.run.smithi112.stdout: mon reweight min bytes per osd = 10 2024-09-11T06:14:42.226 INFO:teuthology.orchestra.run.smithi112.stdout: mgr/telemetry/nag = false 2024-09-11T06:14:42.226 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-11T06:14:42.226 INFO:teuthology.orchestra.run.smithi112.stdout:[mon] 2024-09-11T06:14:42.226 INFO:teuthology.orchestra.run.smithi112.stdout: debug ms = 1 2024-09-11T06:14:42.226 INFO:teuthology.orchestra.run.smithi112.stdout: debug mon = 20 2024-09-11T06:14:42.226 INFO:teuthology.orchestra.run.smithi112.stdout: debug paxos = 20 2024-09-11T06:14:42.226 INFO:teuthology.orchestra.run.smithi112.stdout: debug auth = 20 2024-09-11T06:14:42.226 INFO:teuthology.orchestra.run.smithi112.stdout: mon data avail warn = 5 2024-09-11T06:14:42.226 INFO:teuthology.orchestra.run.smithi112.stdout: mon mgr mkfs grace = 240 2024-09-11T06:14:42.226 INFO:teuthology.orchestra.run.smithi112.stdout: mon reweight min pgs per osd = 4 2024-09-11T06:14:42.226 INFO:teuthology.orchestra.run.smithi112.stdout: mon osd reporter subtree level = osd 2024-09-11T06:14:42.226 INFO:teuthology.orchestra.run.smithi112.stdout: mon osd prime pg temp = true 2024-09-11T06:14:42.226 INFO:teuthology.orchestra.run.smithi112.stdout: mon reweight min bytes per osd = 10 2024-09-11T06:14:42.227 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-11T06:14:42.227 INFO:teuthology.orchestra.run.smithi112.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-11T06:14:42.227 INFO:teuthology.orchestra.run.smithi112.stdout: auth mon ticket ttl = 660 # 11m 2024-09-11T06:14:42.227 INFO:teuthology.orchestra.run.smithi112.stdout: auth service ticket ttl = 240 # 4m 2024-09-11T06:14:42.227 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-11T06:14:42.227 INFO:teuthology.orchestra.run.smithi112.stdout: # don't complain about insecure global_id in the test suite 2024-09-11T06:14:42.227 INFO:teuthology.orchestra.run.smithi112.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-11T06:14:42.227 INFO:teuthology.orchestra.run.smithi112.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-11T06:14:42.227 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-11T06:14:42.227 INFO:teuthology.orchestra.run.smithi112.stdout: # 1m isn't quite enough 2024-09-11T06:14:42.227 INFO:teuthology.orchestra.run.smithi112.stdout: mon_down_mkfs_grace = 2m 2024-09-11T06:14:42.227 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-11T06:14:42.227 INFO:teuthology.orchestra.run.smithi112.stdout: mon_warn_on_filestore_osds = false 2024-09-11T06:14:42.227 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-11T06:14:42.228 INFO:teuthology.orchestra.run.smithi112.stdout:[client] 2024-09-11T06:14:42.228 INFO:teuthology.orchestra.run.smithi112.stdout: rgw cache enabled = true 2024-09-11T06:14:42.228 INFO:teuthology.orchestra.run.smithi112.stdout: rgw enable ops log = true 2024-09-11T06:14:42.228 INFO:teuthology.orchestra.run.smithi112.stdout: rgw enable usage log = true 2024-09-11T06:14:42.228 INFO:teuthology.orchestra.run.smithi112.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-11T06:14:42.228 INFO:teuthology.orchestra.run.smithi112.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-11T06:14:42.228 INFO:teuthology.orchestra.run.smithi112.stdout: debug rbd = 20 2024-09-11T06:14:42.228 INFO:teuthology.orchestra.run.smithi112.stdout: debug rbd_mirror = 15 2024-09-11T06:14:42.228 INFO:teuthology.orchestra.run.smithi112.stdout: log to stderr = False 2024-09-11T06:14:42.228 INFO:teuthology.orchestra.run.smithi112.stdout: rbd default features = 125 2024-09-11T06:14:42.228 INFO:teuthology.orchestra.run.smithi112.stdout:[mon.a] 2024-09-11T06:14:42.228 INFO:teuthology.orchestra.run.smithi112.stdout:[client.mirror.0] 2024-09-11T06:14:42.228 INFO:teuthology.orchestra.run.smithi112.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-11T06:14:42.228 INFO:teuthology.orchestra.run.smithi112.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-11T06:14:42.228 INFO:teuthology.orchestra.run.smithi112.stdout:[client.mirror.1] 2024-09-11T06:14:42.229 INFO:teuthology.orchestra.run.smithi112.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-11T06:14:42.229 INFO:teuthology.orchestra.run.smithi112.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-11T06:14:42.229 INFO:teuthology.orchestra.run.smithi112.stdout:[client.mirror.2] 2024-09-11T06:14:42.229 INFO:teuthology.orchestra.run.smithi112.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-11T06:14:42.229 INFO:teuthology.orchestra.run.smithi112.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-11T06:14:42.229 INFO:teuthology.orchestra.run.smithi112.stdout:[client.mirror.3] 2024-09-11T06:14:42.229 INFO:teuthology.orchestra.run.smithi112.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-11T06:14:42.229 INFO:teuthology.orchestra.run.smithi112.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-11T06:14:42.229 INFO:teuthology.orchestra.run.smithi112.stdout:[client.mirror.4] 2024-09-11T06:14:42.229 INFO:teuthology.orchestra.run.smithi112.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-11T06:14:42.229 INFO:teuthology.orchestra.run.smithi112.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-11T06:14:42.229 INFO:teuthology.orchestra.run.smithi112.stdout:[client.mirror.5] 2024-09-11T06:14:42.229 INFO:teuthology.orchestra.run.smithi112.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-11T06:14:42.229 INFO:teuthology.orchestra.run.smithi112.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-11T06:14:42.229 INFO:teuthology.orchestra.run.smithi112.stdout:[client.mirror.6] 2024-09-11T06:14:42.229 INFO:teuthology.orchestra.run.smithi112.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-11T06:14:42.230 INFO:teuthology.orchestra.run.smithi112.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-11T06:14:42.230 INFO:tasks.ceph:Creating admin key on cluster2.mon.a... 2024-09-11T06:14:42.230 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/cluster2.keyring 2024-09-11T06:14:42.322 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-11T06:14:42.322 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-11T06:14:42.322 DEBUG:teuthology.orchestra.run.smithi112:> dd if=/etc/ceph/cluster2.keyring of=/dev/stdout 2024-09-11T06:14:42.327 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-11T06:14:42.328 DEBUG:teuthology.orchestra.run.smithi112:> dd if=/home/ubuntu/cephtest/cluster2.monmap of=/dev/stdout 2024-09-11T06:14:42.374 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi006.front.sepia.ceph.com 2024-09-11T06:14:42.374 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-09-11T06:14:42.374 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/etc/ceph/cluster2.keyring 2024-09-11T06:14:42.374 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2024-09-11T06:14:42.398 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-09-11T06:14:42.398 DEBUG:teuthology.orchestra.run.smithi006:> dd of=/home/ubuntu/cephtest/cluster2.monmap 2024-09-11T06:14:42.445 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi112.front.sepia.ceph.com 2024-09-11T06:14:42.445 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-11T06:14:42.446 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/etc/ceph/cluster2.keyring 2024-09-11T06:14:42.446 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2024-09-11T06:14:42.469 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-11T06:14:42.469 DEBUG:teuthology.orchestra.run.smithi112:> dd of=/home/ubuntu/cephtest/cluster2.monmap 2024-09-11T06:14:42.518 INFO:tasks.ceph:Setting up mon nodes... 2024-09-11T06:14:42.518 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-11T06:14:42.519 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /var/lib/ceph/mgr/cluster2-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/cluster2-x/keyring 2024-09-11T06:14:42.564 INFO:teuthology.orchestra.run.smithi006.stdout:creating /var/lib/ceph/mgr/cluster2-x/keyring 2024-09-11T06:14:42.567 INFO:tasks.ceph:Setting up mds nodes... 2024-09-11T06:14:42.567 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-11T06:14:42.567 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/cluster2.client.0.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.0.keyring 2024-09-11T06:14:42.643 INFO:teuthology.orchestra.run.smithi006.stdout:creating /etc/ceph/cluster2.client.0.keyring 2024-09-11T06:14:42.654 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror /etc/ceph/cluster2.client.mirror.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.keyring 2024-09-11T06:14:42.691 INFO:teuthology.orchestra.run.smithi112.stdout:creating /etc/ceph/cluster2.client.mirror.keyring 2024-09-11T06:14:42.702 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.0 /etc/ceph/cluster2.client.mirror.0.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.0.keyring 2024-09-11T06:14:42.777 INFO:teuthology.orchestra.run.smithi112.stdout:creating /etc/ceph/cluster2.client.mirror.0.keyring 2024-09-11T06:14:42.787 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.1 /etc/ceph/cluster2.client.mirror.1.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.1.keyring 2024-09-11T06:14:42.858 INFO:teuthology.orchestra.run.smithi112.stdout:creating /etc/ceph/cluster2.client.mirror.1.keyring 2024-09-11T06:14:42.868 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.2 /etc/ceph/cluster2.client.mirror.2.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.2.keyring 2024-09-11T06:14:42.939 INFO:teuthology.orchestra.run.smithi112.stdout:creating /etc/ceph/cluster2.client.mirror.2.keyring 2024-09-11T06:14:42.949 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.3 /etc/ceph/cluster2.client.mirror.3.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.3.keyring 2024-09-11T06:14:43.029 INFO:teuthology.orchestra.run.smithi112.stdout:creating /etc/ceph/cluster2.client.mirror.3.keyring 2024-09-11T06:14:43.039 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.4 /etc/ceph/cluster2.client.mirror.4.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.4.keyring 2024-09-11T06:14:43.116 INFO:teuthology.orchestra.run.smithi112.stdout:creating /etc/ceph/cluster2.client.mirror.4.keyring 2024-09-11T06:14:43.126 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.5 /etc/ceph/cluster2.client.mirror.5.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.5.keyring 2024-09-11T06:14:43.200 INFO:teuthology.orchestra.run.smithi112.stdout:creating /etc/ceph/cluster2.client.mirror.5.keyring 2024-09-11T06:14:43.211 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.6 /etc/ceph/cluster2.client.mirror.6.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.6.keyring 2024-09-11T06:14:43.284 INFO:teuthology.orchestra.run.smithi112.stdout:creating /etc/ceph/cluster2.client.mirror.6.keyring 2024-09-11T06:14:43.294 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-11T06:14:43.294 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi006.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi112.front.sepia.ceph.com'): {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-11T06:14:43.294 DEBUG:teuthology.orchestra.run.smithi112:> sudo mkdir -p /var/lib/ceph/osd/cluster2-0 2024-09-11T06:14:43.348 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-11T06:14:43.348 INFO:tasks.ceph:role: cluster2.osd.0 2024-09-11T06:14:43.348 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi112.front.sepia.ceph.com 2024-09-11T06:14:43.348 DEBUG:teuthology.orchestra.run.smithi112:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-11T06:14:43.408 INFO:teuthology.orchestra.run.smithi112.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-11T06:14:43.408 INFO:teuthology.orchestra.run.smithi112.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-11T06:14:43.408 INFO:teuthology.orchestra.run.smithi112.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-11T06:14:43.408 INFO:teuthology.orchestra.run.smithi112.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-11T06:14:43.408 INFO:teuthology.orchestra.run.smithi112.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-11T06:14:43.408 INFO:teuthology.orchestra.run.smithi112.stdout: = sunit=0 swidth=0 blks 2024-09-11T06:14:43.408 INFO:teuthology.orchestra.run.smithi112.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-11T06:14:43.408 INFO:teuthology.orchestra.run.smithi112.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-11T06:14:43.408 INFO:teuthology.orchestra.run.smithi112.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-11T06:14:43.409 INFO:teuthology.orchestra.run.smithi112.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-11T06:14:43.889 INFO:teuthology.orchestra.run.smithi112.stdout:Discarding blocks...Done. 2024-09-11T06:14:43.891 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi112.front.sepia.ceph.com -o noatime 2024-09-11T06:14:43.891 DEBUG:teuthology.orchestra.run.smithi112:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/cluster2-0 2024-09-11T06:14:44.053 DEBUG:teuthology.orchestra.run.smithi112:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-0 2024-09-11T06:14:44.064 INFO:teuthology.orchestra.run.smithi112.stderr:sudo: /sbin/restorecon: command not found 2024-09-11T06:14:44.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T06:14:44.065 DEBUG:teuthology.orchestra.run.smithi112:> sudo mkdir -p /var/lib/ceph/osd/cluster2-1 2024-09-11T06:14:44.125 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-11T06:14:44.125 INFO:tasks.ceph:role: cluster2.osd.1 2024-09-11T06:14:44.125 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi112.front.sepia.ceph.com 2024-09-11T06:14:44.125 DEBUG:teuthology.orchestra.run.smithi112:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-11T06:14:44.186 INFO:teuthology.orchestra.run.smithi112.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-11T06:14:44.186 INFO:teuthology.orchestra.run.smithi112.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-11T06:14:44.186 INFO:teuthology.orchestra.run.smithi112.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-11T06:14:44.186 INFO:teuthology.orchestra.run.smithi112.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-11T06:14:44.186 INFO:teuthology.orchestra.run.smithi112.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-11T06:14:44.186 INFO:teuthology.orchestra.run.smithi112.stdout: = sunit=0 swidth=0 blks 2024-09-11T06:14:44.186 INFO:teuthology.orchestra.run.smithi112.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-11T06:14:44.186 INFO:teuthology.orchestra.run.smithi112.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-11T06:14:44.187 INFO:teuthology.orchestra.run.smithi112.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-11T06:14:44.187 INFO:teuthology.orchestra.run.smithi112.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-11T06:14:44.636 INFO:teuthology.orchestra.run.smithi112.stdout:Discarding blocks...Done. 2024-09-11T06:14:44.638 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi112.front.sepia.ceph.com -o noatime 2024-09-11T06:14:44.638 DEBUG:teuthology.orchestra.run.smithi112:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/cluster2-1 2024-09-11T06:14:44.731 DEBUG:teuthology.orchestra.run.smithi112:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-1 2024-09-11T06:14:44.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T06:14:44.741 INFO:teuthology.orchestra.run.smithi112.stderr:sudo: /sbin/restorecon: command not found 2024-09-11T06:14:44.741 DEBUG:teuthology.orchestra.run.smithi112:> sudo mkdir -p /var/lib/ceph/osd/cluster2-2 2024-09-11T06:14:44.801 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-11T06:14:44.801 INFO:tasks.ceph:role: cluster2.osd.2 2024-09-11T06:14:44.801 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi112.front.sepia.ceph.com 2024-09-11T06:14:44.801 DEBUG:teuthology.orchestra.run.smithi112:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-11T06:14:44.866 INFO:teuthology.orchestra.run.smithi112.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-11T06:14:44.866 INFO:teuthology.orchestra.run.smithi112.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-11T06:14:44.866 INFO:teuthology.orchestra.run.smithi112.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-11T06:14:44.866 INFO:teuthology.orchestra.run.smithi112.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-11T06:14:44.866 INFO:teuthology.orchestra.run.smithi112.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-11T06:14:44.866 INFO:teuthology.orchestra.run.smithi112.stdout: = sunit=0 swidth=0 blks 2024-09-11T06:14:44.866 INFO:teuthology.orchestra.run.smithi112.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-11T06:14:44.866 INFO:teuthology.orchestra.run.smithi112.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-11T06:14:44.866 INFO:teuthology.orchestra.run.smithi112.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-11T06:14:44.867 INFO:teuthology.orchestra.run.smithi112.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-11T06:14:45.319 INFO:teuthology.orchestra.run.smithi112.stdout:Discarding blocks...Done. 2024-09-11T06:14:45.321 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi112.front.sepia.ceph.com -o noatime 2024-09-11T06:14:45.322 DEBUG:teuthology.orchestra.run.smithi112:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/cluster2-2 2024-09-11T06:14:45.453 DEBUG:teuthology.orchestra.run.smithi112:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-2 2024-09-11T06:14:45.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T06:14:45.465 INFO:teuthology.orchestra.run.smithi112.stderr:sudo: /sbin/restorecon: command not found 2024-09-11T06:14:45.465 DEBUG:teuthology.orchestra.run.smithi112:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/cluster2.monmap 2024-09-11T06:14:45.670 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-11T06:14:45.671+0000 7f79148237c0 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-0/keyring: can't open /var/lib/ceph/osd/cluster2-0/keyring: (2) No such file or directory 2024-09-11T06:14:45.674 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-11T06:14:45.675+0000 7f79148237c0 -1 created new key in keyring /var/lib/ceph/osd/cluster2-0/keyring 2024-09-11T06:14:45.674 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-11T06:14:45.675+0000 7f79148237c0 -1 bdev(0x5591cd51b800 /var/lib/ceph/osd/cluster2-0/block) open stat got: (1) Operation not permitted 2024-09-11T06:14:45.674 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-11T06:14:45.675+0000 7f79148237c0 -1 bluestore(/var/lib/ceph/osd/cluster2-0) _read_fsid unparsable uuid 2024-09-11T06:14:48.826 DEBUG:teuthology.orchestra.run.smithi112:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-0 2024-09-11T06:14:48.840 DEBUG:teuthology.orchestra.run.smithi112:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/cluster2.monmap 2024-09-11T06:14:48.920 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-11T06:14:48.919+0000 7fe0662197c0 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-1/keyring: can't open /var/lib/ceph/osd/cluster2-1/keyring: (2) No such file or directory 2024-09-11T06:14:48.920 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-11T06:14:48.919+0000 7fe0662197c0 -1 created new key in keyring /var/lib/ceph/osd/cluster2-1/keyring 2024-09-11T06:14:48.921 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-11T06:14:48.919+0000 7fe0662197c0 -1 bdev(0x56346e1b5800 /var/lib/ceph/osd/cluster2-1/block) open stat got: (1) Operation not permitted 2024-09-11T06:14:48.921 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-11T06:14:48.919+0000 7fe0662197c0 -1 bluestore(/var/lib/ceph/osd/cluster2-1) _read_fsid unparsable uuid 2024-09-11T06:14:52.053 DEBUG:teuthology.orchestra.run.smithi112:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-1 2024-09-11T06:14:52.114 DEBUG:teuthology.orchestra.run.smithi112:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/cluster2.monmap 2024-09-11T06:14:52.196 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-11T06:14:52.195+0000 7f56a29817c0 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-2/keyring: can't open /var/lib/ceph/osd/cluster2-2/keyring: (2) No such file or directory 2024-09-11T06:14:52.197 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-11T06:14:52.195+0000 7f56a29817c0 -1 created new key in keyring /var/lib/ceph/osd/cluster2-2/keyring 2024-09-11T06:14:52.197 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-11T06:14:52.195+0000 7f56a29817c0 -1 bdev(0x556f27135800 /var/lib/ceph/osd/cluster2-2/block) open stat got: (1) Operation not permitted 2024-09-11T06:14:52.197 INFO:teuthology.orchestra.run.smithi112.stderr:2024-09-11T06:14:52.195+0000 7f56a29817c0 -1 bluestore(/var/lib/ceph/osd/cluster2-2) _read_fsid unparsable uuid 2024-09-11T06:14:55.349 DEBUG:teuthology.orchestra.run.smithi112:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-2 2024-09-11T06:14:55.406 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-11T06:14:55.406 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-09-11T06:14:55.406 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/mgr/cluster2-x/keyring of=/dev/stdout 2024-09-11T06:14:55.421 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-11T06:14:55.421 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/var/lib/ceph/osd/cluster2-0/keyring of=/dev/stdout 2024-09-11T06:14:55.460 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-11T06:14:55.460 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/var/lib/ceph/osd/cluster2-1/keyring of=/dev/stdout 2024-09-11T06:14:55.513 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-11T06:14:55.513 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/var/lib/ceph/osd/cluster2-2/keyring of=/dev/stdout 2024-09-11T06:14:55.569 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-09-11T06:14:55.569 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/etc/ceph/cluster2.client.0.keyring of=/dev/stdout 2024-09-11T06:14:55.575 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-11T06:14:55.575 DEBUG:teuthology.orchestra.run.smithi112:> dd if=/etc/ceph/cluster2.client.mirror.keyring of=/dev/stdout 2024-09-11T06:14:55.618 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-11T06:14:55.618 DEBUG:teuthology.orchestra.run.smithi112:> dd if=/etc/ceph/cluster2.client.mirror.0.keyring of=/dev/stdout 2024-09-11T06:14:55.666 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-11T06:14:55.666 DEBUG:teuthology.orchestra.run.smithi112:> dd if=/etc/ceph/cluster2.client.mirror.1.keyring of=/dev/stdout 2024-09-11T06:14:55.714 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-11T06:14:55.714 DEBUG:teuthology.orchestra.run.smithi112:> dd if=/etc/ceph/cluster2.client.mirror.2.keyring of=/dev/stdout 2024-09-11T06:14:55.762 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-11T06:14:55.762 DEBUG:teuthology.orchestra.run.smithi112:> dd if=/etc/ceph/cluster2.client.mirror.3.keyring of=/dev/stdout 2024-09-11T06:14:55.810 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-11T06:14:55.810 DEBUG:teuthology.orchestra.run.smithi112:> dd if=/etc/ceph/cluster2.client.mirror.4.keyring of=/dev/stdout 2024-09-11T06:14:55.858 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-11T06:14:55.858 DEBUG:teuthology.orchestra.run.smithi112:> dd if=/etc/ceph/cluster2.client.mirror.5.keyring of=/dev/stdout 2024-09-11T06:14:55.906 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-11T06:14:55.906 DEBUG:teuthology.orchestra.run.smithi112:> dd if=/etc/ceph/cluster2.client.mirror.6.keyring of=/dev/stdout 2024-09-11T06:14:55.954 INFO:tasks.ceph:Adding keys to all mons... 2024-09-11T06:14:55.954 DEBUG:teuthology.orchestra.run.smithi112:> sudo tee -a /etc/ceph/cluster2.keyring 2024-09-11T06:14:56.008 INFO:teuthology.orchestra.run.smithi112.stdout:[mgr.x] 2024-09-11T06:14:56.008 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQBSNeFmYz+oIRAAbtz4hU9UebwvSkL/kKpozQ== 2024-09-11T06:14:56.008 INFO:teuthology.orchestra.run.smithi112.stdout:[osd.0] 2024-09-11T06:14:56.009 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQBVNeFmiMIhKBAAVwXkoXS+9KRgfpVhRc+PqA== 2024-09-11T06:14:56.009 INFO:teuthology.orchestra.run.smithi112.stdout:[osd.1] 2024-09-11T06:14:56.009 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQBYNeFmQ4gDNxAAgqbdgy11s/EEOCP+penzLA== 2024-09-11T06:14:56.009 INFO:teuthology.orchestra.run.smithi112.stdout:[osd.2] 2024-09-11T06:14:56.009 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQBcNeFmJpffCxAAnKb3Iwoxg8P156uL51WSDQ== 2024-09-11T06:14:56.009 INFO:teuthology.orchestra.run.smithi112.stdout:[client.0] 2024-09-11T06:14:56.009 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQBSNeFm79hkJhAAoV22AfQfTBSS13iVC6a0oA== 2024-09-11T06:14:56.009 INFO:teuthology.orchestra.run.smithi112.stdout:[client.mirror] 2024-09-11T06:14:56.009 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQBSNeFm6oFmKRAAkxjreuWFM6Q0zObosK44zw== 2024-09-11T06:14:56.009 INFO:teuthology.orchestra.run.smithi112.stdout:[client.mirror.0] 2024-09-11T06:14:56.010 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQBSNeFmEKuJLhAAvnG5ZXNLcfgXZQB8KsAz5Q== 2024-09-11T06:14:56.010 INFO:teuthology.orchestra.run.smithi112.stdout:[client.mirror.1] 2024-09-11T06:14:56.010 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQBSNeFmZQlZMxAAOBZ/Yc+Xge4iPsnUKTQ+Uw== 2024-09-11T06:14:56.010 INFO:teuthology.orchestra.run.smithi112.stdout:[client.mirror.2] 2024-09-11T06:14:56.010 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQBSNeFmTZgqOBAABumsMdfKEU9jaQkT1Oh05A== 2024-09-11T06:14:56.010 INFO:teuthology.orchestra.run.smithi112.stdout:[client.mirror.3] 2024-09-11T06:14:56.010 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQBTNeFm5s/qARAAsP8x/gZnojn3eS26pNU8UQ== 2024-09-11T06:14:56.010 INFO:teuthology.orchestra.run.smithi112.stdout:[client.mirror.4] 2024-09-11T06:14:56.010 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQBTNeFmz1sdBxAAvsbd8A8Dqo/HJZ6OGK4+jw== 2024-09-11T06:14:56.010 INFO:teuthology.orchestra.run.smithi112.stdout:[client.mirror.5] 2024-09-11T06:14:56.010 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQBTNeFmOxYnDBAAhClLzmBjMo3dKDNaQYMbFg== 2024-09-11T06:14:56.010 INFO:teuthology.orchestra.run.smithi112.stdout:[client.mirror.6] 2024-09-11T06:14:56.010 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQBTNeFmOCkfERAApHFB0Na1zttKUquuuRtYzg== 2024-09-11T06:14:56.011 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-11T06:14:56.099 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T06:14:56.139 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T06:14:56.224 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T06:14:56.308 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-11T06:14:56.398 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-11T06:14:56.443 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-11T06:14:56.485 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-11T06:14:56.570 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-11T06:14:56.658 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-11T06:14:56.697 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.4 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-11T06:14:56.785 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.5 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-11T06:14:56.870 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.6 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-11T06:14:56.959 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-11T06:14:56.959 DEBUG:teuthology.orchestra.run.smithi112:> sudo mkdir -p /var/lib/ceph/mon/cluster2-a 2024-09-11T06:14:57.013 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster cluster2 --mkfs -i a --monmap /home/ubuntu/cephtest/cluster2.monmap --keyring /etc/ceph/cluster2.keyring 2024-09-11T06:14:57.110 DEBUG:teuthology.orchestra.run.smithi112:> sudo chown -R ceph:ceph /var/lib/ceph/mon/cluster2-a 2024-09-11T06:14:57.120 DEBUG:teuthology.orchestra.run.smithi112:> rm -- /home/ubuntu/cephtest/cluster2.monmap 2024-09-11T06:14:57.165 INFO:tasks.ceph:Starting mon daemons in cluster cluster2... 2024-09-11T06:14:57.165 INFO:tasks.ceph.cluster2.mon.a:Restarting daemon 2024-09-11T06:14:57.165 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster cluster2 -i a 2024-09-11T06:14:57.211 INFO:tasks.ceph.cluster2.mon.a:Started 2024-09-11T06:14:57.211 INFO:tasks.ceph:Starting mgr daemons in cluster cluster2... 2024-09-11T06:14:57.211 INFO:tasks.ceph.cluster2.mgr.x:Restarting daemon 2024-09-11T06:14:57.211 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster cluster2 -i x 2024-09-11T06:14:57.216 INFO:tasks.ceph.cluster2.mgr.x:Started 2024-09-11T06:14:57.216 INFO:tasks.ceph:Setting crush tunables to default 2024-09-11T06:14:57.216 DEBUG:teuthology.orchestra.run.smithi112:> sudo ceph --cluster cluster2 osd crush tunables default 2024-09-11T06:14:57.467 INFO:teuthology.orchestra.run.smithi112.stderr:adjusted tunables profile to default 2024-09-11T06:14:57.484 INFO:tasks.ceph:check_enable_crimson: False 2024-09-11T06:14:57.484 INFO:tasks.ceph:Starting osd daemons in cluster cluster2... 2024-09-11T06:14:57.485 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-11T06:14:57.485 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/var/lib/ceph/osd/cluster2-0/fsid of=/dev/stdout 2024-09-11T06:14:57.497 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-11T06:14:57.497 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/var/lib/ceph/osd/cluster2-1/fsid of=/dev/stdout 2024-09-11T06:14:57.514 INFO:tasks.ceph.cluster2.mgr.x.smithi006.stderr:2024-09-11T06:14:57.512+0000 7fe28e675180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-11T06:14:57.556 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-11T06:14:57.556 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/var/lib/ceph/osd/cluster2-2/fsid of=/dev/stdout 2024-09-11T06:14:57.608 DEBUG:teuthology.orchestra.run.smithi112:> sudo ceph --cluster cluster2 osd new f199f8ec-e9e7-4467-aed6-9a4445ee94b0 0 2024-09-11T06:14:57.852 INFO:teuthology.orchestra.run.smithi112.stdout:0 2024-09-11T06:14:57.870 DEBUG:teuthology.orchestra.run.smithi112:> sudo ceph --cluster cluster2 osd new 51544368-1990-4034-9462-a6b32e28e9f9 1 2024-09-11T06:14:57.919 INFO:tasks.ceph.cluster2.mgr.x.smithi006.stderr:2024-09-11T06:14:57.916+0000 7fe28e675180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-11T06:14:58.064 INFO:tasks.ceph.cluster2.mgr.x.smithi006.stderr:2024-09-11T06:14:58.060+0000 7fe28e675180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-11T06:14:58.066 INFO:teuthology.orchestra.run.smithi112.stdout:1 2024-09-11T06:14:58.084 DEBUG:teuthology.orchestra.run.smithi112:> sudo ceph --cluster cluster2 osd new e1fa99c8-30e7-4f3b-a7af-16ddf8c55f4c 2 2024-09-11T06:14:58.280 INFO:teuthology.orchestra.run.smithi112.stdout:2 2024-09-11T06:14:58.297 INFO:tasks.ceph.cluster2.osd.0:Restarting daemon 2024-09-11T06:14:58.297 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 0 2024-09-11T06:14:58.299 INFO:tasks.ceph.cluster2.osd.0:Started 2024-09-11T06:14:58.300 INFO:tasks.ceph.cluster2.osd.1:Restarting daemon 2024-09-11T06:14:58.300 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 1 2024-09-11T06:14:58.301 INFO:tasks.ceph.cluster2.osd.1:Started 2024-09-11T06:14:58.301 INFO:tasks.ceph.cluster2.osd.2:Restarting daemon 2024-09-11T06:14:58.301 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 2 2024-09-11T06:14:58.303 INFO:tasks.ceph.cluster2.osd.2:Started 2024-09-11T06:14:58.303 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-09-11T06:14:58.478 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-11T06:14:58.479 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":5,"fsid":"f5a0f59c-ac24-45be-8681-35322ac680b8","created":"2024-09-11T06:14:57.348013+0000","modified":"2024-09-11T06:14:58.282695+0000","last_up_change":"0.000000","last_in_change":"2024-09-11T06:14:58.282695+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f199f8ec-e9e7-4467-aed6-9a4445ee94b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"51544368-1990-4034-9462-a6b32e28e9f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e1fa99c8-30e7-4f3b-a7af-16ddf8c55f4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T06:14:58.494 INFO:tasks.ceph.ceph_manager.cluster2:[] 2024-09-11T06:14:58.494 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-11T06:14:58.503 INFO:tasks.ceph.cluster2.mgr.x.smithi006.stderr:2024-09-11T06:14:58.500+0000 7fe28e675180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-11T06:14:58.556 INFO:tasks.ceph.cluster2.mgr.x.smithi006.stderr:2024-09-11T06:14:58.556+0000 7fe28e675180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-11T06:14:58.607 INFO:tasks.ceph.cluster2.mgr.x.smithi006.stderr:2024-09-11T06:14:58.604+0000 7fe28e675180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-11T06:14:58.684 INFO:tasks.ceph.cluster2.mgr.x.smithi006.stderr:2024-09-11T06:14:58.684+0000 7fe28e675180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-11T06:14:58.736 INFO:tasks.ceph.cluster2.mgr.x.smithi006.stderr:2024-09-11T06:14:58.732+0000 7fe28e675180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-11T06:14:58.796 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2024-09-11T06:14:58.811 INFO:tasks.ceph.cluster2.mgr.x.smithi006.stderr:2024-09-11T06:14:58.808+0000 7fe28e675180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-11T06:14:58.864 INFO:tasks.ceph.cluster2.mgr.x.smithi006.stderr:2024-09-11T06:14:58.860+0000 7fe28e675180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-11T06:14:58.983 INFO:teuthology.misc.health.smithi112.stdout: 2024-09-11T06:14:58.983 INFO:teuthology.misc.health.smithi112.stdout:{"epoch":5,"fsid":"f5a0f59c-ac24-45be-8681-35322ac680b8","created":"2024-09-11T06:14:57.348013+0000","modified":"2024-09-11T06:14:58.282695+0000","last_up_change":"0.000000","last_in_change":"2024-09-11T06:14:58.282695+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f199f8ec-e9e7-4467-aed6-9a4445ee94b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"51544368-1990-4034-9462-a6b32e28e9f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e1fa99c8-30e7-4f3b-a7af-16ddf8c55f4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T06:14:58.999 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-09-11T06:14:59.008 INFO:tasks.ceph.cluster2.mgr.x.smithi006.stderr:2024-09-11T06:14:59.008+0000 7fe28e675180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-11T06:14:59.062 INFO:tasks.ceph.cluster2.mgr.x.smithi006.stderr:2024-09-11T06:14:59.060+0000 7fe28e675180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-11T06:14:59.289 INFO:tasks.ceph.cluster2.mgr.x.smithi006.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-09-11T06:14:59.289 INFO:tasks.ceph.cluster2.mgr.x.smithi006.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-09-11T06:14:59.289 INFO:tasks.ceph.cluster2.mgr.x.smithi006.stderr: from numpy import show_config as show_numpy_config 2024-09-11T06:14:59.296 INFO:tasks.ceph.cluster2.mgr.x.smithi006.stderr:2024-09-11T06:14:59.296+0000 7fe28e675180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-11T06:14:59.398 INFO:tasks.ceph.cluster2.osd.1.smithi112.stderr:2024-09-11T06:14:59.399+0000 7f2cfd1e77c0 -1 Falling back to public interface 2024-09-11T06:14:59.436 INFO:tasks.ceph.cluster2.mgr.x.smithi006.stderr:2024-09-11T06:14:59.432+0000 7fe28e675180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-11T06:14:59.549 INFO:tasks.ceph.cluster2.mgr.x.smithi006.stderr:2024-09-11T06:14:59.548+0000 7fe28e675180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-11T06:14:59.613 INFO:tasks.ceph.cluster2.mgr.x.smithi006.stderr:2024-09-11T06:14:59.612+0000 7fe28e675180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-11T06:14:59.642 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:14:59.643+0000 7efd820007c0 -1 Falling back to public interface 2024-09-11T06:14:59.670 INFO:tasks.ceph.cluster2.osd.0.smithi112.stderr:2024-09-11T06:14:59.671+0000 7f3fb07527c0 -1 Falling back to public interface 2024-09-11T06:14:59.752 INFO:tasks.ceph.cluster2.mgr.x.smithi006.stderr:2024-09-11T06:14:59.748+0000 7fe28e675180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-11T06:15:00.250 INFO:tasks.ceph.cluster2.mgr.x.smithi006.stderr:2024-09-11T06:15:00.248+0000 7fe28e675180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-11T06:15:00.326 INFO:tasks.ceph.cluster2.mgr.x.smithi006.stderr:2024-09-11T06:15:00.324+0000 7fe28e675180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-11T06:15:00.476 INFO:tasks.ceph.cluster2.mgr.x.smithi006.stderr:2024-09-11T06:15:00.472+0000 7fe28e675180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-11T06:15:00.551 INFO:tasks.ceph.cluster2.mgr.x.smithi006.stderr:2024-09-11T06:15:00.548+0000 7fe28e675180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-11T06:15:00.846 INFO:tasks.ceph.cluster2.mgr.x.smithi006.stderr:2024-09-11T06:15:00.844+0000 7fe28e675180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-11T06:15:00.902 INFO:tasks.ceph.cluster2.mgr.x.smithi006.stderr:2024-09-11T06:15:00.900+0000 7fe28e675180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-11T06:15:01.935 INFO:tasks.ceph.cluster2.mgr.x.smithi006.stderr:2024-09-11T06:15:01.932+0000 7fe2352a8640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-09-11T06:15:03.342 INFO:tasks.ceph.cluster2.osd.1.smithi112.stderr:2024-09-11T06:15:03.343+0000 7f2cfd1e77c0 -1 osd.1 0 log_to_monitors true 2024-09-11T06:15:03.547 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:15:03.547+0000 7efd820007c0 -1 osd.2 0 log_to_monitors true 2024-09-11T06:15:03.646 INFO:tasks.ceph.cluster2.osd.0.smithi112.stderr:2024-09-11T06:15:03.647+0000 7f3fb07527c0 -1 osd.0 0 log_to_monitors true 2024-09-11T06:15:05.301 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2024-09-11T06:15:05.604 INFO:teuthology.misc.health.smithi112.stdout: 2024-09-11T06:15:05.604 INFO:teuthology.misc.health.smithi112.stdout:{"epoch":7,"fsid":"f5a0f59c-ac24-45be-8681-35322ac680b8","created":"2024-09-11T06:14:57.348013+0000","modified":"2024-09-11T06:15:04.989395+0000","last_up_change":"0.000000","last_in_change":"2024-09-11T06:14:58.282695+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f199f8ec-e9e7-4467-aed6-9a4445ee94b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"51544368-1990-4034-9462-a6b32e28e9f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e1fa99c8-30e7-4f3b-a7af-16ddf8c55f4c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T06:15:05.621 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-09-11T06:15:07.996 INFO:tasks.ceph.cluster2.osd.0.smithi112.stderr:2024-09-11T06:15:07.995+0000 7f3facf0d640 -1 osd.0 0 waiting for initial osdmap 2024-09-11T06:15:08.014 INFO:tasks.ceph.cluster2.osd.0.smithi112.stderr:2024-09-11T06:15:08.015+0000 7f3fa4530640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T06:15:08.067 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:15:08.067+0000 7efd7dfa9640 -1 osd.2 0 waiting for initial osdmap 2024-09-11T06:15:08.085 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:15:08.087+0000 7efd75dde640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T06:15:08.096 INFO:tasks.ceph.cluster2.osd.1.smithi112.stderr:2024-09-11T06:15:08.095+0000 7f2cf9190640 -1 osd.1 0 waiting for initial osdmap 2024-09-11T06:15:08.113 INFO:tasks.ceph.cluster2.osd.1.smithi112.stderr:2024-09-11T06:15:08.115+0000 7f2cf0fc5640 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T06:15:11.924 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2024-09-11T06:15:12.226 INFO:teuthology.misc.health.smithi112.stdout: 2024-09-11T06:15:12.226 INFO:teuthology.misc.health.smithi112.stdout:{"epoch":11,"fsid":"f5a0f59c-ac24-45be-8681-35322ac680b8","created":"2024-09-11T06:14:57.348013+0000","modified":"2024-09-11T06:15:12.026407+0000","last_up_change":"2024-09-11T06:15:09.003637+0000","last_in_change":"2024-09-11T06:14:58.282695+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-11T06:15:11.008778+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f199f8ec-e9e7-4467-aed6-9a4445ee94b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":2881424358},{"type":"v1","addr":"172.21.15.112:6817","nonce":2881424358}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6818","nonce":2881424358},{"type":"v1","addr":"172.21.15.112:6819","nonce":2881424358}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6822","nonce":2881424358},{"type":"v1","addr":"172.21.15.112:6823","nonce":2881424358}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6820","nonce":2881424358},{"type":"v1","addr":"172.21.15.112:6821","nonce":2881424358}]},"public_addr":"172.21.15.112:6817/2881424358","cluster_addr":"172.21.15.112:6819/2881424358","heartbeat_back_addr":"172.21.15.112:6823/2881424358","heartbeat_front_addr":"172.21.15.112:6821/2881424358","state":["exists","up"]},{"osd":1,"uuid":"51544368-1990-4034-9462-a6b32e28e9f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":1273240578},{"type":"v1","addr":"172.21.15.112:6801","nonce":1273240578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":1273240578},{"type":"v1","addr":"172.21.15.112:6803","nonce":1273240578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":1273240578},{"type":"v1","addr":"172.21.15.112:6807","nonce":1273240578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":1273240578},{"type":"v1","addr":"172.21.15.112:6805","nonce":1273240578}]},"public_addr":"172.21.15.112:6801/1273240578","cluster_addr":"172.21.15.112:6803/1273240578","heartbeat_back_addr":"172.21.15.112:6807/1273240578","heartbeat_front_addr":"172.21.15.112:6805/1273240578","state":["exists","up"]},{"osd":2,"uuid":"e1fa99c8-30e7-4f3b-a7af-16ddf8c55f4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":3588377934},{"type":"v1","addr":"172.21.15.112:6809","nonce":3588377934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":3588377934},{"type":"v1","addr":"172.21.15.112:6811","nonce":3588377934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":3588377934},{"type":"v1","addr":"172.21.15.112:6815","nonce":3588377934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6812","nonce":3588377934},{"type":"v1","addr":"172.21.15.112:6813","nonce":3588377934}]},"public_addr":"172.21.15.112:6809/3588377934","cluster_addr":"172.21.15.112:6811/3588377934","heartbeat_back_addr":"172.21.15.112:6815/3588377934","heartbeat_front_addr":"172.21.15.112:6813/3588377934","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T06:15:04.608603+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T06:15:04.329053+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T06:15:04.503513+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T06:15:12.243 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-09-11T06:15:12.243 INFO:tasks.ceph:Creating RBD pool 2024-09-11T06:15:12.243 DEBUG:teuthology.orchestra.run.smithi112:> sudo ceph --cluster cluster2 osd pool create rbd 8 2024-09-11T06:15:13.126 INFO:teuthology.orchestra.run.smithi112.stderr:pool 'rbd' created 2024-09-11T06:15:13.153 DEBUG:teuthology.orchestra.run.smithi112:> rbd --cluster cluster2 pool init rbd 2024-09-11T06:15:16.171 INFO:tasks.ceph:Starting mds daemons in cluster cluster2... 2024-09-11T06:15:16.172 INFO:tasks.ceph:Waiting until cluster2 daemons up and pgs clean... 2024-09-11T06:15:16.172 INFO:tasks.ceph.ceph_manager.cluster2:waiting for mgr available 2024-09-11T06:15:16.172 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 mgr dump --format=json 2024-09-11T06:15:16.173 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-11T06:15:16.521 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-11T06:15:16.539 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":5,"flags":0,"active_gid":4100,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":1625563887},{"type":"v1","addr":"172.21.15.6:6827","nonce":1625563887}]},"active_addr":"172.21.15.6:6827/1625563887","active_change":"2024-09-11T06:15:00.925717+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.6:0","nonce":3105076200}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.6:0","nonce":1583683320}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.6:0","nonce":4206870806}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.6:0","nonce":931612755}]}]} 2024-09-11T06:15:16.541 INFO:tasks.ceph.ceph_manager.cluster2:mgr available! 2024-09-11T06:15:16.541 INFO:tasks.ceph.ceph_manager.cluster2:waiting for all up 2024-09-11T06:15:16.542 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-09-11T06:15:16.829 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-11T06:15:16.829 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":15,"fsid":"f5a0f59c-ac24-45be-8681-35322ac680b8","created":"2024-09-11T06:14:57.348013+0000","modified":"2024-09-11T06:15:16.149648+0000","last_up_change":"2024-09-11T06:15:09.003637+0000","last_in_change":"2024-09-11T06:14:58.282695+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-11T06:15:11.008778+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-11T06:15:12.586728+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":15,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f199f8ec-e9e7-4467-aed6-9a4445ee94b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":2881424358},{"type":"v1","addr":"172.21.15.112:6817","nonce":2881424358}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6818","nonce":2881424358},{"type":"v1","addr":"172.21.15.112:6819","nonce":2881424358}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6822","nonce":2881424358},{"type":"v1","addr":"172.21.15.112:6823","nonce":2881424358}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6820","nonce":2881424358},{"type":"v1","addr":"172.21.15.112:6821","nonce":2881424358}]},"public_addr":"172.21.15.112:6817/2881424358","cluster_addr":"172.21.15.112:6819/2881424358","heartbeat_back_addr":"172.21.15.112:6823/2881424358","heartbeat_front_addr":"172.21.15.112:6821/2881424358","state":["exists","up"]},{"osd":1,"uuid":"51544368-1990-4034-9462-a6b32e28e9f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":1273240578},{"type":"v1","addr":"172.21.15.112:6801","nonce":1273240578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":1273240578},{"type":"v1","addr":"172.21.15.112:6803","nonce":1273240578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":1273240578},{"type":"v1","addr":"172.21.15.112:6807","nonce":1273240578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":1273240578},{"type":"v1","addr":"172.21.15.112:6805","nonce":1273240578}]},"public_addr":"172.21.15.112:6801/1273240578","cluster_addr":"172.21.15.112:6803/1273240578","heartbeat_back_addr":"172.21.15.112:6807/1273240578","heartbeat_front_addr":"172.21.15.112:6805/1273240578","state":["exists","up"]},{"osd":2,"uuid":"e1fa99c8-30e7-4f3b-a7af-16ddf8c55f4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":3588377934},{"type":"v1","addr":"172.21.15.112:6809","nonce":3588377934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":3588377934},{"type":"v1","addr":"172.21.15.112:6811","nonce":3588377934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":3588377934},{"type":"v1","addr":"172.21.15.112:6815","nonce":3588377934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6812","nonce":3588377934},{"type":"v1","addr":"172.21.15.112:6813","nonce":3588377934}]},"public_addr":"172.21.15.112:6809/3588377934","cluster_addr":"172.21.15.112:6811/3588377934","heartbeat_back_addr":"172.21.15.112:6815/3588377934","heartbeat_front_addr":"172.21.15.112:6813/3588377934","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T06:15:04.608603+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T06:15:04.329053+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T06:15:04.503513+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T06:15:16.848 INFO:tasks.ceph.ceph_manager.cluster2:all up! 2024-09-11T06:15:16.848 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-09-11T06:15:17.114 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-11T06:15:17.114 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":15,"fsid":"f5a0f59c-ac24-45be-8681-35322ac680b8","created":"2024-09-11T06:14:57.348013+0000","modified":"2024-09-11T06:15:16.149648+0000","last_up_change":"2024-09-11T06:15:09.003637+0000","last_in_change":"2024-09-11T06:14:58.282695+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-11T06:15:11.008778+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-11T06:15:12.586728+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":15,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f199f8ec-e9e7-4467-aed6-9a4445ee94b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":2881424358},{"type":"v1","addr":"172.21.15.112:6817","nonce":2881424358}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6818","nonce":2881424358},{"type":"v1","addr":"172.21.15.112:6819","nonce":2881424358}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6822","nonce":2881424358},{"type":"v1","addr":"172.21.15.112:6823","nonce":2881424358}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6820","nonce":2881424358},{"type":"v1","addr":"172.21.15.112:6821","nonce":2881424358}]},"public_addr":"172.21.15.112:6817/2881424358","cluster_addr":"172.21.15.112:6819/2881424358","heartbeat_back_addr":"172.21.15.112:6823/2881424358","heartbeat_front_addr":"172.21.15.112:6821/2881424358","state":["exists","up"]},{"osd":1,"uuid":"51544368-1990-4034-9462-a6b32e28e9f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":1273240578},{"type":"v1","addr":"172.21.15.112:6801","nonce":1273240578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":1273240578},{"type":"v1","addr":"172.21.15.112:6803","nonce":1273240578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":1273240578},{"type":"v1","addr":"172.21.15.112:6807","nonce":1273240578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":1273240578},{"type":"v1","addr":"172.21.15.112:6805","nonce":1273240578}]},"public_addr":"172.21.15.112:6801/1273240578","cluster_addr":"172.21.15.112:6803/1273240578","heartbeat_back_addr":"172.21.15.112:6807/1273240578","heartbeat_front_addr":"172.21.15.112:6805/1273240578","state":["exists","up"]},{"osd":2,"uuid":"e1fa99c8-30e7-4f3b-a7af-16ddf8c55f4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":3588377934},{"type":"v1","addr":"172.21.15.112:6809","nonce":3588377934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":3588377934},{"type":"v1","addr":"172.21.15.112:6811","nonce":3588377934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":3588377934},{"type":"v1","addr":"172.21.15.112:6815","nonce":3588377934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6812","nonce":3588377934},{"type":"v1","addr":"172.21.15.112:6813","nonce":3588377934}]},"public_addr":"172.21.15.112:6809/3588377934","cluster_addr":"172.21.15.112:6811/3588377934","heartbeat_back_addr":"172.21.15.112:6815/3588377934","heartbeat_front_addr":"172.21.15.112:6813/3588377934","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T06:15:04.608603+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T06:15:04.329053+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T06:15:04.503513+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T06:15:17.131 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.0 flush_pg_stats 2024-09-11T06:15:17.132 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.1 flush_pg_stats 2024-09-11T06:15:17.132 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.2 flush_pg_stats 2024-09-11T06:15:17.290 INFO:teuthology.orchestra.run.smithi112.stdout:34359738376 2024-09-11T06:15:17.291 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2024-09-11T06:15:17.303 INFO:teuthology.orchestra.run.smithi112.stdout:34359738376 2024-09-11T06:15:17.303 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2024-09-11T06:15:17.310 INFO:teuthology.orchestra.run.smithi112.stdout:34359738376 2024-09-11T06:15:17.311 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2024-09-11T06:15:17.592 INFO:teuthology.orchestra.run.smithi112.stdout:34359738375 2024-09-11T06:15:17.609 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738376 got 34359738375 for osd.0 2024-09-11T06:15:17.633 INFO:teuthology.orchestra.run.smithi112.stdout:34359738375 2024-09-11T06:15:17.649 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738376 got 34359738375 for osd.1 2024-09-11T06:15:17.651 INFO:teuthology.orchestra.run.smithi112.stdout:34359738375 2024-09-11T06:15:17.668 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738376 got 34359738375 for osd.2 2024-09-11T06:15:18.610 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2024-09-11T06:15:18.650 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2024-09-11T06:15:18.669 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2024-09-11T06:15:18.919 INFO:teuthology.orchestra.run.smithi112.stdout:34359738375 2024-09-11T06:15:18.937 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738376 got 34359738375 for osd.0 2024-09-11T06:15:18.949 INFO:teuthology.orchestra.run.smithi112.stdout:34359738375 2024-09-11T06:15:18.965 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738376 got 34359738375 for osd.1 2024-09-11T06:15:18.988 INFO:teuthology.orchestra.run.smithi112.stdout:34359738375 2024-09-11T06:15:19.005 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738376 got 34359738375 for osd.2 2024-09-11T06:15:19.938 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2024-09-11T06:15:19.966 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2024-09-11T06:15:20.006 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2024-09-11T06:15:20.276 INFO:teuthology.orchestra.run.smithi112.stdout:34359738376 2024-09-11T06:15:20.293 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738376 got 34359738376 for osd.0 2024-09-11T06:15:20.293 DEBUG:teuthology.parallel:result is None 2024-09-11T06:15:20.302 INFO:teuthology.orchestra.run.smithi112.stdout:34359738376 2024-09-11T06:15:20.317 INFO:teuthology.orchestra.run.smithi112.stdout:34359738376 2024-09-11T06:15:20.319 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738376 got 34359738376 for osd.1 2024-09-11T06:15:20.319 DEBUG:teuthology.parallel:result is None 2024-09-11T06:15:20.334 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738376 got 34359738376 for osd.2 2024-09-11T06:15:20.334 DEBUG:teuthology.parallel:result is None 2024-09-11T06:15:20.334 INFO:tasks.ceph.ceph_manager.cluster2:waiting for clean 2024-09-11T06:15:20.334 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-11T06:15:20.652 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-11T06:15:20.652 INFO:teuthology.orchestra.run.smithi112.stderr:dumped all 2024-09-11T06:15:20.669 INFO:teuthology.orchestra.run.smithi112.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2024-09-11T06:15:18.933941+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":13,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":81616,"kb_used_data":504,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283033904,"statfs":{"total":289910292480,"available":289826717696,"internally_reserved":0,"allocated":516096,"data_stored":1294801,"data_compressed":7472,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4785,"internal_metadata":82963791},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.796476"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-11T06:15:16.158300+0000","last_change":"2024-09-11T06:15:16.158701+0000","last_active":"2024-09-11T06:15:16.158300+0000","last_peered":"2024-09-11T06:15:16.158300+0000","last_clean":"2024-09-11T06:15:16.158300+0000","last_became_active":"2024-09-11T06:15:14.143307+0000","last_became_peered":"2024-09-11T06:15:14.143307+0000","last_unstale":"2024-09-11T06:15:16.158300+0000","last_undegraded":"2024-09-11T06:15:16.158300+0000","last_fullsized":"2024-09-11T06:15:16.158300+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:15:13.030949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:15:13.030949+0000","last_clean_scrub_stamp":"2024-09-11T06:15:13.030949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:41:30.583877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019449249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-11T06:15:16.157142+0000","last_change":"2024-09-11T06:15:16.157659+0000","last_active":"2024-09-11T06:15:16.157142+0000","last_peered":"2024-09-11T06:15:16.157142+0000","last_clean":"2024-09-11T06:15:16.157142+0000","last_became_active":"2024-09-11T06:15:14.142832+0000","last_became_peered":"2024-09-11T06:15:14.142832+0000","last_unstale":"2024-09-11T06:15:16.157142+0000","last_undegraded":"2024-09-11T06:15:16.157142+0000","last_fullsized":"2024-09-11T06:15:16.157142+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:15:13.030949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:15:13.030949+0000","last_clean_scrub_stamp":"2024-09-11T06:15:13.030949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T17:10:46.757129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001083002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-11T06:15:16.157644+0000","last_change":"2024-09-11T06:15:16.158167+0000","last_active":"2024-09-11T06:15:16.157644+0000","last_peered":"2024-09-11T06:15:16.157644+0000","last_clean":"2024-09-11T06:15:16.157644+0000","last_became_active":"2024-09-11T06:15:14.143100+0000","last_became_peered":"2024-09-11T06:15:14.143100+0000","last_unstale":"2024-09-11T06:15:16.157644+0000","last_undegraded":"2024-09-11T06:15:16.157644+0000","last_fullsized":"2024-09-11T06:15:16.157644+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:15:13.030949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:15:13.030949+0000","last_clean_scrub_stamp":"2024-09-11T06:15:13.030949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:44.555439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001527928,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-11T06:15:16.157785+0000","last_change":"2024-09-11T06:15:16.158239+0000","last_active":"2024-09-11T06:15:16.157785+0000","last_peered":"2024-09-11T06:15:16.157785+0000","last_clean":"2024-09-11T06:15:16.157785+0000","last_became_active":"2024-09-11T06:15:14.142926+0000","last_became_peered":"2024-09-11T06:15:14.142926+0000","last_unstale":"2024-09-11T06:15:16.157785+0000","last_undegraded":"2024-09-11T06:15:16.157785+0000","last_fullsized":"2024-09-11T06:15:16.157785+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:15:13.030949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:15:13.030949+0000","last_clean_scrub_stamp":"2024-09-11T06:15:13.030949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:52:31.503430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015085280000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"15'2","reported_seq":21,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-11T06:15:16.164947+0000","last_change":"2024-09-11T06:15:16.164947+0000","last_active":"2024-09-11T06:15:16.164947+0000","last_peered":"2024-09-11T06:15:16.164947+0000","last_clean":"2024-09-11T06:15:16.164947+0000","last_became_active":"2024-09-11T06:15:14.142347+0000","last_became_peered":"2024-09-11T06:15:14.142347+0000","last_unstale":"2024-09-11T06:15:16.164947+0000","last_undegraded":"2024-09-11T06:15:16.164947+0000","last_fullsized":"2024-09-11T06:15:16.164947+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:15:13.030949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:15:13.030949+0000","last_clean_scrub_stamp":"2024-09-11T06:15:13.030949+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:26:00.950222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022347420000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":17,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-11T06:15:16.456858+0000","last_change":"2024-09-11T06:15:16.457217+0000","last_active":"2024-09-11T06:15:16.456858+0000","last_peered":"2024-09-11T06:15:16.456858+0000","last_clean":"2024-09-11T06:15:16.456858+0000","last_became_active":"2024-09-11T06:15:14.143512+0000","last_became_peered":"2024-09-11T06:15:14.143512+0000","last_unstale":"2024-09-11T06:15:16.456858+0000","last_undegraded":"2024-09-11T06:15:16.456858+0000","last_fullsized":"2024-09-11T06:15:16.456858+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:15:13.030949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:15:13.030949+0000","last_clean_scrub_stamp":"2024-09-11T06:15:13.030949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:31:31.971513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063951500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":17,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-11T06:15:16.457002+0000","last_change":"2024-09-11T06:15:16.457653+0000","last_active":"2024-09-11T06:15:16.457002+0000","last_peered":"2024-09-11T06:15:16.457002+0000","last_clean":"2024-09-11T06:15:16.457002+0000","last_became_active":"2024-09-11T06:15:14.142323+0000","last_became_peered":"2024-09-11T06:15:14.142323+0000","last_unstale":"2024-09-11T06:15:16.457002+0000","last_undegraded":"2024-09-11T06:15:16.457002+0000","last_fullsized":"2024-09-11T06:15:16.457002+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:15:13.030949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:15:13.030949+0000","last_clean_scrub_stamp":"2024-09-11T06:15:13.030949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:28:11.865252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091634599999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"13'1","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-11T06:15:16.157004+0000","last_change":"2024-09-11T06:15:16.157504+0000","last_active":"2024-09-11T06:15:16.157004+0000","last_peered":"2024-09-11T06:15:16.157004+0000","last_clean":"2024-09-11T06:15:16.157004+0000","last_became_active":"2024-09-11T06:15:14.142073+0000","last_became_peered":"2024-09-11T06:15:14.142073+0000","last_unstale":"2024-09-11T06:15:16.157004+0000","last_undegraded":"2024-09-11T06:15:16.157004+0000","last_fullsized":"2024-09-11T06:15:16.157004+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:15:13.030949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:15:13.030949+0000","last_clean_scrub_stamp":"2024-09-11T06:15:13.030949+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:16:44.214033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010516950000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"12'97","reported_seq":145,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-11T06:15:16.156109+0000","last_change":"2024-09-11T06:15:12.513296+0000","last_active":"2024-09-11T06:15:16.156109+0000","last_peered":"2024-09-11T06:15:16.156109+0000","last_clean":"2024-09-11T06:15:16.156109+0000","last_became_active":"2024-09-11T06:15:12.511458+0000","last_became_peered":"2024-09-11T06:15:12.511458+0000","last_unstale":"2024-09-11T06:15:16.156109+0000","last_undegraded":"2024-09-11T06:15:16.156109+0000","last_fullsized":"2024-09-11T06:15:16.156109+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:15:11.022532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:15:11.022532+0000","last_clean_scrub_stamp":"2024-09-11T06:15:11.022532+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:14:38.108521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7472,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738376,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27156,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344684,"statfs":{"total":96636764160,"available":96608956416,"internally_reserved":0,"allocated":131072,"data_stored":38009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738376,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":192512,"data_stored":628396,"data_compressed":3736,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738376,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":192512,"data_stored":628396,"data_compressed":3736,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3736,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3736,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T06:15:20.670 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-11T06:15:20.950 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-11T06:15:20.951 INFO:teuthology.orchestra.run.smithi112.stderr:dumped all 2024-09-11T06:15:20.968 INFO:teuthology.orchestra.run.smithi112.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2024-09-11T06:15:20.934626+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":81604,"kb_used_data":504,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283033916,"statfs":{"total":289910292480,"available":289826729984,"internally_reserved":0,"allocated":516096,"data_stored":1294801,"data_compressed":7472,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4785,"internal_metadata":82963791},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.797161"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-11T06:15:16.158300+0000","last_change":"2024-09-11T06:15:16.158701+0000","last_active":"2024-09-11T06:15:16.158300+0000","last_peered":"2024-09-11T06:15:16.158300+0000","last_clean":"2024-09-11T06:15:16.158300+0000","last_became_active":"2024-09-11T06:15:14.143307+0000","last_became_peered":"2024-09-11T06:15:14.143307+0000","last_unstale":"2024-09-11T06:15:16.158300+0000","last_undegraded":"2024-09-11T06:15:16.158300+0000","last_fullsized":"2024-09-11T06:15:16.158300+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:15:13.030949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:15:13.030949+0000","last_clean_scrub_stamp":"2024-09-11T06:15:13.030949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:41:30.583877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019449249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-11T06:15:16.157142+0000","last_change":"2024-09-11T06:15:16.157659+0000","last_active":"2024-09-11T06:15:16.157142+0000","last_peered":"2024-09-11T06:15:16.157142+0000","last_clean":"2024-09-11T06:15:16.157142+0000","last_became_active":"2024-09-11T06:15:14.142832+0000","last_became_peered":"2024-09-11T06:15:14.142832+0000","last_unstale":"2024-09-11T06:15:16.157142+0000","last_undegraded":"2024-09-11T06:15:16.157142+0000","last_fullsized":"2024-09-11T06:15:16.157142+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:15:13.030949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:15:13.030949+0000","last_clean_scrub_stamp":"2024-09-11T06:15:13.030949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T17:10:46.757129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001083002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-11T06:15:16.157644+0000","last_change":"2024-09-11T06:15:16.158167+0000","last_active":"2024-09-11T06:15:16.157644+0000","last_peered":"2024-09-11T06:15:16.157644+0000","last_clean":"2024-09-11T06:15:16.157644+0000","last_became_active":"2024-09-11T06:15:14.143100+0000","last_became_peered":"2024-09-11T06:15:14.143100+0000","last_unstale":"2024-09-11T06:15:16.157644+0000","last_undegraded":"2024-09-11T06:15:16.157644+0000","last_fullsized":"2024-09-11T06:15:16.157644+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:15:13.030949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:15:13.030949+0000","last_clean_scrub_stamp":"2024-09-11T06:15:13.030949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:44.555439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001527928,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-11T06:15:16.157785+0000","last_change":"2024-09-11T06:15:16.158239+0000","last_active":"2024-09-11T06:15:16.157785+0000","last_peered":"2024-09-11T06:15:16.157785+0000","last_clean":"2024-09-11T06:15:16.157785+0000","last_became_active":"2024-09-11T06:15:14.142926+0000","last_became_peered":"2024-09-11T06:15:14.142926+0000","last_unstale":"2024-09-11T06:15:16.157785+0000","last_undegraded":"2024-09-11T06:15:16.157785+0000","last_fullsized":"2024-09-11T06:15:16.157785+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:15:13.030949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:15:13.030949+0000","last_clean_scrub_stamp":"2024-09-11T06:15:13.030949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:52:31.503430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015085280000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"15'2","reported_seq":21,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-11T06:15:16.164947+0000","last_change":"2024-09-11T06:15:16.164947+0000","last_active":"2024-09-11T06:15:16.164947+0000","last_peered":"2024-09-11T06:15:16.164947+0000","last_clean":"2024-09-11T06:15:16.164947+0000","last_became_active":"2024-09-11T06:15:14.142347+0000","last_became_peered":"2024-09-11T06:15:14.142347+0000","last_unstale":"2024-09-11T06:15:16.164947+0000","last_undegraded":"2024-09-11T06:15:16.164947+0000","last_fullsized":"2024-09-11T06:15:16.164947+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:15:13.030949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:15:13.030949+0000","last_clean_scrub_stamp":"2024-09-11T06:15:13.030949+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:26:00.950222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022347420000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":17,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-11T06:15:16.456858+0000","last_change":"2024-09-11T06:15:16.457217+0000","last_active":"2024-09-11T06:15:16.456858+0000","last_peered":"2024-09-11T06:15:16.456858+0000","last_clean":"2024-09-11T06:15:16.456858+0000","last_became_active":"2024-09-11T06:15:14.143512+0000","last_became_peered":"2024-09-11T06:15:14.143512+0000","last_unstale":"2024-09-11T06:15:16.456858+0000","last_undegraded":"2024-09-11T06:15:16.456858+0000","last_fullsized":"2024-09-11T06:15:16.456858+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:15:13.030949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:15:13.030949+0000","last_clean_scrub_stamp":"2024-09-11T06:15:13.030949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:31:31.971513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063951500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":17,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-11T06:15:16.457002+0000","last_change":"2024-09-11T06:15:16.457653+0000","last_active":"2024-09-11T06:15:16.457002+0000","last_peered":"2024-09-11T06:15:16.457002+0000","last_clean":"2024-09-11T06:15:16.457002+0000","last_became_active":"2024-09-11T06:15:14.142323+0000","last_became_peered":"2024-09-11T06:15:14.142323+0000","last_unstale":"2024-09-11T06:15:16.457002+0000","last_undegraded":"2024-09-11T06:15:16.457002+0000","last_fullsized":"2024-09-11T06:15:16.457002+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:15:13.030949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:15:13.030949+0000","last_clean_scrub_stamp":"2024-09-11T06:15:13.030949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:28:11.865252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091634599999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"13'1","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-11T06:15:16.157004+0000","last_change":"2024-09-11T06:15:16.157504+0000","last_active":"2024-09-11T06:15:16.157004+0000","last_peered":"2024-09-11T06:15:16.157004+0000","last_clean":"2024-09-11T06:15:16.157004+0000","last_became_active":"2024-09-11T06:15:14.142073+0000","last_became_peered":"2024-09-11T06:15:14.142073+0000","last_unstale":"2024-09-11T06:15:16.157004+0000","last_undegraded":"2024-09-11T06:15:16.157004+0000","last_fullsized":"2024-09-11T06:15:16.157004+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:15:13.030949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:15:13.030949+0000","last_clean_scrub_stamp":"2024-09-11T06:15:13.030949+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:16:44.214033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010516950000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"12'97","reported_seq":145,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-11T06:15:16.156109+0000","last_change":"2024-09-11T06:15:12.513296+0000","last_active":"2024-09-11T06:15:16.156109+0000","last_peered":"2024-09-11T06:15:16.156109+0000","last_clean":"2024-09-11T06:15:16.156109+0000","last_became_active":"2024-09-11T06:15:12.511458+0000","last_became_peered":"2024-09-11T06:15:12.511458+0000","last_unstale":"2024-09-11T06:15:16.156109+0000","last_undegraded":"2024-09-11T06:15:16.156109+0000","last_fullsized":"2024-09-11T06:15:16.156109+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:15:11.022532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:15:11.022532+0000","last_clean_scrub_stamp":"2024-09-11T06:15:11.022532+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:14:38.108521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7472,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738377,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27156,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344684,"statfs":{"total":96636764160,"available":96608956416,"internally_reserved":0,"allocated":131072,"data_stored":38009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738377,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":192512,"data_stored":628396,"data_compressed":3736,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738377,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":192512,"data_stored":628396,"data_compressed":3736,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3736,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3736,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T06:15:20.969 INFO:tasks.ceph.ceph_manager.cluster2:clean! 2024-09-11T06:15:20.969 INFO:tasks.ceph:Waiting until ceph cluster cluster2 is healthy... 2024-09-11T06:15:20.969 INFO:tasks.ceph.ceph_manager.cluster2:wait_until_healthy 2024-09-11T06:15:20.969 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 health --format=json 2024-09-11T06:15:21.274 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-11T06:15:21.275 INFO:teuthology.orchestra.run.smithi112.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-11T06:15:21.293 INFO:tasks.ceph.ceph_manager.cluster2:wait_until_healthy done 2024-09-11T06:15:21.294 INFO:teuthology.run_tasks:Running task exec... 2024-09-11T06:15:21.304 INFO:teuthology.task.exec:Executing custom commands... 2024-09-11T06:15:21.304 INFO:teuthology.task.exec:Running commands on role cluster1.client.mirror.0 host ubuntu@smithi112.front.sepia.ceph.com 2024-09-11T06:15:21.304 DEBUG:teuthology.orchestra.run.smithi112:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.4 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-11T06:15:21.652 INFO:teuthology.orchestra.run.smithi112.stdout:[client.mirror.4] 2024-09-11T06:15:21.652 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQArNeFmlomwLRAAbIuKhdEPl7Ijik3Zc9pQ4g== 2024-09-11T06:15:21.652 INFO:teuthology.orchestra.run.smithi112.stdout: caps mon = "profile rbd-mirror" 2024-09-11T06:15:21.652 INFO:teuthology.orchestra.run.smithi112.stdout: caps osd = "profile rbd" 2024-09-11T06:15:21.653 INFO:teuthology.orchestra.run.smithi112.stderr:updated caps for client.mirror.4 2024-09-11T06:15:21.672 DEBUG:teuthology.orchestra.run.smithi112:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.5 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-11T06:15:22.030 INFO:teuthology.orchestra.run.smithi112.stdout:[client.mirror.5] 2024-09-11T06:15:22.030 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQArNeFmoanLMhAAQ4UCfddDDmGBiaoUWhgAlQ== 2024-09-11T06:15:22.030 INFO:teuthology.orchestra.run.smithi112.stdout: caps mon = "profile rbd-mirror" 2024-09-11T06:15:22.030 INFO:teuthology.orchestra.run.smithi112.stdout: caps osd = "profile rbd" 2024-09-11T06:15:22.031 INFO:teuthology.orchestra.run.smithi112.stderr:updated caps for client.mirror.5 2024-09-11T06:15:22.048 DEBUG:teuthology.orchestra.run.smithi112:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.6 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-11T06:15:22.403 INFO:teuthology.orchestra.run.smithi112.stdout:[client.mirror.6] 2024-09-11T06:15:22.403 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQArNeFmHRe+NxAALFoPACVZgDoZcmPtDHodMA== 2024-09-11T06:15:22.404 INFO:teuthology.orchestra.run.smithi112.stdout: caps mon = "profile rbd-mirror" 2024-09-11T06:15:22.404 INFO:teuthology.orchestra.run.smithi112.stdout: caps osd = "profile rbd" 2024-09-11T06:15:22.404 INFO:teuthology.orchestra.run.smithi112.stderr:updated caps for client.mirror.6 2024-09-11T06:15:22.421 DEBUG:teuthology.orchestra.run.smithi112:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.4 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-11T06:15:22.769 INFO:teuthology.orchestra.run.smithi112.stdout:[client.mirror.4] 2024-09-11T06:15:22.769 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQBTNeFmz1sdBxAAvsbd8A8Dqo/HJZ6OGK4+jw== 2024-09-11T06:15:22.769 INFO:teuthology.orchestra.run.smithi112.stdout: caps mon = "profile rbd-mirror" 2024-09-11T06:15:22.769 INFO:teuthology.orchestra.run.smithi112.stdout: caps osd = "profile rbd" 2024-09-11T06:15:22.769 INFO:teuthology.orchestra.run.smithi112.stderr:updated caps for client.mirror.4 2024-09-11T06:15:22.785 DEBUG:teuthology.orchestra.run.smithi112:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.5 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-11T06:15:23.122 INFO:teuthology.orchestra.run.smithi112.stdout:[client.mirror.5] 2024-09-11T06:15:23.122 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQBTNeFmOxYnDBAAhClLzmBjMo3dKDNaQYMbFg== 2024-09-11T06:15:23.122 INFO:teuthology.orchestra.run.smithi112.stdout: caps mon = "profile rbd-mirror" 2024-09-11T06:15:23.122 INFO:teuthology.orchestra.run.smithi112.stdout: caps osd = "profile rbd" 2024-09-11T06:15:23.123 INFO:teuthology.orchestra.run.smithi112.stderr:updated caps for client.mirror.5 2024-09-11T06:15:23.140 DEBUG:teuthology.orchestra.run.smithi112:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.6 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-11T06:15:23.499 INFO:teuthology.orchestra.run.smithi112.stdout:[client.mirror.6] 2024-09-11T06:15:23.499 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQBTNeFmOCkfERAApHFB0Na1zttKUquuuRtYzg== 2024-09-11T06:15:23.499 INFO:teuthology.orchestra.run.smithi112.stdout: caps mon = "profile rbd-mirror" 2024-09-11T06:15:23.499 INFO:teuthology.orchestra.run.smithi112.stdout: caps osd = "profile rbd" 2024-09-11T06:15:23.499 INFO:teuthology.orchestra.run.smithi112.stderr:updated caps for client.mirror.6 2024-09-11T06:15:23.517 INFO:teuthology.run_tasks:Running task exec... 2024-09-11T06:15:23.525 INFO:teuthology.task.exec:Executing custom commands... 2024-09-11T06:15:23.525 INFO:teuthology.task.exec:Running commands on role cluster1.client.mirror.0 host ubuntu@smithi112.front.sepia.ceph.com 2024-09-11T06:15:23.525 DEBUG:teuthology.orchestra.run.smithi112:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror mon '"'"'profile rbd-mirror-peer'"'"' osd '"'"'profile rbd'"'"'' 2024-09-11T06:15:23.908 INFO:teuthology.orchestra.run.smithi112.stdout:[client.mirror] 2024-09-11T06:15:23.908 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQArNeFmmQjtExAA67y5TzmrVP0wFCygKy7oMQ== 2024-09-11T06:15:23.908 INFO:teuthology.orchestra.run.smithi112.stdout: caps mon = "profile rbd-mirror-peer" 2024-09-11T06:15:23.908 INFO:teuthology.orchestra.run.smithi112.stdout: caps osd = "profile rbd" 2024-09-11T06:15:23.908 INFO:teuthology.orchestra.run.smithi112.stderr:updated caps for client.mirror 2024-09-11T06:15:23.928 DEBUG:teuthology.orchestra.run.smithi112:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.0 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-11T06:15:24.306 INFO:teuthology.orchestra.run.smithi112.stdout:[client.mirror.0] 2024-09-11T06:15:24.306 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQArNeFmmey1GRAAVP7NPZoRCOGiWipbcYzlQA== 2024-09-11T06:15:24.306 INFO:teuthology.orchestra.run.smithi112.stdout: caps mon = "profile rbd-mirror" 2024-09-11T06:15:24.307 INFO:teuthology.orchestra.run.smithi112.stdout: caps osd = "profile rbd" 2024-09-11T06:15:24.307 INFO:teuthology.orchestra.run.smithi112.stderr:updated caps for client.mirror.0 2024-09-11T06:15:24.325 DEBUG:teuthology.orchestra.run.smithi112:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.1 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-11T06:15:24.688 INFO:teuthology.orchestra.run.smithi112.stdout:[client.mirror.1] 2024-09-11T06:15:24.689 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQArNeFmhf68HhAAmFLjhruV41s1zYhJ0T2nRQ== 2024-09-11T06:15:24.689 INFO:teuthology.orchestra.run.smithi112.stdout: caps mon = "profile rbd-mirror" 2024-09-11T06:15:24.689 INFO:teuthology.orchestra.run.smithi112.stdout: caps osd = "profile rbd" 2024-09-11T06:15:24.689 INFO:teuthology.orchestra.run.smithi112.stderr:updated caps for client.mirror.1 2024-09-11T06:15:24.707 DEBUG:teuthology.orchestra.run.smithi112:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.2 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-11T06:15:25.073 INFO:teuthology.orchestra.run.smithi112.stdout:[client.mirror.2] 2024-09-11T06:15:25.073 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQArNeFmxw/NIxAAuJdPtgE0fCgzmSv4fNAQZg== 2024-09-11T06:15:25.073 INFO:teuthology.orchestra.run.smithi112.stdout: caps mon = "profile rbd-mirror" 2024-09-11T06:15:25.074 INFO:teuthology.orchestra.run.smithi112.stdout: caps osd = "profile rbd" 2024-09-11T06:15:25.074 INFO:teuthology.orchestra.run.smithi112.stderr:updated caps for client.mirror.2 2024-09-11T06:15:25.091 DEBUG:teuthology.orchestra.run.smithi112:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.3 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-11T06:15:25.445 INFO:teuthology.orchestra.run.smithi112.stdout:[client.mirror.3] 2024-09-11T06:15:25.445 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQArNeFmgyq2KBAAYUjwbCzNdqe+13lAXmcBqA== 2024-09-11T06:15:25.446 INFO:teuthology.orchestra.run.smithi112.stdout: caps mon = "profile rbd-mirror" 2024-09-11T06:15:25.446 INFO:teuthology.orchestra.run.smithi112.stdout: caps osd = "profile rbd" 2024-09-11T06:15:25.446 INFO:teuthology.orchestra.run.smithi112.stderr:updated caps for client.mirror.3 2024-09-11T06:15:25.463 DEBUG:teuthology.orchestra.run.smithi112:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror mon '"'"'profile rbd-mirror-peer'"'"' osd '"'"'profile rbd'"'"'' 2024-09-11T06:15:25.800 INFO:teuthology.orchestra.run.smithi112.stdout:[client.mirror] 2024-09-11T06:15:25.800 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQBSNeFm6oFmKRAAkxjreuWFM6Q0zObosK44zw== 2024-09-11T06:15:25.801 INFO:teuthology.orchestra.run.smithi112.stdout: caps mon = "profile rbd-mirror-peer" 2024-09-11T06:15:25.801 INFO:teuthology.orchestra.run.smithi112.stdout: caps osd = "profile rbd" 2024-09-11T06:15:25.801 INFO:teuthology.orchestra.run.smithi112.stderr:updated caps for client.mirror 2024-09-11T06:15:25.819 DEBUG:teuthology.orchestra.run.smithi112:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.0 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-11T06:15:26.178 INFO:teuthology.orchestra.run.smithi112.stdout:[client.mirror.0] 2024-09-11T06:15:26.178 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQBSNeFmEKuJLhAAvnG5ZXNLcfgXZQB8KsAz5Q== 2024-09-11T06:15:26.179 INFO:teuthology.orchestra.run.smithi112.stdout: caps mon = "profile rbd-mirror" 2024-09-11T06:15:26.179 INFO:teuthology.orchestra.run.smithi112.stdout: caps osd = "profile rbd" 2024-09-11T06:15:26.179 INFO:teuthology.orchestra.run.smithi112.stderr:updated caps for client.mirror.0 2024-09-11T06:15:26.198 DEBUG:teuthology.orchestra.run.smithi112:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.1 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-11T06:15:26.536 INFO:teuthology.orchestra.run.smithi112.stdout:[client.mirror.1] 2024-09-11T06:15:26.536 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQBSNeFmZQlZMxAAOBZ/Yc+Xge4iPsnUKTQ+Uw== 2024-09-11T06:15:26.536 INFO:teuthology.orchestra.run.smithi112.stdout: caps mon = "profile rbd-mirror" 2024-09-11T06:15:26.536 INFO:teuthology.orchestra.run.smithi112.stdout: caps osd = "profile rbd" 2024-09-11T06:15:26.537 INFO:teuthology.orchestra.run.smithi112.stderr:updated caps for client.mirror.1 2024-09-11T06:15:26.555 DEBUG:teuthology.orchestra.run.smithi112:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.2 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-11T06:15:26.904 INFO:teuthology.orchestra.run.smithi112.stdout:[client.mirror.2] 2024-09-11T06:15:26.904 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQBSNeFmTZgqOBAABumsMdfKEU9jaQkT1Oh05A== 2024-09-11T06:15:26.904 INFO:teuthology.orchestra.run.smithi112.stdout: caps mon = "profile rbd-mirror" 2024-09-11T06:15:26.904 INFO:teuthology.orchestra.run.smithi112.stdout: caps osd = "profile rbd" 2024-09-11T06:15:26.904 INFO:teuthology.orchestra.run.smithi112.stderr:updated caps for client.mirror.2 2024-09-11T06:15:26.922 DEBUG:teuthology.orchestra.run.smithi112:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.3 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-11T06:15:27.265 INFO:teuthology.orchestra.run.smithi112.stdout:[client.mirror.3] 2024-09-11T06:15:27.265 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQBTNeFm5s/qARAAsP8x/gZnojn3eS26pNU8UQ== 2024-09-11T06:15:27.265 INFO:teuthology.orchestra.run.smithi112.stdout: caps mon = "profile rbd-mirror" 2024-09-11T06:15:27.265 INFO:teuthology.orchestra.run.smithi112.stdout: caps osd = "profile rbd" 2024-09-11T06:15:27.265 INFO:teuthology.orchestra.run.smithi112.stderr:updated caps for client.mirror.3 2024-09-11T06:15:27.284 INFO:teuthology.run_tasks:Running task workunit... 2024-09-11T06:15:27.299 INFO:tasks.workunit:Pulling workunits from ref daaf78de334647fc0c800f4ac495cc9475a26bae 2024-09-11T06:15:27.299 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-11T06:15:27.300 DEBUG:teuthology.orchestra.run.smithi112:> stat -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-09-11T06:15:27.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T06:15:27.305 INFO:teuthology.orchestra.run.smithi112.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.cluster1.mirror': No such file or directory 2024-09-11T06:15:27.305 DEBUG:teuthology.orchestra.run.smithi112:> mkdir -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-09-11T06:15:27.353 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-09-11T06:15:27.353 DEBUG:teuthology.orchestra.run.smithi112:> cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror && mkdir -- client.mirror 2024-09-11T06:15:27.401 INFO:tasks.workunit:timeout=3h 2024-09-11T06:15:27.401 INFO:tasks.workunit:cleanup=True 2024-09-11T06:15:27.402 DEBUG:teuthology.orchestra.run.smithi112:> rm -rf /home/ubuntu/cephtest/clone.cluster1.client.mirror && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.cluster1.client.mirror && cd /home/ubuntu/cephtest/clone.cluster1.client.mirror && git checkout daaf78de334647fc0c800f4ac495cc9475a26bae 2024-09-11T06:15:27.451 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:Cloning into '/home/ubuntu/cephtest/clone.cluster1.client.mirror'... 2024-09-11T06:16:22.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:Note: switching to 'daaf78de334647fc0c800f4ac495cc9475a26bae'. 2024-09-11T06:16:22.812 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: 2024-09-11T06:16:22.812 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-11T06:16:22.812 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-11T06:16:22.812 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:state without impacting any branches by switching back to a branch. 2024-09-11T06:16:22.812 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: 2024-09-11T06:16:22.812 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-11T06:16:22.812 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-11T06:16:22.812 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: 2024-09-11T06:16:22.812 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: git switch -c 2024-09-11T06:16:22.813 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: 2024-09-11T06:16:22.813 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:Or undo this operation with: 2024-09-11T06:16:22.813 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: 2024-09-11T06:16:22.813 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: git switch - 2024-09-11T06:16:22.813 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: 2024-09-11T06:16:22.813 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-11T06:16:22.813 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: 2024-09-11T06:16:22.813 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:HEAD is now at daaf78de334 Merge pull request #59682 from zdover23/wip-doc-2024-09-10-backport-59677-to-squid 2024-09-11T06:16:22.818 DEBUG:teuthology.orchestra.run.smithi112:> cd -- /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.cluster1.client.mirror 2024-09-11T06:16:22.868 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-09-11T06:16:22.871 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/direct_io' 2024-09-11T06:16:22.871 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-09-11T06:16:23.634 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-09-11T06:16:23.711 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-09-11T06:16:23.758 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/direct_io' 2024-09-11T06:16:23.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/fs' 2024-09-11T06:16:23.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-09-11T06:16:23.804 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/fs' 2024-09-11T06:16:23.807 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-11T06:16:23.808 DEBUG:teuthology.orchestra.run.smithi112:> dd if=/home/ubuntu/cephtest/workunits.list.cluster1.client.mirror of=/dev/stdout 2024-09-11T06:16:23.859 INFO:tasks.workunit:Running workunits matching rbd/rbd_mirror_stress.sh on cluster1.client.mirror... 2024-09-11T06:16:23.860 INFO:tasks.workunit:Running workunit rbd/rbd_mirror_stress.sh... 2024-09-11T06:16:23.860 DEBUG:teuthology.orchestra.run.smithi112:workunit test rbd/rbd_mirror_stress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=daaf78de334647fc0c800f4ac495cc9475a26bae TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS='' RBD_MIRROR_MIN_COMPAT_CLIENT=octopus RBD_MIRROR_USE_EXISTING_CLUSTER=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2024-09-11T06:16:23.912 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ IMAGE_COUNT=50 2024-09-11T06:16:23.912 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ export LOCKDEP=0 2024-09-11T06:16:23.912 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ LOCKDEP=0 2024-09-11T06:16:23.912 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ dirname /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2024-09-11T06:16:23.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ . /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_helpers.sh 2024-09-11T06:16:23.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ type xmlstarlet 2024-09-11T06:16:23.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ XMLSTARLET=xmlstarlet 2024-09-11T06:16:23.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ RBD_MIRROR_INSTANCES=2 2024-09-11T06:16:23.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ CLUSTER1=cluster1 2024-09-11T06:16:23.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ CLUSTER2=cluster2 2024-09-11T06:16:23.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ PEER_CLUSTER_SUFFIX= 2024-09-11T06:16:23.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ POOL=mirror 2024-09-11T06:16:23.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ PARENT_POOL=mirror_parent 2024-09-11T06:16:23.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ NS1=ns1 2024-09-11T06:16:23.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ NS2=ns2 2024-09-11T06:16:23.915 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ TEMPDIR= 2024-09-11T06:16:23.915 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ CEPH_ID=mirror 2024-09-11T06:16:23.915 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ RBD_IMAGE_FEATURES=layering,exclusive-lock,journaling 2024-09-11T06:16:23.915 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ MIRROR_USER_ID_PREFIX=mirror. 2024-09-11T06:16:23.915 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ RBD_MIRROR_MODE=journal 2024-09-11T06:16:23.915 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ MIRROR_POOL_MODE=pool 2024-09-11T06:16:23.915 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' journal = snapshot ']' 2024-09-11T06:16:23.915 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ export 'CEPH_ARGS=--id mirror' 2024-09-11T06:16:23.915 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ CEPH_ARGS='--id mirror' 2024-09-11T06:16:23.915 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ LAST_MIRROR_INSTANCE=1 2024-09-11T06:16:23.915 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ dirname /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2024-09-11T06:16:23.916 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ readlink -f /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/../../../src 2024-09-11T06:16:23.916 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror/src 2024-09-11T06:16:23.916 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ CEPH_BIN=. 2024-09-11T06:16:23.916 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ CEPH_SRC=. 2024-09-11T06:16:23.916 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' -e CMakeCache.txt ']' 2024-09-11T06:16:23.917 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 -gt 0 ']' 2024-09-11T06:16:23.917 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ setup 2024-09-11T06:16:23.917 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local c 2024-09-11T06:16:23.917 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ trap 'cleanup $?' INT TERM EXIT 2024-09-11T06:16:23.918 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ setup_tempdir 2024-09-11T06:16:23.918 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n '' ']' 2024-09-11T06:16:23.918 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ mktemp -d 2024-09-11T06:16:23.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ TEMPDIR=/tmp/tmp.Dxwd1ESqKa 2024-09-11T06:16:23.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -z 1 ']' 2024-09-11T06:16:23.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ setup_pools cluster1 cluster2 2024-09-11T06:16:23.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:16:23.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local remote_cluster=cluster2 2024-09-11T06:16:23.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local mon_map_file 2024-09-11T06:16:23.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local mon_addr 2024-09-11T06:16:23.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local admin_key_file 2024-09-11T06:16:23.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local uuid 2024-09-11T06:16:23.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ CEPH_ARGS= 2024-09-11T06:16:23.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --cluster cluster1 osd pool create mirror 64 64 2024-09-11T06:16:24.394 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:pool 'mirror' already exists 2024-09-11T06:16:24.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ CEPH_ARGS= 2024-09-11T06:16:24.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --cluster cluster1 osd pool create mirror_parent 64 64 2024-09-11T06:16:25.412 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:pool 'mirror_parent' already exists 2024-09-11T06:16:25.429 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ CEPH_ARGS= 2024-09-11T06:16:25.430 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 pool init mirror 2024-09-11T06:16:28.375 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ CEPH_ARGS= 2024-09-11T06:16:28.375 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 pool init mirror_parent 2024-09-11T06:16:31.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n '' ']' 2024-09-11T06:16:31.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ CEPH_ARGS= 2024-09-11T06:16:31.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 mirror pool enable --site-name cluster1 mirror pool 2024-09-11T06:16:31.469 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 mirror pool enable mirror_parent image 2024-09-11T06:16:31.517 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 namespace create mirror/ns1 2024-09-11T06:16:31.565 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 namespace create mirror/ns2 2024-09-11T06:16:31.612 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 mirror pool enable mirror/ns1 pool 2024-09-11T06:16:31.665 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 mirror pool enable mirror/ns2 image 2024-09-11T06:16:31.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -z ']' 2024-09-11T06:16:31.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -z ']' 2024-09-11T06:16:31.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ peer_add cluster1 mirror cluster2 2024-09-11T06:16:31.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:16:31.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:16:31.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:16:31.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:16:31.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local client_cluster=cluster2 2024-09-11T06:16:31.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:16:31.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local remote_cluster=cluster2 2024-09-11T06:16:31.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local uuid_var_name 2024-09-11T06:16:31.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n '' ']' 2024-09-11T06:16:31.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local error_code 2024-09-11T06:16:31.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local peer_uuid 2024-09-11T06:16:31.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 16 32 2024-09-11T06:16:31.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set +e 2024-09-11T06:16:31.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster1 mirror pool peer add mirror cluster2 2024-09-11T06:16:31.757 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ peer_uuid=6a4eb298-6275-47c2-ac3f-ccf3a04c278b 2024-09-11T06:16:31.758 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ error_code=0 2024-09-11T06:16:31.758 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set -e 2024-09-11T06:16:31.758 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 -eq 17 ']' 2024-09-11T06:16:31.758 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 0 -eq 0 2024-09-11T06:16:31.758 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n '' ']' 2024-09-11T06:16:31.758 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:16:31.758 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ peer_add cluster1 mirror_parent cluster2 2024-09-11T06:16:31.758 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:16:31.758 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:16:31.758 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror_parent 2024-09-11T06:16:31.758 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:16:31.758 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local client_cluster=cluster2 2024-09-11T06:16:31.758 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:16:31.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local remote_cluster=cluster2 2024-09-11T06:16:31.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local uuid_var_name 2024-09-11T06:16:31.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n '' ']' 2024-09-11T06:16:31.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local error_code 2024-09-11T06:16:31.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local peer_uuid 2024-09-11T06:16:31.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 16 32 2024-09-11T06:16:31.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set +e 2024-09-11T06:16:31.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster1 mirror pool peer add mirror_parent cluster2 2024-09-11T06:16:31.801 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ peer_uuid=b8288773-8be8-42fd-af01-e952dc01e5fb 2024-09-11T06:16:31.801 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ error_code=0 2024-09-11T06:16:31.801 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set -e 2024-09-11T06:16:31.801 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 -eq 17 ']' 2024-09-11T06:16:31.801 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 0 -eq 0 2024-09-11T06:16:31.802 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n '' ']' 2024-09-11T06:16:31.802 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:16:31.802 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ setup_pools cluster2 cluster1 2024-09-11T06:16:31.802 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:16:31.802 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local remote_cluster=cluster1 2024-09-11T06:16:31.802 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local mon_map_file 2024-09-11T06:16:31.802 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local mon_addr 2024-09-11T06:16:31.802 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local admin_key_file 2024-09-11T06:16:31.802 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local uuid 2024-09-11T06:16:31.802 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ CEPH_ARGS= 2024-09-11T06:16:31.802 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --cluster cluster2 osd pool create mirror 64 64 2024-09-11T06:16:32.468 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:pool 'mirror' already exists 2024-09-11T06:16:32.484 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ CEPH_ARGS= 2024-09-11T06:16:32.485 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --cluster cluster2 osd pool create mirror_parent 64 64 2024-09-11T06:16:33.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:pool 'mirror_parent' already exists 2024-09-11T06:16:33.522 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ CEPH_ARGS= 2024-09-11T06:16:33.522 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 pool init mirror 2024-09-11T06:16:36.418 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ CEPH_ARGS= 2024-09-11T06:16:36.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 pool init mirror_parent 2024-09-11T06:16:39.441 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n '' ']' 2024-09-11T06:16:39.442 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ CEPH_ARGS= 2024-09-11T06:16:39.442 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 mirror pool enable --site-name cluster2 mirror pool 2024-09-11T06:16:39.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 mirror pool enable mirror_parent image 2024-09-11T06:16:39.535 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 namespace create mirror/ns1 2024-09-11T06:16:39.575 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 namespace create mirror/ns2 2024-09-11T06:16:39.618 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 mirror pool enable mirror/ns1 pool 2024-09-11T06:16:39.659 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 mirror pool enable mirror/ns2 image 2024-09-11T06:16:39.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -z ']' 2024-09-11T06:16:39.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -z ']' 2024-09-11T06:16:39.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ peer_add cluster2 mirror cluster1 2024-09-11T06:16:39.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:16:39.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:16:39.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:16:39.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:16:39.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local client_cluster=cluster1 2024-09-11T06:16:39.700 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:16:39.700 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local remote_cluster=cluster1 2024-09-11T06:16:39.700 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local uuid_var_name 2024-09-11T06:16:39.700 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n '' ']' 2024-09-11T06:16:39.700 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local error_code 2024-09-11T06:16:39.700 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local peer_uuid 2024-09-11T06:16:39.700 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 16 32 2024-09-11T06:16:39.700 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set +e 2024-09-11T06:16:39.700 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 mirror pool peer add mirror cluster1 2024-09-11T06:16:39.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ peer_uuid=7c8f27fe-aee8-4aac-bbb7-107e57eeee09 2024-09-11T06:16:39.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ error_code=0 2024-09-11T06:16:39.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set -e 2024-09-11T06:16:39.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 -eq 17 ']' 2024-09-11T06:16:39.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 0 -eq 0 2024-09-11T06:16:39.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n '' ']' 2024-09-11T06:16:39.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:16:39.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ peer_add cluster2 mirror_parent cluster1 2024-09-11T06:16:39.739 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:16:39.739 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:16:39.739 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror_parent 2024-09-11T06:16:39.739 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:16:39.739 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local client_cluster=cluster1 2024-09-11T06:16:39.739 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:16:39.739 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local remote_cluster=cluster1 2024-09-11T06:16:39.739 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local uuid_var_name 2024-09-11T06:16:39.739 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n '' ']' 2024-09-11T06:16:39.739 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local error_code 2024-09-11T06:16:39.739 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local peer_uuid 2024-09-11T06:16:39.739 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 16 32 2024-09-11T06:16:39.739 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set +e 2024-09-11T06:16:39.740 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 mirror pool peer add mirror_parent cluster1 2024-09-11T06:16:39.777 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ peer_uuid=50094659-8cb2-43e7-a877-eebbaee9b121 2024-09-11T06:16:39.777 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ error_code=0 2024-09-11T06:16:39.777 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set -e 2024-09-11T06:16:39.777 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 -eq 17 ']' 2024-09-11T06:16:39.777 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 0 -eq 0 2024-09-11T06:16:39.777 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n '' ']' 2024-09-11T06:16:39.777 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:16:39.777 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n octopus ']' 2024-09-11T06:16:39.777 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ CEPH_ARGS= 2024-09-11T06:16:39.777 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --cluster cluster1 osd set-require-min-compat-client octopus 2024-09-11T06:16:41.977 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:set require_min_compat_client to octopus 2024-09-11T06:16:42.005 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ CEPH_ARGS= 2024-09-11T06:16:42.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --cluster cluster2 osd set-require-min-compat-client octopus 2024-09-11T06:16:44.406 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:set require_min_compat_client to octopus 2024-09-11T06:16:44.433 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ start_mirrors cluster1 2024-09-11T06:16:44.433 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:16:44.433 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:16:44.551 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:16:44.551 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ start_mirror cluster1:0 2024-09-11T06:16:44.551 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:16:44.551 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:16:44.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:16:44.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:16:44.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:16:44.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:16:44.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:16:44.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:16:44.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:16:44.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:16:44.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:16:44.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:16:44.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:16:44.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:16:44.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd-mirror --cluster cluster1 --id mirror.0 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2024-09-11T06:16:44.592 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:16:44.592 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ start_mirror cluster1:1 2024-09-11T06:16:44.592 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:1 2024-09-11T06:16:44.592 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:16:44.592 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-11T06:16:44.592 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:1 2024-09-11T06:16:44.592 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:16:44.593 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:16:44.593 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:16:44.593 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=1 2024-09-11T06:16:44.593 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-11T06:16:44.593 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:16:44.593 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:16:44.593 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=1 2024-09-11T06:16:44.593 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=1 2024-09-11T06:16:44.593 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:16:44.593 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd-mirror --cluster cluster1 --id mirror.1 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2024-09-11T06:16:44.626 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ start_mirrors cluster2 2024-09-11T06:16:44.626 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:16:44.626 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:16:44.627 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:16:44.628 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ start_mirror cluster2:0 2024-09-11T06:16:44.628 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2:0 2024-09-11T06:16:44.628 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:16:44.628 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-09-11T06:16:44.628 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster2:0 2024-09-11T06:16:44.628 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:16:44.628 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:16:44.628 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster2 2024-09-11T06:16:44.628 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:16:44.628 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster2:0 ']' 2024-09-11T06:16:44.628 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster2 2024-09-11T06:16:44.628 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster2 2024-09-11T06:16:44.628 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:16:44.629 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:16:44.629 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:16:44.629 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd-mirror --cluster cluster2 --id mirror.0 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2024-09-11T06:16:44.661 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:16:44.661 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ start_mirror cluster2:1 2024-09-11T06:16:44.661 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2:1 2024-09-11T06:16:44.661 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:16:44.661 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster2:1 cluster instance 2024-09-11T06:16:44.661 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster2:1 2024-09-11T06:16:44.661 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:16:44.661 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:16:44.661 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster2 2024-09-11T06:16:44.661 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=1 2024-09-11T06:16:44.661 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 1 = cluster2:1 ']' 2024-09-11T06:16:44.661 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster2 2024-09-11T06:16:44.661 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster2 2024-09-11T06:16:44.662 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=1 2024-09-11T06:16:44.662 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=1 2024-09-11T06:16:44.662 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:16:44.662 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd-mirror --cluster cluster2 --id mirror.1 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2024-09-11T06:16:44.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ testlog 'TEST: add image and test replay after client crashes' 2024-09-11T06:16:44.700 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ tee -a /tmp/tmp.Dxwd1ESqKa/rbd-mirror.test.log 2024-09-11T06:16:44.700 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ date '+%F %T' 2024-09-11T06:16:44.701 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ echo 2024-09-11 06:16:44 TEST: add image and test replay after client crashes 2024-09-11T06:16:44.702 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:2024-09-11 06:16:44 TEST: add image and test replay after client crashes 2024-09-11T06:16:44.702 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=test 2024-09-11T06:16:44.702 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_image_and_enable_mirror cluster2 mirror test journal 512M 2024-09-11T06:16:44.702 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:16:44.702 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:16:44.702 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:16:44.702 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:16:44.702 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:16:44.702 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:16:44.702 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local mode=journal 2024-09-11T06:16:44.702 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n journal ']' 2024-09-11T06:16:44.702 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:16:44.702 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_image cluster2 mirror test 512M 2024-09-11T06:16:44.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:16:44.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:16:44.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:16:44.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:16:44.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:16:44.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:16:44.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local size=128 2024-09-11T06:16:44.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n 512M ']' 2024-09-11T06:16:44.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ size=512M 2024-09-11T06:16:44.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:16:44.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,journaling mirror/test 2024-09-11T06:16:44.776 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T06:16:44.776 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:16:44.776 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' pool = image ']' 2024-09-11T06:16:44.776 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ case "${pool}" in 2024-09-11T06:16:44.776 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 1 2024-09-11T06:16:44.776 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-11T06:16:44.776 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:16:44.776 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:16:44.777 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:16:44.777 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-11T06:16:44.777 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:16:44.777 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:16:44.777 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:16:44.777 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=started 2024-09-11T06:16:44.777 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:16:44.777 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:16:44.777 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:16:45.777 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-11T06:16:45.777 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:16:45.778 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:16:45.778 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:16:45.778 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=started 2024-09-11T06:16:45.778 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local status_result 2024-09-11T06:16:45.778 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=stopped 2024-09-11T06:16:45.779 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-11T06:16:45.779 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_instance=cluster1 2024-09-11T06:16:45.779 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:16:45.779 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -i state 2024-09-11T06:16:45.779 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:16:45.779 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance=cluster1 2024-09-11T06:16:45.779 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local loop_instance 2024-09-11T06:16:45.779 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:16:45.779 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sleep 0 2024-09-11T06:16:45.780 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:16:45.781 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ seq 0 1 2024-09-11T06:16:45.783 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:16:45.783 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-11T06:16:45.783 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1:0 2024-09-11T06:16:45.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:16:45.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:16:45.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:16:45.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:16:45.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=cluster 2024-09-11T06:16:45.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:16:45.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:16:45.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:16:45.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:16:45.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval cluster=cluster1 2024-09-11T06:16:45.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:16:45.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:16:45.785 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:16:45.785 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:16:45.785 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local local_cluster=cluster1:0 2024-09-11T06:16:45.785 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster=cluster1 2024-09-11T06:16:45.785 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance 2024-09-11T06:16:45.785 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:16:45.785 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local val=cluster1:0 2024-09-11T06:16:45.785 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T06:16:45.785 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance_var_name=instance 2024-09-11T06:16:45.785 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:16:45.785 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:16:45.785 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T06:16:45.785 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval local_cluster=cluster1 2024-09-11T06:16:45.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ local_cluster=cluster1 2024-09-11T06:16:45.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval instance=0 2024-09-11T06:16:45.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ instance=0 2024-09-11T06:16:45.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:16:45.804 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:16:45.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:16:45.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:16:45.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-11T06:16:45.912 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-11T06:16:45.912 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T06:16:45.912 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:admin_socket: invalid command 2024-09-11T06:16:45.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:16:45.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-11T06:16:45.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1:1 2024-09-11T06:16:45.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:16:45.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:16:45.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-11T06:16:45.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:1 2024-09-11T06:16:45.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=cluster 2024-09-11T06:16:45.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:16:45.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:16:45.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=1 2024-09-11T06:16:45.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T06:16:45.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval cluster=cluster1 2024-09-11T06:16:45.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:16:45.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=1 2024-09-11T06:16:45.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=1 2024-09-11T06:16:45.915 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-11T06:16:45.916 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local local_cluster=cluster1:1 2024-09-11T06:16:45.916 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster=cluster1 2024-09-11T06:16:45.916 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance 2024-09-11T06:16:45.916 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T06:16:45.916 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local val=cluster1:1 2024-09-11T06:16:45.916 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T06:16:45.916 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance_var_name=instance 2024-09-11T06:16:45.916 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:16:45.916 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=1 2024-09-11T06:16:45.916 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-11T06:16:45.916 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval local_cluster=cluster1 2024-09-11T06:16:45.916 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ local_cluster=cluster1 2024-09-11T06:16:45.917 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval instance=1 2024-09-11T06:16:45.917 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ instance=1 2024-09-11T06:16:45.917 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T06:16:45.940 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:16:45.940 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:16:45.940 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:16:45.941 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-11T06:16:46.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-11T06:16:46.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T06:16:46.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:admin_socket: invalid command 2024-09-11T06:16:46.040 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:16:46.040 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sleep 1 2024-09-11T06:16:47.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:16:47.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ seq 0 1 2024-09-11T06:16:47.046 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:16:47.046 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-11T06:16:47.046 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1:0 2024-09-11T06:16:47.046 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:16:47.046 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:16:47.046 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:16:47.046 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:16:47.047 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=cluster 2024-09-11T06:16:47.047 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:16:47.047 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:16:47.047 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:16:47.047 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:16:47.047 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval cluster=cluster1 2024-09-11T06:16:47.047 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:16:47.047 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:16:47.047 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:16:47.047 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:16:47.047 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local local_cluster=cluster1:0 2024-09-11T06:16:47.047 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster=cluster1 2024-09-11T06:16:47.047 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance 2024-09-11T06:16:47.047 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:16:47.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local val=cluster1:0 2024-09-11T06:16:47.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T06:16:47.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance_var_name=instance 2024-09-11T06:16:47.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:16:47.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:16:47.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T06:16:47.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval local_cluster=cluster1 2024-09-11T06:16:47.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ local_cluster=cluster1 2024-09-11T06:16:47.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval instance=0 2024-09-11T06:16:47.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ instance=0 2024-09-11T06:16:47.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:16:47.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:16:47.071 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:16:47.071 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:16:47.071 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-11T06:16:47.176 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-11T06:16:47.176 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T06:16:47.177 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:admin_socket: invalid command 2024-09-11T06:16:47.179 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:16:47.179 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-11T06:16:47.179 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1:1 2024-09-11T06:16:47.179 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:16:47.179 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:16:47.179 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-11T06:16:47.179 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:1 2024-09-11T06:16:47.179 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=cluster 2024-09-11T06:16:47.180 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:16:47.180 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:16:47.180 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=1 2024-09-11T06:16:47.180 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T06:16:47.180 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval cluster=cluster1 2024-09-11T06:16:47.180 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:16:47.180 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=1 2024-09-11T06:16:47.180 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=1 2024-09-11T06:16:47.180 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-11T06:16:47.180 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local local_cluster=cluster1:1 2024-09-11T06:16:47.181 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster=cluster1 2024-09-11T06:16:47.181 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance 2024-09-11T06:16:47.181 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T06:16:47.181 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local val=cluster1:1 2024-09-11T06:16:47.181 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T06:16:47.181 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance_var_name=instance 2024-09-11T06:16:47.181 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:16:47.181 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=1 2024-09-11T06:16:47.181 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-11T06:16:47.181 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval local_cluster=cluster1 2024-09-11T06:16:47.181 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ local_cluster=cluster1 2024-09-11T06:16:47.181 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval instance=1 2024-09-11T06:16:47.181 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ instance=1 2024-09-11T06:16:47.182 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T06:16:47.202 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:16:47.202 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:16:47.203 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:16:47.203 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-11T06:16:47.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-11T06:16:47.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T06:16:47.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:admin_socket: invalid command 2024-09-11T06:16:47.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:16:47.305 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sleep 2 2024-09-11T06:16:49.307 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:16:49.307 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ seq 0 1 2024-09-11T06:16:49.310 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:16:49.310 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-11T06:16:49.310 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1:0 2024-09-11T06:16:49.310 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:16:49.310 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:16:49.311 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:16:49.311 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:16:49.311 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=cluster 2024-09-11T06:16:49.311 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:16:49.311 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:16:49.311 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:16:49.311 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:16:49.311 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval cluster=cluster1 2024-09-11T06:16:49.311 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:16:49.311 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:16:49.311 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:16:49.311 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:16:49.311 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local local_cluster=cluster1:0 2024-09-11T06:16:49.312 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster=cluster1 2024-09-11T06:16:49.312 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance 2024-09-11T06:16:49.312 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:16:49.312 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local val=cluster1:0 2024-09-11T06:16:49.312 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T06:16:49.312 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance_var_name=instance 2024-09-11T06:16:49.312 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:16:49.313 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:16:49.313 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T06:16:49.313 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval local_cluster=cluster1 2024-09-11T06:16:49.313 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ local_cluster=cluster1 2024-09-11T06:16:49.313 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval instance=0 2024-09-11T06:16:49.313 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ instance=0 2024-09-11T06:16:49.313 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:16:49.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:16:49.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:16:49.337 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:16:49.337 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-11T06:16:49.463 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ return 0 2024-09-11T06:16:49.464 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T06:16:49.464 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ echo ' "state": "Replaying"' 2024-09-11T06:16:49.464 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ grep -i Replaying 2024-09-11T06:16:49.466 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: "state": "Replaying" 2024-09-11T06:16:49.467 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_state=started 2024-09-11T06:16:49.467 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test started = started 2024-09-11T06:16:49.467 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:16:49.467 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ clean_snap_name= 2024-09-11T06:16:49.467 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 1 10 2024-09-11T06:16:49.468 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 10` 2024-09-11T06:16:49.468 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ stress_write_image cluster2 mirror test 2024-09-11T06:16:49.468 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:16:49.468 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:16:49.469 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:16:49.469 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T06:16:49.470 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local duration=9 2024-09-11T06:16:49.470 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set +e 2024-09-11T06:16:49.470 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-11T06:16:50.525 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:100 IOs, 28380160 bytes 2024-09-11T06:16:51.237 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:200 IOs, 61948928 bytes 2024-09-11T06:16:52.357 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:300 IOs, 86878208 bytes 2024-09-11T06:16:54.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:400 IOs, 149152768 bytes 2024-09-11T06:16:55.966 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:500 IOs, 214526976 bytes 2024-09-11T06:16:58.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:600 IOs, 310696960 bytes 2024-09-11T06:16:58.477 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ error_code=124 2024-09-11T06:16:58.478 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set -e 2024-09-11T06:16:58.478 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T06:16:58.478 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:16:58.478 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-11T06:16:58.478 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:16:58.478 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:16:58.478 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:16:58.478 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state_pattern=up+replaying 2024-09-11T06:16:58.478 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local description_pattern= 2024-09-11T06:16:58.478 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local service_pattern= 2024-09-11T06:16:58.478 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:16:58.478 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:16:59.479 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-11T06:16:59.479 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:16:59.479 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:16:59.479 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:16:59.479 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state_pattern=up+replaying 2024-09-11T06:16:59.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local description_pattern= 2024-09-11T06:16:59.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local service_pattern= 2024-09-11T06:16:59.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local status 2024-09-11T06:16:59.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ CEPH_ARGS= 2024-09-11T06:16:59.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-11T06:16:59.561 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ status='test: 2024-09-11T06:16:59.561 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: global_id: a0f7d515-874e-4c67-a26b-4958c7ab3083 2024-09-11T06:16:59.562 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: state: up+replaying 2024-09-11T06:16:59.562 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}} 2024-09-11T06:16:59.562 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: service: mirror.0 on smithi112 2024-09-11T06:16:59.562 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: last_update: 2024-09-11 06:16:48 2024-09-11T06:16:59.562 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: peer_sites: 2024-09-11T06:16:59.562 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: name: cluster2 2024-09-11T06:16:59.562 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: state: up+stopped 2024-09-11T06:16:59.562 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: description: local image is primary 2024-09-11T06:16:59.562 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: last_update: 2024-09-11 06:16:54' 2024-09-11T06:16:59.562 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ grep '^ state: .*up+replaying' 2024-09-11T06:16:59.563 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: state: up+replaying 2024-09-11T06:16:59.563 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ grep '^ description: .*' 2024-09-11T06:16:59.563 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}} 2024-09-11T06:16:59.564 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n '' ']' 2024-09-11T06:16:59.564 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ echo up+replaying 2024-09-11T06:16:59.564 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ grep '^up+' 2024-09-11T06:16:59.564 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:up+replaying 2024-09-11T06:16:59.565 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ grep 'service: *mirror..* on ' 2024-09-11T06:16:59.565 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: service: mirror.0 on smithi112 2024-09-11T06:16:59.565 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local last_update 2024-09-11T06:16:59.565 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-11T06:16:59.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_update='2024-09-11 06:16:48' 2024-09-11T06:16:59.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-11 06:16:48' 2024-09-11T06:16:59.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:16:59.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:16:59.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:16:59.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state_pattern=up+replaying 2024-09-11T06:16:59.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'prev_last_update=2024-09-11 06:16:48' 2024-09-11T06:16:59.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local status 2024-09-11T06:16:59.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ CEPH_ARGS= 2024-09-11T06:16:59.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-11T06:16:59.645 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ status='OKOKOK142554262mirror.0smithi11219.1.1-298-gb53b6d3etrueOK42594265mirror.1smithi11219.1.1-298-gb53b6d3efalseOKtesta0f7d515-874e-4c67-a26b-4958c7ab3083up+replayingreplaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}}42554262mirror.0smithi1122024-09-11 06:16:48cluster20bfc6eda-f3c5-424a-8ff7-5c4d3d9fd488up+stoppedlocal image is primary2024-09-11 06:16:54' 2024-09-11T06:16:59.646 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local last_update state 2024-09-11T06:16:59.646 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-11T06:16:59.677 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_update='2024-09-11 06:16:48' 2024-09-11T06:16:59.678 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-11T06:16:59.681 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ state=up+replaying 2024-09-11T06:16:59.681 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ echo up+replaying 2024-09-11T06:16:59.681 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ grep up+replaying 2024-09-11T06:16:59.682 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:up+replaying 2024-09-11T06:16:59.682 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:16:59.682 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:16:59.682 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ snap_name=snap1 2024-09-11T06:16:59.682 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_snap cluster2 mirror test snap1 2024-09-11T06:16:59.683 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:16:59.683 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:16:59.683 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:16:59.683 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap1 2024-09-11T06:16:59.683 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap1 --debug-rbd=20 --debug-journaler=20 2024-09-11T06:17:31.451 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-11T06:17:31.451 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:17:31.451 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:17:31.451 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:17:31.451 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-11T06:17:31.451 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:17:31.451 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:17:31.452 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:17:31.452 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=started 2024-09-11T06:17:31.452 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:17:31.452 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:17:31.452 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:17:32.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-11T06:17:32.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:17:32.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:17:32.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:17:32.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=started 2024-09-11T06:17:32.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local status_result 2024-09-11T06:17:32.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=stopped 2024-09-11T06:17:32.455 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-11T06:17:32.455 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -i state 2024-09-11T06:17:32.455 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_instance=cluster1 2024-09-11T06:17:32.455 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:17:32.455 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:17:32.455 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance=cluster1 2024-09-11T06:17:32.455 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local loop_instance 2024-09-11T06:17:32.455 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:17:32.455 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sleep 0 2024-09-11T06:17:32.456 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:17:32.456 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ seq 0 1 2024-09-11T06:17:32.458 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:17:32.458 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-11T06:17:32.458 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1:0 2024-09-11T06:17:32.458 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:17:32.458 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:17:32.458 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:17:32.458 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:17:32.458 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=cluster 2024-09-11T06:17:32.458 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:17:32.458 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:17:32.458 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:17:32.458 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:17:32.459 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval cluster=cluster1 2024-09-11T06:17:32.459 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:17:32.459 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:17:32.460 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:17:32.460 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:17:32.460 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local local_cluster=cluster1:0 2024-09-11T06:17:32.460 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster=cluster1 2024-09-11T06:17:32.460 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance 2024-09-11T06:17:32.460 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:17:32.461 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local val=cluster1:0 2024-09-11T06:17:32.461 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T06:17:32.461 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance_var_name=instance 2024-09-11T06:17:32.461 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:17:32.461 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:17:32.461 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T06:17:32.461 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval local_cluster=cluster1 2024-09-11T06:17:32.461 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ local_cluster=cluster1 2024-09-11T06:17:32.461 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval instance=0 2024-09-11T06:17:32.461 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ instance=0 2024-09-11T06:17:32.461 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:17:32.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:32.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:32.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:32.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-11T06:17:32.602 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ return 0 2024-09-11T06:17:32.603 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T06:17:32.603 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ echo ' "state": "Replaying"' 2024-09-11T06:17:32.604 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ grep -i Replaying 2024-09-11T06:17:32.606 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: "state": "Replaying" 2024-09-11T06:17:32.607 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_state=started 2024-09-11T06:17:32.607 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test started = started 2024-09-11T06:17:32.607 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:17:32.607 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-11T06:17:32.608 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:17:32.608 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:17:32.608 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:17:32.608 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:17:32.608 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' journal = journal ']' 2024-09-11T06:17:32.608 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-09-11T06:17:32.608 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:17:32.608 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:17:32.608 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:17:32.608 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:17:32.608 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-11T06:17:32.608 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-11T06:17:32.608 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:17:32.609 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:17:32.609 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:17:32.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:17:32.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:17:32.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:17:32.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:17:32.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:17:32.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:17:32.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:17:32.810 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:17:32.810 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:17:32.810 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:17:32.810 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:17:32.810 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:17:32.810 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:17:32.810 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:17:32.810 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:17:32.810 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:17:32.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:17:32.813 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:17:32.813 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:17:32.813 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:17:32.813 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:17:32.813 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:17:32.813 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:17:32.813 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:17:32.813 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:17:32.813 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:17:32.813 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:17:32.813 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:17:32.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:17:32.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:17:32.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:17:32.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:17:32.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:17:32.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:17:32.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:17:32.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:17:32.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:17:32.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:17:32.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:17:32.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:17:32.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:17:32.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:17:32.815 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:17:32.815 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:17:32.815 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:17:32.815 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:17:32.815 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:17:32.815 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:17:32.815 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:17:32.840 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:32.840 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:32.840 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:32.840 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:17:32.972 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:17:32.973 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:17:32.973 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:32.973 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:32.973 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:32.973 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:17:32.973 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:32.973 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:32.973 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:32.973 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:17:32.974 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:17:32.974 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:17:32.974 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:17:32.975 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:32.976 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:17:32.976 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:33.021 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 2 2024-09-11T06:17:33.021 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:17:33.021 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:17:33.021 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:17:33.021 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=10, tag_tid=2, entry_tid=9798], [object_number=9, tag_tid=2, entry_tid=9797], [object_number=8, tag_tid=2, entry_tid=9796], [object_number=11, tag_tid=2, entry_tid=9795]]], state=connected] 2024-09-11T06:17:33.024 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:33.025 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:33.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:17:33.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:33.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:33.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:33.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:17:33.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:33.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:33.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:33.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:17:33.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:17:33.028 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:17:33.028 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:17:33.028 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:33.028 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:17:33.028 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:33.073 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 2 2024-09-11T06:17:33.073 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:17:33.073 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:17:33.073 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:17:33.073 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=10, tag_tid=2, entry_tid=9798], [object_number=9, tag_tid=2, entry_tid=9797], [object_number=8, tag_tid=2, entry_tid=9796], [object_number=11, tag_tid=2, entry_tid=9795]]], state=connected] 2024-09-11T06:17:33.076 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:33.078 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=10, tag_tid=2, entry_tid=9798' 2024-09-11T06:17:33.078 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=21, tag_tid=3, entry_tid=1' -a 'object_number=21, tag_tid=3, entry_tid=1' = 'object_number=10, tag_tid=2, entry_tid=9798' 2024-09-11T06:17:33.078 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=10, tag_tid=2, entry_tid=9798' '!=' '' 2024-09-11T06:17:33.078 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:17:33.078 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=10, tag_tid=2, entry_tid=9798' = '' 2024-09-11T06:17:33.079 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=10, tag_tid=2, entry_tid=9798' 2024-09-11T06:17:33.079 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:33.079 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:17:33.079 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:33.079 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=3 2024-09-11T06:17:33.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=10, tag_tid=2, entry_tid=9798' 2024-09-11T06:17:33.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:17:33.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:33.081 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=2 2024-09-11T06:17:33.082 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:33.082 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:17:33.082 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:33.085 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:17:33.085 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=10, tag_tid=2, entry_tid=9798' 2024-09-11T06:17:33.086 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:17:33.086 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:33.086 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=9798 2024-09-11T06:17:33.086 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 3 = 2 -a 1 -le 9798 2024-09-11T06:17:33.086 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:17:33.086 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:17:33.087 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:17:33.287 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:17:33.287 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:17:33.287 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:17:33.287 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:17:33.287 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:17:33.287 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:17:33.287 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:17:33.287 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:17:33.287 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:17:33.287 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:17:33.287 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:17:33.288 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:17:33.288 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:17:33.288 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:17:33.288 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:17:33.288 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:17:33.288 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:17:33.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:17:33.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:17:33.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:17:33.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:17:33.293 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:17:33.293 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:17:33.293 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:17:33.293 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:17:33.293 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:17:33.293 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:17:33.293 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:17:33.293 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:17:33.294 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:17:33.294 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:17:33.294 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:17:33.294 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:17:33.294 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:17:33.294 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:17:33.294 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:17:33.294 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:17:33.294 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:17:33.294 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:17:33.294 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:17:33.294 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:17:33.295 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:17:33.295 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:17:33.295 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:17:33.295 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:17:33.295 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:17:33.295 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:17:33.295 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:17:33.295 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:17:33.319 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:33.319 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:33.319 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:33.319 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:17:33.448 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:17:33.448 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:17:33.448 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:33.448 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:33.448 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:33.448 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:17:33.448 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:33.448 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:33.449 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:33.449 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:17:33.449 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:17:33.449 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:17:33.449 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:17:33.452 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:33.452 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:17:33.452 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:33.502 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 2 2024-09-11T06:17:33.503 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:17:33.503 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:17:33.503 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:17:33.503 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=11, tag_tid=2, entry_tid=9911], [object_number=10, tag_tid=2, entry_tid=9910], [object_number=9, tag_tid=2, entry_tid=9909], [object_number=8, tag_tid=2, entry_tid=9908]]], state=connected] 2024-09-11T06:17:33.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:33.507 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:33.508 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:17:33.508 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:33.509 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:33.509 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:33.509 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:17:33.509 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:33.509 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:33.509 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:33.509 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:17:33.509 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:17:33.509 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:17:33.509 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:17:33.510 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:33.510 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:17:33.510 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:33.553 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 2 2024-09-11T06:17:33.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:17:33.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:17:33.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:17:33.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=11, tag_tid=2, entry_tid=9911], [object_number=10, tag_tid=2, entry_tid=9910], [object_number=9, tag_tid=2, entry_tid=9909], [object_number=8, tag_tid=2, entry_tid=9908]]], state=connected] 2024-09-11T06:17:33.557 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:33.559 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=11, tag_tid=2, entry_tid=9911' 2024-09-11T06:17:33.559 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=21, tag_tid=3, entry_tid=1' -a 'object_number=21, tag_tid=3, entry_tid=1' = 'object_number=11, tag_tid=2, entry_tid=9911' 2024-09-11T06:17:33.559 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=11, tag_tid=2, entry_tid=9911' '!=' 'object_number=10, tag_tid=2, entry_tid=9798' 2024-09-11T06:17:33.559 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:17:33.559 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=11, tag_tid=2, entry_tid=9911' = 'object_number=10, tag_tid=2, entry_tid=9798' 2024-09-11T06:17:33.559 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=11, tag_tid=2, entry_tid=9911' 2024-09-11T06:17:33.559 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:33.559 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:17:33.559 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:33.560 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=3 2024-09-11T06:17:33.561 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=11, tag_tid=2, entry_tid=9911' 2024-09-11T06:17:33.561 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:17:33.561 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:33.562 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=2 2024-09-11T06:17:33.562 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:33.563 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:17:33.563 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:33.563 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:17:33.564 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=11, tag_tid=2, entry_tid=9911' 2024-09-11T06:17:33.564 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:17:33.564 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:33.566 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=9911 2024-09-11T06:17:33.566 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 3 = 2 -a 1 -le 9911 2024-09-11T06:17:33.566 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:17:33.566 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:17:33.566 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:17:33.768 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:17:33.768 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:17:33.768 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:17:33.768 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:17:33.768 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:17:33.768 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:17:33.768 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:17:33.768 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:17:33.768 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:17:33.768 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:17:33.768 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:17:33.769 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:17:33.769 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:17:33.769 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:17:33.769 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:17:33.769 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:17:33.769 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:17:33.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:17:33.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:17:33.772 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:17:33.772 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:17:33.772 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:17:33.772 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:17:33.772 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:17:33.772 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:17:33.772 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:17:33.772 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:17:33.772 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:17:33.772 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:17:33.772 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:17:33.772 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:17:33.772 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:17:33.773 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:17:33.773 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:17:33.773 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:17:33.773 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:17:33.773 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:17:33.773 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:17:33.773 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:17:33.773 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:17:33.773 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:17:33.773 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:17:33.773 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:17:33.773 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:17:33.773 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:17:33.773 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:17:33.774 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:17:33.774 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:17:33.774 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:17:33.796 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:33.797 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:33.797 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:33.797 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:17:33.928 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:17:33.929 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:17:33.929 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:33.929 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:33.930 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:33.930 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:17:33.930 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:33.930 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:33.930 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:33.930 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:17:33.930 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:17:33.930 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:17:33.930 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:17:33.932 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:33.932 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:17:33.932 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:33.977 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 2 2024-09-11T06:17:33.977 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:17:33.977 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:17:33.977 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:17:33.977 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=8, tag_tid=2, entry_tid=10024], [object_number=11, tag_tid=2, entry_tid=10023], [object_number=10, tag_tid=2, entry_tid=10022], [object_number=9, tag_tid=2, entry_tid=10021]]], state=connected] 2024-09-11T06:17:33.980 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:33.982 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:33.983 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:17:33.983 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:33.983 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:33.983 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:33.983 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:17:33.983 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:33.983 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:33.983 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:33.983 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:17:33.983 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:17:33.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:17:33.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:17:33.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:33.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:17:33.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:34.030 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 2 2024-09-11T06:17:34.030 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:17:34.030 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:17:34.030 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:17:34.030 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=8, tag_tid=2, entry_tid=10024], [object_number=11, tag_tid=2, entry_tid=10023], [object_number=10, tag_tid=2, entry_tid=10022], [object_number=9, tag_tid=2, entry_tid=10021]]], state=connected] 2024-09-11T06:17:34.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:34.035 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=8, tag_tid=2, entry_tid=10024' 2024-09-11T06:17:34.035 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=21, tag_tid=3, entry_tid=1' -a 'object_number=21, tag_tid=3, entry_tid=1' = 'object_number=8, tag_tid=2, entry_tid=10024' 2024-09-11T06:17:34.035 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=8, tag_tid=2, entry_tid=10024' '!=' 'object_number=11, tag_tid=2, entry_tid=9911' 2024-09-11T06:17:34.036 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:17:34.036 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=8, tag_tid=2, entry_tid=10024' = 'object_number=11, tag_tid=2, entry_tid=9911' 2024-09-11T06:17:34.036 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=8, tag_tid=2, entry_tid=10024' 2024-09-11T06:17:34.036 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:34.036 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:17:34.036 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:34.037 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=3 2024-09-11T06:17:34.037 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=8, tag_tid=2, entry_tid=10024' 2024-09-11T06:17:34.037 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:17:34.037 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:34.038 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=2 2024-09-11T06:17:34.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:34.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:17:34.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:34.040 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:17:34.041 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=8, tag_tid=2, entry_tid=10024' 2024-09-11T06:17:34.041 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:17:34.041 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:34.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=10024 2024-09-11T06:17:34.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 3 = 2 -a 1 -le 10024 2024-09-11T06:17:34.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:17:34.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:17:34.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:17:34.244 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:17:34.244 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:17:34.244 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:17:34.244 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:17:34.244 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:17:34.244 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:17:34.244 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:17:34.244 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:17:34.244 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:17:34.244 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:17:34.245 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:17:34.245 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:17:34.245 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:17:34.245 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:17:34.245 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:17:34.245 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:17:34.245 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:17:34.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:17:34.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:17:34.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:17:34.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:17:34.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:17:34.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:17:34.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:17:34.250 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:17:34.250 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:17:34.250 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:17:34.250 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:17:34.250 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:17:34.250 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:17:34.250 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:17:34.250 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:17:34.250 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:17:34.250 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:17:34.250 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:17:34.250 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:17:34.250 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:17:34.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:17:34.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:17:34.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:17:34.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:17:34.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:17:34.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:17:34.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:17:34.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:17:34.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:17:34.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:17:34.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:17:34.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:17:34.275 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:34.275 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:34.276 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:34.276 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:17:34.407 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:17:34.408 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:17:34.408 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:34.408 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:34.409 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:34.409 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:17:34.409 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:34.409 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:34.409 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:34.409 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:17:34.409 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:17:34.409 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:17:34.409 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:17:34.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:34.411 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:17:34.411 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:34.465 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 2 2024-09-11T06:17:34.465 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:17:34.465 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:17:34.465 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:17:34.465 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=10, tag_tid=2, entry_tid=10142], [object_number=9, tag_tid=2, entry_tid=10141], [object_number=8, tag_tid=2, entry_tid=10140], [object_number=11, tag_tid=2, entry_tid=10139]]], state=connected] 2024-09-11T06:17:34.468 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:34.473 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:34.473 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:17:34.473 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:34.473 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:34.473 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:34.473 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:17:34.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:34.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:34.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:34.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:17:34.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:17:34.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:17:34.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:17:34.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:34.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:17:34.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:34.518 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 2 2024-09-11T06:17:34.518 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:17:34.518 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:17:34.518 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:17:34.518 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=10, tag_tid=2, entry_tid=10142], [object_number=9, tag_tid=2, entry_tid=10141], [object_number=8, tag_tid=2, entry_tid=10140], [object_number=11, tag_tid=2, entry_tid=10139]]], state=connected] 2024-09-11T06:17:34.521 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:34.523 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=10, tag_tid=2, entry_tid=10142' 2024-09-11T06:17:34.523 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=21, tag_tid=3, entry_tid=1' -a 'object_number=21, tag_tid=3, entry_tid=1' = 'object_number=10, tag_tid=2, entry_tid=10142' 2024-09-11T06:17:34.523 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=10, tag_tid=2, entry_tid=10142' '!=' 'object_number=8, tag_tid=2, entry_tid=10024' 2024-09-11T06:17:34.523 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:17:34.524 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=10, tag_tid=2, entry_tid=10142' = 'object_number=8, tag_tid=2, entry_tid=10024' 2024-09-11T06:17:34.524 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=10, tag_tid=2, entry_tid=10142' 2024-09-11T06:17:34.524 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:34.524 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:17:34.524 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:34.525 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=3 2024-09-11T06:17:34.525 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=10, tag_tid=2, entry_tid=10142' 2024-09-11T06:17:34.526 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:17:34.526 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:34.526 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=2 2024-09-11T06:17:34.527 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:34.527 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:17:34.527 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:34.528 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:17:34.529 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=10, tag_tid=2, entry_tid=10142' 2024-09-11T06:17:34.529 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:17:34.529 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:34.530 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=10142 2024-09-11T06:17:34.530 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 3 = 2 -a 1 -le 10142 2024-09-11T06:17:34.530 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:17:34.530 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:17:34.531 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:17:34.732 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:17:34.732 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:17:34.732 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:17:34.732 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:17:34.732 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:17:34.732 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:17:34.732 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:17:34.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:17:34.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:17:34.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:17:34.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:17:34.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:17:34.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:17:34.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:17:34.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:17:34.734 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:17:34.734 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:17:34.735 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:17:34.735 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:17:34.735 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:17:34.735 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:17:34.735 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:17:34.736 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:17:34.736 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:17:34.736 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:17:34.736 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:17:34.736 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:17:34.736 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:17:34.736 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:17:34.736 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:17:34.736 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:17:34.736 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:17:34.736 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:17:34.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:17:34.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:17:34.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:17:34.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:17:34.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:17:34.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:17:34.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:17:34.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:17:34.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:17:34.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:17:34.739 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:17:34.739 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:17:34.739 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:17:34.739 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:17:34.739 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:17:34.739 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:17:34.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:34.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:34.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:34.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:17:34.890 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:17:34.891 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:17:34.891 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:34.891 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:34.891 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:34.892 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:17:34.892 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:34.892 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:34.892 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:34.892 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:17:34.892 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:17:34.892 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:17:34.892 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:17:34.893 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:34.893 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:17:34.893 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:34.944 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 2 2024-09-11T06:17:34.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:17:34.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:17:34.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:17:34.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=10265], [object_number=8, tag_tid=2, entry_tid=10264], [object_number=11, tag_tid=2, entry_tid=10263], [object_number=10, tag_tid=2, entry_tid=10262]]], state=connected] 2024-09-11T06:17:34.948 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:34.949 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:34.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:17:34.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:34.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:34.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:34.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:17:34.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:34.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:34.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:34.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:17:34.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:17:34.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:17:34.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:17:34.952 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:34.952 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:17:34.952 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:34.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 2 2024-09-11T06:17:34.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:17:34.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:17:34.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:17:34.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=10265], [object_number=8, tag_tid=2, entry_tid=10264], [object_number=11, tag_tid=2, entry_tid=10263], [object_number=10, tag_tid=2, entry_tid=10262]]], state=connected] 2024-09-11T06:17:35.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:35.003 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=10265' 2024-09-11T06:17:35.003 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=21, tag_tid=3, entry_tid=1' -a 'object_number=21, tag_tid=3, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=10265' 2024-09-11T06:17:35.003 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=9, tag_tid=2, entry_tid=10265' '!=' 'object_number=10, tag_tid=2, entry_tid=10142' 2024-09-11T06:17:35.003 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:17:35.003 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=9, tag_tid=2, entry_tid=10265' = 'object_number=10, tag_tid=2, entry_tid=10142' 2024-09-11T06:17:35.003 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=9, tag_tid=2, entry_tid=10265' 2024-09-11T06:17:35.004 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:35.004 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:17:35.004 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:35.005 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=3 2024-09-11T06:17:35.005 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=10265' 2024-09-11T06:17:35.005 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:17:35.005 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:35.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=2 2024-09-11T06:17:35.007 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:35.007 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:17:35.007 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:35.008 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:17:35.009 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=10265' 2024-09-11T06:17:35.009 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:17:35.009 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:35.010 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=10265 2024-09-11T06:17:35.010 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 3 = 2 -a 1 -le 10265 2024-09-11T06:17:35.010 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:17:35.010 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:17:35.010 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:17:35.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:17:35.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:17:35.213 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:17:35.213 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:17:35.213 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:17:35.213 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:17:35.213 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:17:35.213 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:17:35.213 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:17:35.213 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:17:35.213 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:17:35.213 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:17:35.213 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:17:35.214 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:17:35.214 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:17:35.216 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:17:35.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:17:35.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:17:35.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:17:35.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:17:35.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:17:35.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:17:35.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:17:35.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:17:35.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:17:35.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:17:35.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:17:35.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:17:35.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:17:35.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:17:35.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:17:35.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:17:35.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:17:35.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:17:35.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:17:35.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:17:35.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:17:35.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:17:35.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:17:35.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:17:35.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:17:35.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:17:35.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:17:35.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:17:35.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:17:35.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:17:35.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:17:35.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:17:35.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:17:35.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:35.244 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:35.244 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:35.244 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:17:35.372 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:17:35.374 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:17:35.374 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:35.374 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:35.374 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:35.374 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:17:35.374 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:35.374 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:35.374 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:35.374 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:17:35.375 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:17:35.375 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:17:35.375 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:17:35.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:35.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:17:35.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:35.427 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 2 2024-09-11T06:17:35.427 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:17:35.428 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:17:35.428 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:17:35.428 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=11, tag_tid=2, entry_tid=10391], [object_number=10, tag_tid=2, entry_tid=10390], [object_number=9, tag_tid=2, entry_tid=10389], [object_number=8, tag_tid=2, entry_tid=10388]]], state=connected] 2024-09-11T06:17:35.431 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:35.433 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:35.433 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:17:35.433 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:35.433 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:35.433 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:35.433 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:17:35.433 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:35.433 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:35.434 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:35.434 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:17:35.434 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:17:35.434 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:17:35.434 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:17:35.434 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:35.435 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:17:35.435 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:35.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 2 2024-09-11T06:17:35.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:17:35.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:17:35.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:17:35.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=11, tag_tid=2, entry_tid=10391], [object_number=10, tag_tid=2, entry_tid=10390], [object_number=9, tag_tid=2, entry_tid=10389], [object_number=8, tag_tid=2, entry_tid=10388]]], state=connected] 2024-09-11T06:17:35.483 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:35.485 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=11, tag_tid=2, entry_tid=10391' 2024-09-11T06:17:35.486 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=21, tag_tid=3, entry_tid=1' -a 'object_number=21, tag_tid=3, entry_tid=1' = 'object_number=11, tag_tid=2, entry_tid=10391' 2024-09-11T06:17:35.486 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=11, tag_tid=2, entry_tid=10391' '!=' 'object_number=9, tag_tid=2, entry_tid=10265' 2024-09-11T06:17:35.486 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:17:35.486 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=11, tag_tid=2, entry_tid=10391' = 'object_number=9, tag_tid=2, entry_tid=10265' 2024-09-11T06:17:35.486 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=11, tag_tid=2, entry_tid=10391' 2024-09-11T06:17:35.486 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:35.486 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:17:35.486 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:35.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=3 2024-09-11T06:17:35.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=11, tag_tid=2, entry_tid=10391' 2024-09-11T06:17:35.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:17:35.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:35.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=2 2024-09-11T06:17:35.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:35.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:17:35.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:35.490 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:17:35.491 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=11, tag_tid=2, entry_tid=10391' 2024-09-11T06:17:35.491 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:17:35.491 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:35.492 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=10391 2024-09-11T06:17:35.492 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 3 = 2 -a 1 -le 10391 2024-09-11T06:17:35.492 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:17:35.492 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:17:35.492 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:17:35.694 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:17:35.694 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:17:35.694 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:17:35.694 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:17:35.694 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:17:35.694 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:17:35.694 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:17:35.695 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:17:35.695 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:17:35.695 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:17:35.695 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:17:35.695 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:17:35.695 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:17:35.695 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:17:35.695 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:17:35.695 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:17:35.696 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:17:35.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:17:35.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:17:35.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:17:35.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:17:35.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:17:35.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:17:35.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:17:35.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:17:35.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:17:35.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:17:35.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:17:35.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:17:35.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:17:35.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:17:35.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:17:35.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:17:35.700 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:17:35.700 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:17:35.700 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:17:35.700 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:17:35.700 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:17:35.700 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:17:35.700 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:17:35.700 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:17:35.700 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:17:35.700 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:17:35.700 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:17:35.700 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:17:35.700 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:17:35.700 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:17:35.700 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:17:35.701 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:17:35.724 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:35.725 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:35.725 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:35.725 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:17:35.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:17:35.854 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:17:35.854 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:35.854 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:35.854 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:35.854 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:17:35.855 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:35.855 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:35.855 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:35.855 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:17:35.855 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:17:35.855 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:17:35.855 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:17:35.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:35.857 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:17:35.857 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:35.902 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 2 2024-09-11T06:17:35.902 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:17:35.902 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:17:35.902 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:17:35.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=10, tag_tid=2, entry_tid=10514], [object_number=9, tag_tid=2, entry_tid=10513], [object_number=8, tag_tid=2, entry_tid=10512], [object_number=11, tag_tid=2, entry_tid=10511]]], state=connected] 2024-09-11T06:17:35.909 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:35.909 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:35.909 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:17:35.909 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:35.909 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:35.909 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:35.910 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:17:35.910 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:35.910 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:35.910 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:35.910 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:17:35.910 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:17:35.910 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:17:35.910 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:17:35.911 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:35.911 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:17:35.911 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:35.954 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 2 2024-09-11T06:17:35.954 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:17:35.954 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:17:35.954 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:17:35.954 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=10, tag_tid=2, entry_tid=10514], [object_number=9, tag_tid=2, entry_tid=10513], [object_number=8, tag_tid=2, entry_tid=10512], [object_number=11, tag_tid=2, entry_tid=10511]]], state=connected] 2024-09-11T06:17:35.957 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:35.959 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=10, tag_tid=2, entry_tid=10514' 2024-09-11T06:17:35.960 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=21, tag_tid=3, entry_tid=1' -a 'object_number=21, tag_tid=3, entry_tid=1' = 'object_number=10, tag_tid=2, entry_tid=10514' 2024-09-11T06:17:35.960 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=10, tag_tid=2, entry_tid=10514' '!=' 'object_number=11, tag_tid=2, entry_tid=10391' 2024-09-11T06:17:35.960 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:17:35.960 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=10, tag_tid=2, entry_tid=10514' = 'object_number=11, tag_tid=2, entry_tid=10391' 2024-09-11T06:17:35.960 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=10, tag_tid=2, entry_tid=10514' 2024-09-11T06:17:35.960 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:35.960 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:17:35.960 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:35.960 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=3 2024-09-11T06:17:35.961 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=10, tag_tid=2, entry_tid=10514' 2024-09-11T06:17:35.961 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:17:35.961 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:35.962 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=2 2024-09-11T06:17:35.963 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:35.963 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:17:35.963 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:35.964 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:17:35.965 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=10, tag_tid=2, entry_tid=10514' 2024-09-11T06:17:35.965 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:17:35.965 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:35.966 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=10514 2024-09-11T06:17:35.966 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 3 = 2 -a 1 -le 10514 2024-09-11T06:17:35.966 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:17:35.966 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:17:35.966 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:17:36.168 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:17:36.168 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:17:36.168 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:17:36.169 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:17:36.169 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:17:36.169 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:17:36.169 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:17:36.169 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:17:36.169 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:17:36.169 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:17:36.169 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:17:36.169 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:17:36.169 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:17:36.169 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:17:36.169 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:17:36.170 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:17:36.170 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:17:36.174 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:17:36.174 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:17:36.174 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:17:36.174 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:17:36.174 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:17:36.174 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:17:36.174 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:17:36.175 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:17:36.175 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:17:36.175 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:17:36.175 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:17:36.175 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:17:36.175 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:17:36.175 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:17:36.175 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:17:36.175 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:17:36.175 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:17:36.175 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:17:36.175 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:17:36.175 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:17:36.175 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:17:36.176 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:17:36.176 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:17:36.176 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:17:36.176 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:17:36.176 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:17:36.176 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:17:36.176 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:17:36.176 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:17:36.176 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:17:36.176 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:17:36.176 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:17:36.199 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:36.199 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:36.199 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:36.199 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:17:36.386 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:17:36.387 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:17:36.387 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:36.387 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:36.387 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:36.387 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:17:36.387 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:36.387 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:36.387 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:36.388 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:17:36.388 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:17:36.388 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:17:36.388 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:17:36.389 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:36.390 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:17:36.390 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:36.447 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 2 2024-09-11T06:17:36.447 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:17:36.447 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:17:36.447 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:17:36.447 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=10557], [object_number=12, tag_tid=2, entry_tid=10556], [object_number=11, tag_tid=2, entry_tid=10555], [object_number=10, tag_tid=2, entry_tid=10554]]], state=connected] 2024-09-11T06:17:36.450 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:36.451 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:36.452 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:17:36.452 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:36.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:36.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:36.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:17:36.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:36.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:36.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:36.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:17:36.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:17:36.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:17:36.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:17:36.454 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:36.454 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:17:36.454 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:36.499 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 2 2024-09-11T06:17:36.499 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:17:36.499 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:17:36.499 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:17:36.499 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=10557], [object_number=12, tag_tid=2, entry_tid=10556], [object_number=11, tag_tid=2, entry_tid=10555], [object_number=10, tag_tid=2, entry_tid=10554]]], state=connected] 2024-09-11T06:17:36.502 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:36.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=13, tag_tid=2, entry_tid=10557' 2024-09-11T06:17:36.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=21, tag_tid=3, entry_tid=1' -a 'object_number=21, tag_tid=3, entry_tid=1' = 'object_number=13, tag_tid=2, entry_tid=10557' 2024-09-11T06:17:36.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=13, tag_tid=2, entry_tid=10557' '!=' 'object_number=10, tag_tid=2, entry_tid=10514' 2024-09-11T06:17:36.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:17:36.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=13, tag_tid=2, entry_tid=10557' = 'object_number=10, tag_tid=2, entry_tid=10514' 2024-09-11T06:17:36.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=13, tag_tid=2, entry_tid=10557' 2024-09-11T06:17:36.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:36.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:17:36.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:36.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=3 2024-09-11T06:17:36.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=13, tag_tid=2, entry_tid=10557' 2024-09-11T06:17:36.507 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:17:36.507 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:36.507 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=2 2024-09-11T06:17:36.508 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:36.508 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:17:36.508 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:36.509 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:17:36.510 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=13, tag_tid=2, entry_tid=10557' 2024-09-11T06:17:36.510 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:17:36.510 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:36.511 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=10557 2024-09-11T06:17:36.511 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 3 = 2 -a 1 -le 10557 2024-09-11T06:17:36.511 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:17:36.511 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:17:36.511 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:17:36.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:17:36.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:17:36.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:17:36.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:17:36.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:17:36.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:17:36.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:17:36.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:17:36.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:17:36.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:17:36.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:17:36.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:17:36.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:17:36.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:17:36.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:17:36.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:17:36.716 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:17:36.716 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:17:36.716 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:17:36.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:17:36.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:17:36.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:17:36.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:17:36.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:17:36.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:17:36.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:17:36.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:17:36.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:17:36.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:17:36.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:17:36.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:17:36.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:17:36.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:17:36.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:17:36.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:17:36.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:17:36.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:17:36.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:17:36.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:17:36.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:17:36.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:17:36.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:17:36.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:17:36.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:17:36.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:17:36.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:17:36.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:17:36.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:17:36.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:17:36.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:36.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:36.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:36.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:17:36.897 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:17:36.898 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:17:36.898 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:36.899 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:36.899 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:36.899 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:17:36.899 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:36.899 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:36.899 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:36.899 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:17:36.899 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:17:36.900 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:17:36.900 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:17:36.901 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:36.901 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:17:36.901 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:36.948 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 3 2024-09-11T06:17:36.948 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:17:36.949 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:17:36.949 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:17:36.949 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=12, tag_tid=2, entry_tid=10648], [object_number=15, tag_tid=2, entry_tid=10647], [object_number=14, tag_tid=2, entry_tid=10646], [object_number=13, tag_tid=2, entry_tid=10645]]], state=connected] 2024-09-11T06:17:36.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:36.953 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:36.954 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:17:36.954 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:36.954 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:36.954 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:36.954 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:17:36.954 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:36.954 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:36.955 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:36.955 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:17:36.955 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:17:36.955 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:17:36.955 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:17:36.955 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:36.955 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:17:36.956 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:37.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 3 2024-09-11T06:17:37.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:17:37.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:17:37.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:17:37.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=12, tag_tid=2, entry_tid=10648], [object_number=15, tag_tid=2, entry_tid=10647], [object_number=14, tag_tid=2, entry_tid=10646], [object_number=13, tag_tid=2, entry_tid=10645]]], state=connected] 2024-09-11T06:17:37.004 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:37.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=12, tag_tid=2, entry_tid=10648' 2024-09-11T06:17:37.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=21, tag_tid=3, entry_tid=1' -a 'object_number=21, tag_tid=3, entry_tid=1' = 'object_number=12, tag_tid=2, entry_tid=10648' 2024-09-11T06:17:37.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=12, tag_tid=2, entry_tid=10648' '!=' 'object_number=13, tag_tid=2, entry_tid=10557' 2024-09-11T06:17:37.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:17:37.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=12, tag_tid=2, entry_tid=10648' = 'object_number=13, tag_tid=2, entry_tid=10557' 2024-09-11T06:17:37.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=12, tag_tid=2, entry_tid=10648' 2024-09-11T06:17:37.007 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:37.007 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:17:37.007 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:37.007 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=3 2024-09-11T06:17:37.008 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=12, tag_tid=2, entry_tid=10648' 2024-09-11T06:17:37.008 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:17:37.008 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:37.009 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=2 2024-09-11T06:17:37.009 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:37.010 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:17:37.010 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:37.010 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:17:37.011 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=12, tag_tid=2, entry_tid=10648' 2024-09-11T06:17:37.011 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:17:37.011 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:37.013 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=10648 2024-09-11T06:17:37.013 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 3 = 2 -a 1 -le 10648 2024-09-11T06:17:37.013 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:17:37.013 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:17:37.013 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:17:37.214 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:17:37.214 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:17:37.214 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:17:37.215 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:17:37.215 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:17:37.215 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:17:37.215 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:17:37.215 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:17:37.215 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:17:37.215 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:17:37.215 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:17:37.215 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:17:37.215 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:17:37.215 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:17:37.215 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:17:37.216 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:17:37.216 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:17:37.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:17:37.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:17:37.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:17:37.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:17:37.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:17:37.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:17:37.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:17:37.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:17:37.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:17:37.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:17:37.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:17:37.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:17:37.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:17:37.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:17:37.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:17:37.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:17:37.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:17:37.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:17:37.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:17:37.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:17:37.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:17:37.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:17:37.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:17:37.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:17:37.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:17:37.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:17:37.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:17:37.221 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:17:37.221 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:17:37.221 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:17:37.221 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:17:37.221 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:17:37.245 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:37.245 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:37.245 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:37.245 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:17:37.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:17:37.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:17:37.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:37.378 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:37.378 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:37.378 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:17:37.378 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:37.378 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:37.378 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:37.378 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:17:37.379 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:17:37.379 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:17:37.379 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:17:37.380 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:37.381 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:17:37.381 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:37.433 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 3 2024-09-11T06:17:37.433 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:17:37.433 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:17:37.433 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:17:37.433 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=10761], [object_number=12, tag_tid=2, entry_tid=10760], [object_number=15, tag_tid=2, entry_tid=10759], [object_number=14, tag_tid=2, entry_tid=10758]]], state=connected] 2024-09-11T06:17:37.436 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:37.438 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:37.439 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:17:37.439 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:37.439 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:37.439 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:37.440 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:17:37.440 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:37.440 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:37.440 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:37.440 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:17:37.440 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:17:37.440 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:17:37.440 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:17:37.441 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:37.441 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:17:37.441 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:37.486 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 3 2024-09-11T06:17:37.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:17:37.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:17:37.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:17:37.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=10761], [object_number=12, tag_tid=2, entry_tid=10760], [object_number=15, tag_tid=2, entry_tid=10759], [object_number=14, tag_tid=2, entry_tid=10758]]], state=connected] 2024-09-11T06:17:37.490 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:37.492 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=13, tag_tid=2, entry_tid=10761' 2024-09-11T06:17:37.492 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=21, tag_tid=3, entry_tid=1' -a 'object_number=21, tag_tid=3, entry_tid=1' = 'object_number=13, tag_tid=2, entry_tid=10761' 2024-09-11T06:17:37.492 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=13, tag_tid=2, entry_tid=10761' '!=' 'object_number=12, tag_tid=2, entry_tid=10648' 2024-09-11T06:17:37.492 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:17:37.492 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=13, tag_tid=2, entry_tid=10761' = 'object_number=12, tag_tid=2, entry_tid=10648' 2024-09-11T06:17:37.492 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=13, tag_tid=2, entry_tid=10761' 2024-09-11T06:17:37.492 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:37.492 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:17:37.492 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:37.493 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=3 2024-09-11T06:17:37.493 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=13, tag_tid=2, entry_tid=10761' 2024-09-11T06:17:37.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:17:37.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:37.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=2 2024-09-11T06:17:37.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:37.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:17:37.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:37.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:17:37.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=13, tag_tid=2, entry_tid=10761' 2024-09-11T06:17:37.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:17:37.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:37.498 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=10761 2024-09-11T06:17:37.498 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 3 = 2 -a 1 -le 10761 2024-09-11T06:17:37.498 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:17:37.498 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:17:37.498 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:17:37.700 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:17:37.700 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:17:37.701 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:17:37.701 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:17:37.701 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:17:37.701 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:17:37.701 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:17:37.701 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:17:37.701 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:17:37.701 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:17:37.701 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:17:37.701 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:17:37.701 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:17:37.701 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:17:37.701 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:17:37.702 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:17:37.702 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:17:37.706 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:17:37.706 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:17:37.706 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:17:37.706 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:17:37.706 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:17:37.706 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:17:37.706 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:17:37.706 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:17:37.706 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:17:37.706 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:17:37.707 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:17:37.707 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:17:37.707 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:17:37.707 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:17:37.707 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:17:37.707 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:17:37.707 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:17:37.707 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:17:37.707 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:17:37.707 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:17:37.707 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:17:37.707 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:17:37.707 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:17:37.707 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:17:37.708 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:17:37.708 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:17:37.708 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:17:37.708 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:17:37.708 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:17:37.708 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:17:37.708 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:17:37.708 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:17:37.732 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:37.732 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:37.732 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:37.732 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:17:37.861 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:17:37.862 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:17:37.862 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:37.862 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:37.862 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:37.862 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:17:37.862 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:37.862 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:37.862 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:37.862 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:17:37.863 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:17:37.863 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:17:37.863 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:17:37.864 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:37.864 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:17:37.865 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:37.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 3 2024-09-11T06:17:37.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:17:37.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:17:37.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:17:37.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=12, tag_tid=2, entry_tid=10872], [object_number=15, tag_tid=2, entry_tid=10871], [object_number=14, tag_tid=2, entry_tid=10870], [object_number=13, tag_tid=2, entry_tid=10869]]], state=connected] 2024-09-11T06:17:37.923 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:37.925 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:37.926 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:17:37.926 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:37.926 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:37.926 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:37.926 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:17:37.926 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:37.926 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:37.926 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:37.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:17:37.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:17:37.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:17:37.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:17:37.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:37.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:17:37.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:37.973 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 3 2024-09-11T06:17:37.973 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:17:37.973 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:17:37.973 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:17:37.973 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=12, tag_tid=2, entry_tid=10872], [object_number=15, tag_tid=2, entry_tid=10871], [object_number=14, tag_tid=2, entry_tid=10870], [object_number=13, tag_tid=2, entry_tid=10869]]], state=connected] 2024-09-11T06:17:37.976 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:37.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=12, tag_tid=2, entry_tid=10872' 2024-09-11T06:17:37.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=21, tag_tid=3, entry_tid=1' -a 'object_number=21, tag_tid=3, entry_tid=1' = 'object_number=12, tag_tid=2, entry_tid=10872' 2024-09-11T06:17:37.979 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=12, tag_tid=2, entry_tid=10872' '!=' 'object_number=13, tag_tid=2, entry_tid=10761' 2024-09-11T06:17:37.979 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:17:37.979 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=12, tag_tid=2, entry_tid=10872' = 'object_number=13, tag_tid=2, entry_tid=10761' 2024-09-11T06:17:37.979 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=12, tag_tid=2, entry_tid=10872' 2024-09-11T06:17:37.979 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:37.979 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:17:37.979 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:37.980 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=3 2024-09-11T06:17:37.980 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=12, tag_tid=2, entry_tid=10872' 2024-09-11T06:17:37.980 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:17:37.981 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:37.981 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=2 2024-09-11T06:17:37.982 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:37.982 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:17:37.982 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:37.983 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:17:37.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=12, tag_tid=2, entry_tid=10872' 2024-09-11T06:17:37.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:17:37.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:37.985 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=10872 2024-09-11T06:17:37.985 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 3 = 2 -a 1 -le 10872 2024-09-11T06:17:37.985 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:17:37.985 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:17:37.986 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:17:38.187 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:17:38.187 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:17:38.187 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:17:38.187 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:17:38.187 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:17:38.187 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:17:38.188 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:17:38.188 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:17:38.188 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:17:38.188 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:17:38.188 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:17:38.188 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:17:38.188 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:17:38.188 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:17:38.188 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:17:38.188 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:17:38.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:17:38.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:17:38.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:17:38.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:17:38.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:17:38.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:17:38.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:17:38.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:17:38.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:17:38.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:17:38.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:17:38.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:17:38.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:17:38.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:17:38.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:17:38.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:17:38.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:17:38.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:17:38.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:17:38.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:17:38.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:17:38.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:17:38.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:17:38.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:17:38.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:17:38.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:17:38.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:17:38.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:17:38.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:17:38.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:17:38.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:17:38.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:17:38.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:17:38.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:38.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:38.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:38.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:17:38.348 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:17:38.350 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:17:38.350 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:38.350 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:38.350 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:38.350 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:17:38.350 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:38.350 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:38.350 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:38.350 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:17:38.350 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:17:38.351 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:17:38.351 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:17:38.351 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:38.352 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:17:38.352 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:38.397 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 3 2024-09-11T06:17:38.397 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:17:38.397 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:17:38.397 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:17:38.397 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=15, tag_tid=2, entry_tid=10991], [object_number=14, tag_tid=2, entry_tid=10990], [object_number=13, tag_tid=2, entry_tid=10989], [object_number=12, tag_tid=2, entry_tid=10988]]], state=connected] 2024-09-11T06:17:38.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:38.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:38.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:17:38.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:38.403 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:38.403 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:38.403 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:17:38.403 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:38.403 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:38.403 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:38.403 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:17:38.403 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:17:38.403 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:17:38.403 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:17:38.404 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:38.404 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:17:38.404 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:38.450 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 3 2024-09-11T06:17:38.451 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:17:38.451 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:17:38.451 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:17:38.451 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=15, tag_tid=2, entry_tid=10991], [object_number=14, tag_tid=2, entry_tid=10990], [object_number=13, tag_tid=2, entry_tid=10989], [object_number=12, tag_tid=2, entry_tid=10988]]], state=connected] 2024-09-11T06:17:38.454 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:38.456 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=15, tag_tid=2, entry_tid=10991' 2024-09-11T06:17:38.456 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=21, tag_tid=3, entry_tid=1' -a 'object_number=21, tag_tid=3, entry_tid=1' = 'object_number=15, tag_tid=2, entry_tid=10991' 2024-09-11T06:17:38.456 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=15, tag_tid=2, entry_tid=10991' '!=' 'object_number=12, tag_tid=2, entry_tid=10872' 2024-09-11T06:17:38.456 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:17:38.456 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=15, tag_tid=2, entry_tid=10991' = 'object_number=12, tag_tid=2, entry_tid=10872' 2024-09-11T06:17:38.456 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=15, tag_tid=2, entry_tid=10991' 2024-09-11T06:17:38.457 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:38.457 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:17:38.457 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:38.457 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=3 2024-09-11T06:17:38.458 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=15, tag_tid=2, entry_tid=10991' 2024-09-11T06:17:38.458 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:17:38.458 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:38.459 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=2 2024-09-11T06:17:38.459 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:38.460 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:17:38.460 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:38.460 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:17:38.461 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=15, tag_tid=2, entry_tid=10991' 2024-09-11T06:17:38.461 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:17:38.461 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:38.463 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=10991 2024-09-11T06:17:38.463 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 3 = 2 -a 1 -le 10991 2024-09-11T06:17:38.463 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:17:38.463 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:17:38.463 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:17:38.665 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:17:38.665 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:17:38.665 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:17:38.665 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:17:38.665 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:17:38.665 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:17:38.665 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:17:38.665 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:17:38.665 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:17:38.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:17:38.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:17:38.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:17:38.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:17:38.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:17:38.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:17:38.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:17:38.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:17:38.669 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:17:38.669 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:17:38.669 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:17:38.669 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:17:38.670 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:17:38.670 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:17:38.670 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:17:38.670 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:17:38.670 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:17:38.670 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:17:38.670 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:17:38.670 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:17:38.670 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:17:38.670 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:17:38.670 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:17:38.670 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:17:38.670 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:17:38.670 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:17:38.671 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:17:38.671 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:17:38.671 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:17:38.671 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:17:38.671 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:17:38.671 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:17:38.671 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:17:38.671 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:17:38.671 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:17:38.672 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:17:38.672 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:17:38.672 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:17:38.672 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:17:38.672 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:17:38.695 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:38.695 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:38.696 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:38.696 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:17:38.823 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:17:38.824 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:17:38.824 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:38.824 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:38.824 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:38.825 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:17:38.825 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:38.825 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:38.825 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:38.825 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:17:38.826 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:17:38.826 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:17:38.826 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:17:38.828 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:38.829 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:17:38.829 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:38.878 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 3 2024-09-11T06:17:38.878 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:17:38.878 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:17:38.878 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:17:38.878 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=11109], [object_number=12, tag_tid=2, entry_tid=11108], [object_number=15, tag_tid=2, entry_tid=11107], [object_number=14, tag_tid=2, entry_tid=11106]]], state=connected] 2024-09-11T06:17:38.881 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:38.883 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:38.884 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:17:38.884 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:38.884 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:38.884 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:38.885 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:17:38.885 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:38.885 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:38.885 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:38.885 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:17:38.885 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:17:38.885 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:17:38.885 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:17:38.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:38.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:17:38.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:38.930 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 3 2024-09-11T06:17:38.930 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:17:38.930 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:17:38.931 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:17:38.931 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=11109], [object_number=12, tag_tid=2, entry_tid=11108], [object_number=15, tag_tid=2, entry_tid=11107], [object_number=14, tag_tid=2, entry_tid=11106]]], state=connected] 2024-09-11T06:17:38.934 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:38.935 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=13, tag_tid=2, entry_tid=11109' 2024-09-11T06:17:38.935 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=21, tag_tid=3, entry_tid=1' -a 'object_number=21, tag_tid=3, entry_tid=1' = 'object_number=13, tag_tid=2, entry_tid=11109' 2024-09-11T06:17:38.936 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=13, tag_tid=2, entry_tid=11109' '!=' 'object_number=15, tag_tid=2, entry_tid=10991' 2024-09-11T06:17:38.936 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:17:38.936 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=13, tag_tid=2, entry_tid=11109' = 'object_number=15, tag_tid=2, entry_tid=10991' 2024-09-11T06:17:38.936 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=13, tag_tid=2, entry_tid=11109' 2024-09-11T06:17:38.936 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:38.936 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:17:38.937 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:38.937 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=3 2024-09-11T06:17:38.937 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=13, tag_tid=2, entry_tid=11109' 2024-09-11T06:17:38.937 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:17:38.938 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:38.938 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=2 2024-09-11T06:17:38.939 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:38.939 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:17:38.939 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:38.940 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:17:38.941 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=13, tag_tid=2, entry_tid=11109' 2024-09-11T06:17:38.941 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:17:38.941 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:38.942 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=11109 2024-09-11T06:17:38.942 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 3 = 2 -a 1 -le 11109 2024-09-11T06:17:38.942 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:17:38.942 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:17:38.942 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:17:39.145 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:17:39.145 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:17:39.145 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:17:39.145 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:17:39.146 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:17:39.146 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:17:39.146 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:17:39.146 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:17:39.146 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:17:39.146 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:17:39.147 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:17:39.147 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:17:39.147 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:17:39.147 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:17:39.147 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:17:39.147 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:17:39.152 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:17:39.152 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:17:39.152 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:17:39.152 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:17:39.152 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:17:39.152 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:17:39.152 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:17:39.152 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:17:39.153 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:17:39.153 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:17:39.153 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:17:39.153 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:17:39.153 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:17:39.153 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:17:39.153 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:17:39.153 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:17:39.154 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:17:39.154 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:17:39.154 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:17:39.154 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:17:39.154 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:17:39.154 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:17:39.154 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:17:39.155 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:17:39.155 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:17:39.155 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:17:39.155 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:17:39.155 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:17:39.155 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:17:39.155 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:17:39.156 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:17:39.156 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:17:39.156 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:17:39.177 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:39.177 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:39.177 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:39.177 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:17:39.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:17:39.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:17:39.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:39.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:39.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:39.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:17:39.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:39.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:39.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:39.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:17:39.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:17:39.305 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:17:39.305 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:17:39.305 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:39.306 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:17:39.306 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:39.356 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 3 2024-09-11T06:17:39.357 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:17:39.357 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:17:39.357 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:17:39.357 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=11221], [object_number=12, tag_tid=2, entry_tid=11220], [object_number=15, tag_tid=2, entry_tid=11219], [object_number=14, tag_tid=2, entry_tid=11218]]], state=connected] 2024-09-11T06:17:39.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:39.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:39.362 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:17:39.362 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:39.363 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:39.363 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:39.363 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:17:39.363 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:39.363 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:39.363 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:39.363 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:17:39.363 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:17:39.363 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:17:39.363 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:17:39.364 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:39.364 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:17:39.364 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:39.409 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 3 2024-09-11T06:17:39.409 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:17:39.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:17:39.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:17:39.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=11221], [object_number=12, tag_tid=2, entry_tid=11220], [object_number=15, tag_tid=2, entry_tid=11219], [object_number=14, tag_tid=2, entry_tid=11218]]], state=connected] 2024-09-11T06:17:39.413 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:39.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=13, tag_tid=2, entry_tid=11221' 2024-09-11T06:17:39.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=21, tag_tid=3, entry_tid=1' -a 'object_number=21, tag_tid=3, entry_tid=1' = 'object_number=13, tag_tid=2, entry_tid=11221' 2024-09-11T06:17:39.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=13, tag_tid=2, entry_tid=11221' '!=' 'object_number=13, tag_tid=2, entry_tid=11109' 2024-09-11T06:17:39.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:17:39.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=13, tag_tid=2, entry_tid=11221' = 'object_number=13, tag_tid=2, entry_tid=11109' 2024-09-11T06:17:39.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=13, tag_tid=2, entry_tid=11221' 2024-09-11T06:17:39.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:39.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:17:39.416 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:39.416 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=3 2024-09-11T06:17:39.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=13, tag_tid=2, entry_tid=11221' 2024-09-11T06:17:39.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:17:39.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:39.418 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=2 2024-09-11T06:17:39.418 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:39.418 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:17:39.418 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:39.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:17:39.420 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=13, tag_tid=2, entry_tid=11221' 2024-09-11T06:17:39.420 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:17:39.420 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:39.421 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=11221 2024-09-11T06:17:39.421 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 3 = 2 -a 1 -le 11221 2024-09-11T06:17:39.421 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:17:39.422 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:17:39.422 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:17:39.623 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:17:39.623 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:17:39.623 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:17:39.624 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:17:39.624 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:17:39.624 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:17:39.624 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:17:39.624 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:17:39.624 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:17:39.624 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:17:39.624 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:17:39.624 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:17:39.624 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:17:39.624 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:17:39.624 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:17:39.625 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:17:39.625 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:17:39.626 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:17:39.627 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:17:39.627 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:17:39.627 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:17:39.627 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:17:39.627 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:17:39.627 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:17:39.627 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:17:39.627 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:17:39.627 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:17:39.627 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:17:39.627 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:17:39.627 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:17:39.628 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:17:39.628 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:17:39.628 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:17:39.629 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:17:39.629 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:17:39.629 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:17:39.629 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:17:39.629 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:17:39.629 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:17:39.629 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:17:39.629 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:17:39.629 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:17:39.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:17:39.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:17:39.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:17:39.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:17:39.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:17:39.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:17:39.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:17:39.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:39.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:39.654 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:39.654 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:17:39.787 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:17:39.788 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:17:39.788 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:39.788 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:39.789 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:39.789 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:17:39.789 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:39.789 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:39.789 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:39.789 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:17:39.789 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:17:39.789 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:17:39.789 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:17:39.790 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:39.790 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:17:39.790 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:39.837 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 3 2024-09-11T06:17:39.837 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:17:39.837 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:17:39.837 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:17:39.837 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=12, tag_tid=2, entry_tid=11316], [object_number=15, tag_tid=2, entry_tid=11315], [object_number=14, tag_tid=2, entry_tid=11314], [object_number=13, tag_tid=2, entry_tid=11313]]], state=connected] 2024-09-11T06:17:39.840 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:39.842 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:39.843 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:17:39.843 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:39.843 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:39.843 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:39.843 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:17:39.843 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:39.843 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:39.843 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:39.843 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:17:39.843 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:17:39.844 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:17:39.844 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:17:39.844 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:39.844 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:17:39.844 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:39.889 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 3 2024-09-11T06:17:39.889 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:17:39.889 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:17:39.890 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:17:39.890 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=12, tag_tid=2, entry_tid=11316], [object_number=15, tag_tid=2, entry_tid=11315], [object_number=14, tag_tid=2, entry_tid=11314], [object_number=13, tag_tid=2, entry_tid=11313]]], state=connected] 2024-09-11T06:17:39.892 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:39.894 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=12, tag_tid=2, entry_tid=11316' 2024-09-11T06:17:39.894 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=21, tag_tid=3, entry_tid=1' -a 'object_number=21, tag_tid=3, entry_tid=1' = 'object_number=12, tag_tid=2, entry_tid=11316' 2024-09-11T06:17:39.894 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=12, tag_tid=2, entry_tid=11316' '!=' 'object_number=13, tag_tid=2, entry_tid=11221' 2024-09-11T06:17:39.895 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:17:39.895 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=12, tag_tid=2, entry_tid=11316' = 'object_number=13, tag_tid=2, entry_tid=11221' 2024-09-11T06:17:39.895 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=12, tag_tid=2, entry_tid=11316' 2024-09-11T06:17:39.895 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:39.895 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:17:39.896 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:39.897 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=3 2024-09-11T06:17:39.897 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=12, tag_tid=2, entry_tid=11316' 2024-09-11T06:17:39.897 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:17:39.897 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:39.897 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=2 2024-09-11T06:17:39.898 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:39.898 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:17:39.898 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:39.899 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:17:39.900 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=12, tag_tid=2, entry_tid=11316' 2024-09-11T06:17:39.900 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:17:39.900 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:39.901 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=11316 2024-09-11T06:17:39.901 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 3 = 2 -a 1 -le 11316 2024-09-11T06:17:39.901 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:17:39.901 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:17:39.902 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:17:40.103 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:17:40.103 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:17:40.103 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:17:40.104 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:17:40.104 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:17:40.104 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:17:40.104 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:17:40.104 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:17:40.104 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:17:40.104 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:17:40.104 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:17:40.105 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:17:40.105 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:17:40.105 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:17:40.105 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:17:40.108 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:17:40.108 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:17:40.108 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:17:40.108 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:17:40.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:17:40.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:17:40.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:17:40.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:17:40.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:17:40.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:17:40.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:17:40.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:17:40.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:17:40.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:17:40.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:17:40.110 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:17:40.110 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:17:40.110 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:17:40.110 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:17:40.110 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:17:40.110 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:17:40.110 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:17:40.110 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:17:40.110 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:17:40.110 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:17:40.110 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:17:40.110 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:17:40.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:17:40.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:17:40.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:17:40.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:17:40.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:17:40.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:17:40.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:17:40.133 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:40.133 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:40.133 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:40.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:17:40.257 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:17:40.258 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:17:40.258 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:40.258 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:40.258 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:40.259 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:17:40.259 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:40.259 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:40.259 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:40.259 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:17:40.259 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:17:40.260 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:17:40.260 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:17:40.261 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:40.261 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:17:40.261 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:40.306 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 3 2024-09-11T06:17:40.306 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:17:40.306 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:17:40.306 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:17:40.306 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=14, tag_tid=2, entry_tid=11410], [object_number=13, tag_tid=2, entry_tid=11409], [object_number=12, tag_tid=2, entry_tid=11408], [object_number=15, tag_tid=2, entry_tid=11407]]], state=connected] 2024-09-11T06:17:40.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:40.310 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:40.312 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:17:40.312 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:40.312 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:40.312 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:40.312 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:17:40.312 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:40.312 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:40.312 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:40.312 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:17:40.312 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:17:40.313 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:17:40.313 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:17:40.314 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:40.314 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:17:40.314 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:40.358 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 3 2024-09-11T06:17:40.358 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:17:40.358 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:17:40.359 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:17:40.359 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=14, tag_tid=2, entry_tid=11410], [object_number=13, tag_tid=2, entry_tid=11409], [object_number=12, tag_tid=2, entry_tid=11408], [object_number=15, tag_tid=2, entry_tid=11407]]], state=connected] 2024-09-11T06:17:40.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:40.363 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=14, tag_tid=2, entry_tid=11410' 2024-09-11T06:17:40.363 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=21, tag_tid=3, entry_tid=1' -a 'object_number=21, tag_tid=3, entry_tid=1' = 'object_number=14, tag_tid=2, entry_tid=11410' 2024-09-11T06:17:40.363 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=14, tag_tid=2, entry_tid=11410' '!=' 'object_number=12, tag_tid=2, entry_tid=11316' 2024-09-11T06:17:40.363 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:17:40.363 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=14, tag_tid=2, entry_tid=11410' = 'object_number=12, tag_tid=2, entry_tid=11316' 2024-09-11T06:17:40.364 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=14, tag_tid=2, entry_tid=11410' 2024-09-11T06:17:40.364 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:40.364 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:17:40.364 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:40.365 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=3 2024-09-11T06:17:40.365 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=14, tag_tid=2, entry_tid=11410' 2024-09-11T06:17:40.366 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:17:40.366 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:40.367 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=2 2024-09-11T06:17:40.368 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:40.368 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:17:40.368 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:40.369 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:17:40.369 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=14, tag_tid=2, entry_tid=11410' 2024-09-11T06:17:40.369 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:17:40.369 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:40.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=11410 2024-09-11T06:17:40.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 3 = 2 -a 1 -le 11410 2024-09-11T06:17:40.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:17:40.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:17:40.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:17:40.573 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:17:40.573 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:17:40.573 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:17:40.573 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:17:40.573 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:17:40.573 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:17:40.573 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:17:40.573 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:17:40.574 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:17:40.574 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:17:40.574 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:17:40.574 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:17:40.574 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:17:40.574 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:17:40.574 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:17:40.574 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:17:40.574 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:17:40.577 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:17:40.577 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:17:40.577 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:17:40.577 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:17:40.577 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:17:40.577 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:17:40.577 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:17:40.577 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:17:40.578 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:17:40.578 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:17:40.578 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:17:40.578 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:17:40.578 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:17:40.578 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:17:40.578 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:17:40.578 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:17:40.578 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:17:40.578 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:17:40.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:17:40.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:17:40.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:17:40.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:17:40.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:17:40.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:17:40.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:17:40.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:17:40.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:17:40.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:17:40.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:17:40.580 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:17:40.580 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:17:40.580 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:17:40.604 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:40.605 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:40.605 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:40.605 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:17:40.740 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:17:40.741 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:17:40.741 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:40.741 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:40.741 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:40.741 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:17:40.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:40.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:40.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:40.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:17:40.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:17:40.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:17:40.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:17:40.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:40.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:17:40.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:40.789 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 3 2024-09-11T06:17:40.789 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:17:40.789 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:17:40.789 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:17:40.789 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=11501], [object_number=12, tag_tid=2, entry_tid=11500], [object_number=15, tag_tid=2, entry_tid=11499], [object_number=14, tag_tid=2, entry_tid=11498]]], state=connected] 2024-09-11T06:17:40.792 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:40.793 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:40.794 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:17:40.794 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:40.795 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:40.795 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:40.795 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:17:40.795 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:40.795 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:40.795 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:40.795 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:17:40.795 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:17:40.795 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:17:40.795 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:17:40.796 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:40.796 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:17:40.796 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:40.840 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 3 2024-09-11T06:17:40.841 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:17:40.841 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:17:40.841 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:17:40.841 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=11501], [object_number=12, tag_tid=2, entry_tid=11500], [object_number=15, tag_tid=2, entry_tid=11499], [object_number=14, tag_tid=2, entry_tid=11498]]], state=connected] 2024-09-11T06:17:40.844 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:40.845 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=13, tag_tid=2, entry_tid=11501' 2024-09-11T06:17:40.845 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=21, tag_tid=3, entry_tid=1' -a 'object_number=21, tag_tid=3, entry_tid=1' = 'object_number=13, tag_tid=2, entry_tid=11501' 2024-09-11T06:17:40.846 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=13, tag_tid=2, entry_tid=11501' '!=' 'object_number=14, tag_tid=2, entry_tid=11410' 2024-09-11T06:17:40.846 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:17:40.846 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=13, tag_tid=2, entry_tid=11501' = 'object_number=14, tag_tid=2, entry_tid=11410' 2024-09-11T06:17:40.846 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=13, tag_tid=2, entry_tid=11501' 2024-09-11T06:17:40.846 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:40.846 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:17:40.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:40.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=3 2024-09-11T06:17:40.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=13, tag_tid=2, entry_tid=11501' 2024-09-11T06:17:40.848 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:17:40.848 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:40.848 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=2 2024-09-11T06:17:40.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:40.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:17:40.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:40.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:17:40.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=13, tag_tid=2, entry_tid=11501' 2024-09-11T06:17:40.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:17:40.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:40.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=11501 2024-09-11T06:17:40.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 3 = 2 -a 1 -le 11501 2024-09-11T06:17:40.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:17:40.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:17:40.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:17:41.054 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:17:41.054 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:17:41.054 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:17:41.054 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:17:41.054 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:17:41.054 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:17:41.054 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:17:41.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:17:41.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:17:41.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:17:41.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:17:41.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:17:41.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:17:41.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:17:41.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:17:41.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:17:41.056 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:17:41.058 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:17:41.058 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:17:41.058 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:17:41.059 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:17:41.059 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:17:41.059 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:17:41.059 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:17:41.059 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:17:41.059 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:17:41.059 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:17:41.059 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:17:41.059 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:17:41.059 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:17:41.059 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:17:41.059 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:17:41.059 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:17:41.060 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:17:41.060 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:17:41.060 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:17:41.060 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:17:41.060 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:17:41.060 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:17:41.060 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:17:41.060 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:17:41.060 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:17:41.060 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:17:41.060 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:17:41.060 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:17:41.060 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:17:41.060 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:17:41.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:17:41.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:17:41.084 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:41.085 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:41.085 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:41.085 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:17:41.221 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:17:41.221 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:17:41.221 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:41.221 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:41.221 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:41.221 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:17:41.221 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:41.221 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:41.222 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:41.222 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:17:41.222 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:17:41.222 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:17:41.222 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:17:41.225 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:41.225 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:17:41.225 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:41.269 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 3 2024-09-11T06:17:41.269 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:17:41.269 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:17:41.270 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:17:41.270 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=15, tag_tid=2, entry_tid=11595], [object_number=14, tag_tid=2, entry_tid=11594], [object_number=13, tag_tid=2, entry_tid=11593], [object_number=12, tag_tid=2, entry_tid=11592]]], state=connected] 2024-09-11T06:17:41.273 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:41.274 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:41.275 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:17:41.275 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:41.275 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:41.276 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:41.276 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:17:41.276 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:41.276 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:41.276 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:41.276 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:17:41.276 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:17:41.276 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:17:41.276 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:17:41.277 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:41.277 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:17:41.277 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:41.322 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 3 2024-09-11T06:17:41.323 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:17:41.323 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:17:41.323 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:17:41.323 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=15, tag_tid=2, entry_tid=11595], [object_number=14, tag_tid=2, entry_tid=11594], [object_number=13, tag_tid=2, entry_tid=11593], [object_number=12, tag_tid=2, entry_tid=11592]]], state=connected] 2024-09-11T06:17:41.326 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:41.327 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=15, tag_tid=2, entry_tid=11595' 2024-09-11T06:17:41.327 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=21, tag_tid=3, entry_tid=1' -a 'object_number=21, tag_tid=3, entry_tid=1' = 'object_number=15, tag_tid=2, entry_tid=11595' 2024-09-11T06:17:41.327 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=15, tag_tid=2, entry_tid=11595' '!=' 'object_number=13, tag_tid=2, entry_tid=11501' 2024-09-11T06:17:41.328 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:17:41.328 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=15, tag_tid=2, entry_tid=11595' = 'object_number=13, tag_tid=2, entry_tid=11501' 2024-09-11T06:17:41.328 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=15, tag_tid=2, entry_tid=11595' 2024-09-11T06:17:41.328 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:41.328 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:17:41.328 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:41.329 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=3 2024-09-11T06:17:41.329 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=15, tag_tid=2, entry_tid=11595' 2024-09-11T06:17:41.330 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:17:41.330 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:41.330 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=2 2024-09-11T06:17:41.331 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:41.332 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:17:41.332 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:41.332 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:17:41.333 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=15, tag_tid=2, entry_tid=11595' 2024-09-11T06:17:41.333 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:17:41.333 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:41.334 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=11595 2024-09-11T06:17:41.334 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 3 = 2 -a 1 -le 11595 2024-09-11T06:17:41.334 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:17:41.334 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:17:41.334 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:17:41.536 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:17:41.536 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:17:41.537 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:17:41.537 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:17:41.537 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:17:41.537 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:17:41.537 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:17:41.537 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:17:41.537 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:17:41.537 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:17:41.537 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:17:41.537 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:17:41.537 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:17:41.537 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:17:41.537 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:17:41.538 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:17:41.538 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:17:41.541 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:17:41.541 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:17:41.541 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:17:41.541 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:17:41.541 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:17:41.541 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:17:41.541 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:17:41.541 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:17:41.541 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:17:41.541 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:17:41.541 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:17:41.541 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:17:41.541 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:17:41.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:17:41.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:17:41.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:17:41.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:17:41.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:17:41.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:17:41.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:17:41.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:17:41.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:17:41.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:17:41.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:17:41.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:17:41.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:17:41.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:17:41.543 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:17:41.543 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:17:41.543 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:17:41.543 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:17:41.543 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:17:41.566 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:41.566 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:41.566 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:41.566 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:17:41.695 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:17:41.696 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:17:41.696 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:41.696 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:41.696 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:41.696 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:17:41.696 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:41.696 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:41.696 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:41.696 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:17:41.697 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:17:41.697 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:17:41.697 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:17:41.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:41.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:17:41.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:41.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 3 2024-09-11T06:17:41.756 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:17:41.756 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:17:41.756 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:17:41.756 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=14, tag_tid=2, entry_tid=11686], [object_number=13, tag_tid=2, entry_tid=11685], [object_number=12, tag_tid=2, entry_tid=11684], [object_number=15, tag_tid=2, entry_tid=11683]]], state=connected] 2024-09-11T06:17:41.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:41.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:41.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:17:41.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:41.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:41.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:41.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:17:41.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:41.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:41.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:41.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:17:41.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:17:41.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:17:41.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:17:41.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:41.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:17:41.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:41.810 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 3 2024-09-11T06:17:41.810 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:17:41.810 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:17:41.810 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:17:41.810 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=14, tag_tid=2, entry_tid=11686], [object_number=13, tag_tid=2, entry_tid=11685], [object_number=12, tag_tid=2, entry_tid=11684], [object_number=15, tag_tid=2, entry_tid=11683]]], state=connected] 2024-09-11T06:17:41.813 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:41.816 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=14, tag_tid=2, entry_tid=11686' 2024-09-11T06:17:41.816 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=21, tag_tid=3, entry_tid=1' -a 'object_number=21, tag_tid=3, entry_tid=1' = 'object_number=14, tag_tid=2, entry_tid=11686' 2024-09-11T06:17:41.816 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=14, tag_tid=2, entry_tid=11686' '!=' 'object_number=15, tag_tid=2, entry_tid=11595' 2024-09-11T06:17:41.816 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:17:41.817 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=14, tag_tid=2, entry_tid=11686' = 'object_number=15, tag_tid=2, entry_tid=11595' 2024-09-11T06:17:41.817 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=14, tag_tid=2, entry_tid=11686' 2024-09-11T06:17:41.817 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:41.817 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:17:41.817 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:41.818 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=3 2024-09-11T06:17:41.818 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=14, tag_tid=2, entry_tid=11686' 2024-09-11T06:17:41.819 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:17:41.819 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:41.819 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=2 2024-09-11T06:17:41.820 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:41.820 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:17:41.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:41.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:17:41.822 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=14, tag_tid=2, entry_tid=11686' 2024-09-11T06:17:41.822 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:17:41.822 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:41.822 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=11686 2024-09-11T06:17:41.823 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 3 = 2 -a 1 -le 11686 2024-09-11T06:17:41.823 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:17:41.823 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:17:41.823 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:17:42.024 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:17:42.024 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:17:42.024 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:17:42.025 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:17:42.025 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:17:42.025 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:17:42.025 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:17:42.025 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:17:42.025 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:17:42.025 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:17:42.025 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:17:42.025 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:17:42.025 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:17:42.025 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:17:42.025 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:17:42.026 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:17:42.026 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:17:42.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:17:42.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:17:42.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:17:42.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:17:42.028 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:17:42.028 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:17:42.028 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:17:42.028 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:17:42.028 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:17:42.028 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:17:42.028 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:17:42.028 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:17:42.028 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:17:42.028 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:17:42.028 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:17:42.029 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:17:42.030 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:17:42.030 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:17:42.030 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:17:42.030 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:17:42.030 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:17:42.030 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:17:42.030 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:17:42.030 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:17:42.030 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:17:42.030 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:17:42.030 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:17:42.031 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:17:42.031 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:17:42.031 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:17:42.031 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:17:42.031 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:17:42.053 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:42.054 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:42.054 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:42.054 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:17:42.182 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:17:42.183 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:17:42.183 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:42.183 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:42.183 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:42.184 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:17:42.184 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:42.184 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:42.184 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:42.184 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:17:42.184 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:17:42.184 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:17:42.184 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:17:42.186 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:42.186 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:17:42.186 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:42.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 3 2024-09-11T06:17:42.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:17:42.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:17:42.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:17:42.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=15, tag_tid=2, entry_tid=11779], [object_number=14, tag_tid=2, entry_tid=11778], [object_number=13, tag_tid=2, entry_tid=11777], [object_number=12, tag_tid=2, entry_tid=11776]]], state=connected] 2024-09-11T06:17:42.244 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:42.245 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:42.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:17:42.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:42.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:42.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:42.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:17:42.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:42.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:42.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:42.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:17:42.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:17:42.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:17:42.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:17:42.248 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:42.248 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:17:42.248 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:42.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 3 2024-09-11T06:17:42.293 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:17:42.293 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:17:42.293 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:17:42.293 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=15, tag_tid=2, entry_tid=11779], [object_number=14, tag_tid=2, entry_tid=11778], [object_number=13, tag_tid=2, entry_tid=11777], [object_number=12, tag_tid=2, entry_tid=11776]]], state=connected] 2024-09-11T06:17:42.296 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:42.298 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=15, tag_tid=2, entry_tid=11779' 2024-09-11T06:17:42.298 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=21, tag_tid=3, entry_tid=1' -a 'object_number=21, tag_tid=3, entry_tid=1' = 'object_number=15, tag_tid=2, entry_tid=11779' 2024-09-11T06:17:42.298 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=15, tag_tid=2, entry_tid=11779' '!=' 'object_number=14, tag_tid=2, entry_tid=11686' 2024-09-11T06:17:42.298 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:17:42.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=15, tag_tid=2, entry_tid=11779' = 'object_number=14, tag_tid=2, entry_tid=11686' 2024-09-11T06:17:42.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=15, tag_tid=2, entry_tid=11779' 2024-09-11T06:17:42.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:42.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:17:42.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:42.300 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=3 2024-09-11T06:17:42.300 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=15, tag_tid=2, entry_tid=11779' 2024-09-11T06:17:42.301 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:17:42.301 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:42.301 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=2 2024-09-11T06:17:42.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:42.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:17:42.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:42.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:17:42.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=15, tag_tid=2, entry_tid=11779' 2024-09-11T06:17:42.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:17:42.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:42.305 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=11779 2024-09-11T06:17:42.305 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 3 = 2 -a 1 -le 11779 2024-09-11T06:17:42.305 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:17:42.305 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:17:42.305 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:17:42.507 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:17:42.507 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:17:42.507 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:17:42.507 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:17:42.508 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:17:42.508 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:17:42.508 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:17:42.508 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:17:42.508 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:17:42.508 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:17:42.508 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:17:42.508 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:17:42.508 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:17:42.508 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:17:42.508 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:17:42.509 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:17:42.509 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:17:42.511 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:17:42.511 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:17:42.511 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:17:42.511 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:17:42.511 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:17:42.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:17:42.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:17:42.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:17:42.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:17:42.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:17:42.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:17:42.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:17:42.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:17:42.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:17:42.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:17:42.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:17:42.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:17:42.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:17:42.513 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:17:42.513 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:17:42.513 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:17:42.513 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:17:42.513 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:17:42.513 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:17:42.513 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:17:42.513 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:17:42.513 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:17:42.513 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:17:42.513 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:17:42.514 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:17:42.514 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:17:42.514 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:17:42.537 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:42.537 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:42.538 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:42.538 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:17:42.669 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:17:42.669 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:17:42.669 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:42.669 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:42.669 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:42.670 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:17:42.670 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:42.670 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:42.670 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:42.670 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:17:42.671 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:17:42.671 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:17:42.671 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:17:42.672 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:42.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:17:42.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:42.725 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 3 2024-09-11T06:17:42.725 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:17:42.725 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:17:42.725 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:17:42.725 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=11869], [object_number=12, tag_tid=2, entry_tid=11868], [object_number=15, tag_tid=2, entry_tid=11867], [object_number=14, tag_tid=2, entry_tid=11866]]], state=connected] 2024-09-11T06:17:42.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:42.729 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:42.731 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:17:42.731 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:42.731 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:42.731 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:42.731 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:17:42.731 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:42.731 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:42.731 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:42.731 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:17:42.731 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:17:42.731 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:17:42.732 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:17:42.732 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:42.732 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:17:42.732 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:42.777 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 3 2024-09-11T06:17:42.777 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:17:42.778 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:17:42.778 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:17:42.778 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=11869], [object_number=12, tag_tid=2, entry_tid=11868], [object_number=15, tag_tid=2, entry_tid=11867], [object_number=14, tag_tid=2, entry_tid=11866]]], state=connected] 2024-09-11T06:17:42.781 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:42.783 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=13, tag_tid=2, entry_tid=11869' 2024-09-11T06:17:42.783 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=21, tag_tid=3, entry_tid=1' -a 'object_number=21, tag_tid=3, entry_tid=1' = 'object_number=13, tag_tid=2, entry_tid=11869' 2024-09-11T06:17:42.783 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=13, tag_tid=2, entry_tid=11869' '!=' 'object_number=15, tag_tid=2, entry_tid=11779' 2024-09-11T06:17:42.783 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:17:42.783 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=13, tag_tid=2, entry_tid=11869' = 'object_number=15, tag_tid=2, entry_tid=11779' 2024-09-11T06:17:42.783 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=13, tag_tid=2, entry_tid=11869' 2024-09-11T06:17:42.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:42.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:17:42.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:42.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=3 2024-09-11T06:17:42.785 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=13, tag_tid=2, entry_tid=11869' 2024-09-11T06:17:42.785 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:17:42.785 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:42.785 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=2 2024-09-11T06:17:42.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:42.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:17:42.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:42.787 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:17:42.788 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=13, tag_tid=2, entry_tid=11869' 2024-09-11T06:17:42.788 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:17:42.788 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:42.789 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=11869 2024-09-11T06:17:42.789 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 3 = 2 -a 1 -le 11869 2024-09-11T06:17:42.789 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:17:42.790 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:17:42.790 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:17:42.991 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:17:42.991 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:17:42.991 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:17:42.991 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:17:42.991 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:17:42.992 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:17:42.992 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:17:42.992 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:17:42.992 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:17:42.992 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:17:42.992 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:17:42.992 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:17:42.992 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:17:42.992 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:17:42.992 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:17:42.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:17:42.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:17:42.995 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:17:42.996 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:17:42.996 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:17:42.996 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:17:42.996 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:17:42.996 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:17:42.996 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:17:42.996 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:17:42.996 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:17:42.996 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:17:42.996 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:17:42.997 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:17:42.997 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:17:42.997 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:17:42.997 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:17:42.997 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:17:42.997 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:17:42.997 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:17:42.997 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:17:42.997 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:17:42.997 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:17:42.997 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:17:42.997 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:17:42.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:17:42.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:17:42.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:17:42.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:17:42.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:17:42.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:17:42.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:17:42.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:17:42.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:17:43.020 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:43.021 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:43.021 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:43.021 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:17:43.149 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:17:43.150 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:17:43.150 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:43.150 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:43.150 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:43.150 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:17:43.150 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:43.150 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:43.150 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:43.150 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:17:43.151 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:17:43.151 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:17:43.151 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:17:43.153 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:43.153 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:17:43.153 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:43.199 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 3 2024-09-11T06:17:43.199 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:17:43.199 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:17:43.199 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:17:43.199 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=12, tag_tid=2, entry_tid=11972], [object_number=15, tag_tid=2, entry_tid=11971], [object_number=14, tag_tid=2, entry_tid=11970], [object_number=13, tag_tid=2, entry_tid=11969]]], state=connected] 2024-09-11T06:17:43.202 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:43.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:43.205 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:17:43.205 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:43.205 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:43.205 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:43.205 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:17:43.205 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:43.205 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:43.206 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:43.206 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:17:43.206 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:17:43.206 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:17:43.206 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:17:43.206 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:43.206 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:17:43.207 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:43.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 3 2024-09-11T06:17:43.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:17:43.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:17:43.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:17:43.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=12, tag_tid=2, entry_tid=11972], [object_number=15, tag_tid=2, entry_tid=11971], [object_number=14, tag_tid=2, entry_tid=11970], [object_number=13, tag_tid=2, entry_tid=11969]]], state=connected] 2024-09-11T06:17:43.255 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:43.257 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=12, tag_tid=2, entry_tid=11972' 2024-09-11T06:17:43.257 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=21, tag_tid=3, entry_tid=1' -a 'object_number=21, tag_tid=3, entry_tid=1' = 'object_number=12, tag_tid=2, entry_tid=11972' 2024-09-11T06:17:43.257 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=12, tag_tid=2, entry_tid=11972' '!=' 'object_number=13, tag_tid=2, entry_tid=11869' 2024-09-11T06:17:43.257 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:17:43.257 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=12, tag_tid=2, entry_tid=11972' = 'object_number=13, tag_tid=2, entry_tid=11869' 2024-09-11T06:17:43.257 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=12, tag_tid=2, entry_tid=11972' 2024-09-11T06:17:43.257 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:43.257 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:17:43.257 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:43.258 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=3 2024-09-11T06:17:43.258 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=12, tag_tid=2, entry_tid=11972' 2024-09-11T06:17:43.258 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:17:43.259 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:43.259 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=2 2024-09-11T06:17:43.260 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:43.260 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:17:43.260 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:43.261 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:17:43.262 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=12, tag_tid=2, entry_tid=11972' 2024-09-11T06:17:43.262 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:17:43.262 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:43.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=11972 2024-09-11T06:17:43.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 3 = 2 -a 1 -le 11972 2024-09-11T06:17:43.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:17:43.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:17:43.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:17:43.466 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:17:43.466 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:17:43.466 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:17:43.466 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:17:43.466 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:17:43.466 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:17:43.466 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:17:43.466 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:17:43.467 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:17:43.467 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:17:43.467 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:17:43.467 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:17:43.467 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:17:43.467 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:17:43.467 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:17:43.468 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:17:43.468 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:17:43.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:17:43.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:17:43.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:17:43.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:17:43.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:17:43.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:17:43.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:17:43.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:17:43.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:17:43.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:17:43.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:17:43.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:17:43.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:17:43.476 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:17:43.476 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:17:43.476 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:17:43.476 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:17:43.476 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:17:43.476 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:17:43.476 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:17:43.476 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:17:43.476 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:17:43.476 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:17:43.477 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:17:43.477 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:17:43.477 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:17:43.477 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:17:43.477 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:17:43.477 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:17:43.477 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:17:43.477 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:17:43.477 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:17:43.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:43.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:43.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:43.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:17:43.625 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:17:43.627 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:17:43.627 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:43.627 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:43.627 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:43.627 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:17:43.627 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:43.627 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:43.627 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:43.627 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:17:43.628 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:17:43.628 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:17:43.628 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:17:43.629 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:43.629 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:17:43.629 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:43.681 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 3 2024-09-11T06:17:43.681 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:17:43.681 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:17:43.682 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:17:43.682 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=14, tag_tid=2, entry_tid=12074], [object_number=13, tag_tid=2, entry_tid=12073], [object_number=12, tag_tid=2, entry_tid=12072], [object_number=15, tag_tid=2, entry_tid=12071]]], state=connected] 2024-09-11T06:17:43.684 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:43.686 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:43.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:17:43.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:43.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:43.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:43.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:17:43.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:43.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:43.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:43.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:17:43.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:17:43.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:17:43.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:17:43.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:43.689 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:17:43.689 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:43.735 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 3 2024-09-11T06:17:43.735 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:17:43.735 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:17:43.736 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:17:43.736 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=14, tag_tid=2, entry_tid=12074], [object_number=13, tag_tid=2, entry_tid=12073], [object_number=12, tag_tid=2, entry_tid=12072], [object_number=15, tag_tid=2, entry_tid=12071]]], state=connected] 2024-09-11T06:17:43.739 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:43.741 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=14, tag_tid=2, entry_tid=12074' 2024-09-11T06:17:43.741 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=21, tag_tid=3, entry_tid=1' -a 'object_number=21, tag_tid=3, entry_tid=1' = 'object_number=14, tag_tid=2, entry_tid=12074' 2024-09-11T06:17:43.741 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=14, tag_tid=2, entry_tid=12074' '!=' 'object_number=12, tag_tid=2, entry_tid=11972' 2024-09-11T06:17:43.741 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:17:43.741 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=14, tag_tid=2, entry_tid=12074' = 'object_number=12, tag_tid=2, entry_tid=11972' 2024-09-11T06:17:43.741 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=14, tag_tid=2, entry_tid=12074' 2024-09-11T06:17:43.741 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:43.741 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:17:43.741 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:43.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=3 2024-09-11T06:17:43.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=14, tag_tid=2, entry_tid=12074' 2024-09-11T06:17:43.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:17:43.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:43.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=2 2024-09-11T06:17:43.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:43.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:17:43.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:43.745 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:17:43.746 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=14, tag_tid=2, entry_tid=12074' 2024-09-11T06:17:43.746 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:17:43.746 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:43.747 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=12074 2024-09-11T06:17:43.747 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 3 = 2 -a 1 -le 12074 2024-09-11T06:17:43.747 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:17:43.748 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:17:43.748 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:17:43.949 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:17:43.949 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:17:43.949 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:17:43.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:17:43.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:17:43.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:17:43.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:17:43.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:17:43.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:17:43.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:17:43.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:17:43.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:17:43.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:17:43.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:17:43.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:17:43.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:17:43.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:17:43.954 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:17:43.954 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:17:43.954 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:17:43.954 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:17:43.954 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:17:43.954 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:17:43.954 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:17:43.954 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:17:43.954 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:17:43.954 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:17:43.954 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:17:43.954 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:17:43.954 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:17:43.955 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:17:43.955 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:17:43.955 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:17:43.955 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:17:43.955 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:17:43.955 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:17:43.955 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:17:43.955 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:17:43.955 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:17:43.955 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:17:43.955 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:17:43.955 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:17:43.955 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:17:43.955 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:17:43.955 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:17:43.956 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:17:43.956 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:17:43.956 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:17:43.956 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:17:43.980 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:43.980 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:43.980 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:43.980 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:17:44.110 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:17:44.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:17:44.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:44.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:44.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:44.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:17:44.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:44.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:44.112 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:44.112 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:17:44.112 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:17:44.112 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:17:44.112 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:17:44.114 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:44.115 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:17:44.115 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:44.160 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 3 2024-09-11T06:17:44.160 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:17:44.160 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:17:44.160 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:17:44.160 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=12173], [object_number=12, tag_tid=2, entry_tid=12172], [object_number=15, tag_tid=2, entry_tid=12171], [object_number=14, tag_tid=2, entry_tid=12170]]], state=connected] 2024-09-11T06:17:44.163 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:44.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:44.166 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:17:44.166 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:44.166 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:44.166 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:44.166 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:17:44.166 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:44.166 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:44.166 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:44.167 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:17:44.167 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:17:44.167 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:17:44.167 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:17:44.167 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:44.167 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:17:44.167 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:44.424 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 3 2024-09-11T06:17:44.424 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:17:44.424 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:17:44.424 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:17:44.425 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=12173], [object_number=12, tag_tid=2, entry_tid=12172], [object_number=15, tag_tid=2, entry_tid=12171], [object_number=14, tag_tid=2, entry_tid=12170]]], state=connected] 2024-09-11T06:17:44.428 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:44.430 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=13, tag_tid=2, entry_tid=12173' 2024-09-11T06:17:44.430 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=21, tag_tid=3, entry_tid=1' -a 'object_number=21, tag_tid=3, entry_tid=1' = 'object_number=13, tag_tid=2, entry_tid=12173' 2024-09-11T06:17:44.430 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=13, tag_tid=2, entry_tid=12173' '!=' 'object_number=14, tag_tid=2, entry_tid=12074' 2024-09-11T06:17:44.430 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:17:44.430 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=13, tag_tid=2, entry_tid=12173' = 'object_number=14, tag_tid=2, entry_tid=12074' 2024-09-11T06:17:44.430 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=13, tag_tid=2, entry_tid=12173' 2024-09-11T06:17:44.431 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:44.431 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:17:44.431 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:44.432 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=3 2024-09-11T06:17:44.433 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=13, tag_tid=2, entry_tid=12173' 2024-09-11T06:17:44.433 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:17:44.433 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:44.434 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=2 2024-09-11T06:17:44.435 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:44.435 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:17:44.435 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:44.436 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:17:44.437 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=13, tag_tid=2, entry_tid=12173' 2024-09-11T06:17:44.437 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:17:44.437 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:44.439 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=12173 2024-09-11T06:17:44.439 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 3 = 2 -a 1 -le 12173 2024-09-11T06:17:44.439 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:17:44.439 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:17:44.439 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:17:44.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:17:44.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:17:44.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:17:44.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:17:44.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:17:44.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:17:44.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:17:44.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:17:44.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:17:44.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:17:44.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:17:44.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:17:44.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:17:44.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:17:44.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:17:44.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:17:44.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:17:44.646 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:17:44.646 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:17:44.646 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:17:44.646 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:17:44.646 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:17:44.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:17:44.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:17:44.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:17:44.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:17:44.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:17:44.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:17:44.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:17:44.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:17:44.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:17:44.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:17:44.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:17:44.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:17:44.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:17:44.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:17:44.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:17:44.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:17:44.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:17:44.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:17:44.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:17:44.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:17:44.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:17:44.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:17:44.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:17:44.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:17:44.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:17:44.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:17:44.649 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:17:44.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:44.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:44.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:44.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:17:44.810 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:17:44.812 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:17:44.812 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:44.812 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:44.812 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:44.812 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:17:44.812 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:44.812 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:44.812 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:44.812 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:17:44.812 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:17:44.812 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:17:44.813 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:17:44.813 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:44.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:17:44.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:45.283 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 3 2024-09-11T06:17:45.283 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:17:45.283 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:17:45.283 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:17:45.283 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=12321], [object_number=12, tag_tid=2, entry_tid=12320], [object_number=15, tag_tid=2, entry_tid=12319], [object_number=14, tag_tid=2, entry_tid=12318]]], state=connected] 2024-09-11T06:17:45.288 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:45.290 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:45.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:17:45.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:45.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:45.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:45.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:17:45.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:45.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:45.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:45.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:17:45.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:17:45.293 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:17:45.293 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:17:45.295 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:45.295 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:17:45.295 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:45.350 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 3 2024-09-11T06:17:45.350 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:17:45.350 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:17:45.350 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:17:45.350 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=12321], [object_number=12, tag_tid=2, entry_tid=12320], [object_number=15, tag_tid=2, entry_tid=12319], [object_number=14, tag_tid=2, entry_tid=12318]]], state=connected] 2024-09-11T06:17:45.354 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:45.356 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=13, tag_tid=2, entry_tid=12321' 2024-09-11T06:17:45.356 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=21, tag_tid=3, entry_tid=1' -a 'object_number=21, tag_tid=3, entry_tid=1' = 'object_number=13, tag_tid=2, entry_tid=12321' 2024-09-11T06:17:45.356 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=13, tag_tid=2, entry_tid=12321' '!=' 'object_number=13, tag_tid=2, entry_tid=12173' 2024-09-11T06:17:45.356 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:17:45.356 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=13, tag_tid=2, entry_tid=12321' = 'object_number=13, tag_tid=2, entry_tid=12173' 2024-09-11T06:17:45.356 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=13, tag_tid=2, entry_tid=12321' 2024-09-11T06:17:45.357 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:45.357 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:17:45.357 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:45.358 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=3 2024-09-11T06:17:45.358 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=13, tag_tid=2, entry_tid=12321' 2024-09-11T06:17:45.358 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:17:45.358 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:45.359 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=2 2024-09-11T06:17:45.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:45.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:17:45.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:45.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:17:45.362 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=13, tag_tid=2, entry_tid=12321' 2024-09-11T06:17:45.362 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:17:45.362 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:45.363 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=12321 2024-09-11T06:17:45.363 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 3 = 2 -a 1 -le 12321 2024-09-11T06:17:45.364 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:17:45.364 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:17:45.364 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:17:45.565 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:17:45.565 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:17:45.565 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:17:45.565 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:17:45.566 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:17:45.566 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:17:45.566 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:17:45.566 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:17:45.566 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:17:45.566 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:17:45.566 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:17:45.566 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:17:45.566 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:17:45.566 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:17:45.566 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:17:45.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:17:45.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:17:45.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:17:45.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:17:45.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:17:45.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:17:45.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:17:45.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:17:45.572 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:17:45.572 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:17:45.572 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:17:45.572 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:17:45.572 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:17:45.572 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:17:45.572 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:17:45.572 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:17:45.572 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:17:45.572 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:17:45.572 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:17:45.572 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:17:45.573 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:17:45.573 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:17:45.573 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:17:45.573 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:17:45.573 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:17:45.573 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:17:45.573 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:17:45.573 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:17:45.573 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:17:45.573 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:17:45.573 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:17:45.573 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:17:45.574 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:17:45.574 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:17:45.597 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:45.597 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:45.597 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:45.597 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:17:45.725 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:17:45.725 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:17:45.725 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:45.725 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:45.725 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:45.725 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:17:45.725 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:45.725 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:45.726 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:45.726 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:17:45.726 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:17:45.726 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:17:45.726 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:17:45.726 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:45.727 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:45.727 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:17:45.777 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 3 2024-09-11T06:17:45.777 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:17:45.777 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:17:45.778 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:17:45.778 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=14, tag_tid=2, entry_tid=12538], [object_number=13, tag_tid=2, entry_tid=12537], [object_number=12, tag_tid=2, entry_tid=12536], [object_number=15, tag_tid=2, entry_tid=12535]]], state=connected] 2024-09-11T06:17:45.781 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:45.782 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:45.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:17:45.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:45.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:45.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:45.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:17:45.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:45.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:45.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:45.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:17:45.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:17:45.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:17:45.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:17:45.785 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:45.785 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:17:45.785 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:45.830 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 3 2024-09-11T06:17:45.830 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:17:45.830 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:17:45.830 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:17:45.830 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=14, tag_tid=2, entry_tid=12538], [object_number=13, tag_tid=2, entry_tid=12537], [object_number=12, tag_tid=2, entry_tid=12536], [object_number=15, tag_tid=2, entry_tid=12535]]], state=connected] 2024-09-11T06:17:45.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:45.836 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=14, tag_tid=2, entry_tid=12538' 2024-09-11T06:17:45.836 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=21, tag_tid=3, entry_tid=1' -a 'object_number=21, tag_tid=3, entry_tid=1' = 'object_number=14, tag_tid=2, entry_tid=12538' 2024-09-11T06:17:45.836 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=14, tag_tid=2, entry_tid=12538' '!=' 'object_number=13, tag_tid=2, entry_tid=12321' 2024-09-11T06:17:45.836 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:17:45.836 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=14, tag_tid=2, entry_tid=12538' = 'object_number=13, tag_tid=2, entry_tid=12321' 2024-09-11T06:17:45.836 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=14, tag_tid=2, entry_tid=12538' 2024-09-11T06:17:45.836 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:45.836 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:17:45.836 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:45.837 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=3 2024-09-11T06:17:45.837 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=14, tag_tid=2, entry_tid=12538' 2024-09-11T06:17:45.837 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:17:45.838 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:45.838 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=2 2024-09-11T06:17:45.839 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:45.839 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:17:45.839 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:45.840 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:17:45.841 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=14, tag_tid=2, entry_tid=12538' 2024-09-11T06:17:45.841 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:17:45.841 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:45.842 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=12538 2024-09-11T06:17:45.842 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 3 = 2 -a 1 -le 12538 2024-09-11T06:17:45.842 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:17:45.842 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:17:45.842 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:17:46.044 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:17:46.044 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:17:46.044 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:17:46.044 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:17:46.044 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:17:46.044 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:17:46.044 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:17:46.045 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:17:46.045 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:17:46.045 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:17:46.045 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:17:46.045 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:17:46.045 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:17:46.045 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:17:46.045 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:17:46.046 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:17:46.046 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:17:46.047 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:17:46.047 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:17:46.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:17:46.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:17:46.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:17:46.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:17:46.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:17:46.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:17:46.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:17:46.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:17:46.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:17:46.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:17:46.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:17:46.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:17:46.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:17:46.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:17:46.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:17:46.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:17:46.050 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:17:46.050 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:17:46.050 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:17:46.050 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:17:46.050 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:17:46.050 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:17:46.050 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:17:46.050 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:17:46.050 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:17:46.050 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:17:46.050 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:17:46.050 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:17:46.050 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:17:46.051 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:17:46.074 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:46.074 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:46.074 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:46.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:17:46.210 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:17:46.211 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:17:46.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:46.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:46.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:46.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:17:46.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:46.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:46.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:46.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:17:46.213 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:17:46.213 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:17:46.213 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:17:46.215 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:46.215 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:17:46.215 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:46.260 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 3 2024-09-11T06:17:46.260 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:17:46.260 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:17:46.260 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:17:46.260 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=15, tag_tid=2, entry_tid=12647], [object_number=14, tag_tid=2, entry_tid=12646], [object_number=13, tag_tid=2, entry_tid=12645], [object_number=12, tag_tid=2, entry_tid=12644]]], state=connected] 2024-09-11T06:17:46.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:46.265 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:46.266 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:17:46.266 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:46.266 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:46.266 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:46.266 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:17:46.266 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:46.266 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:46.266 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:46.266 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:17:46.266 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:17:46.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:17:46.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:17:46.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:46.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:17:46.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:46.312 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 3 2024-09-11T06:17:46.313 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:17:46.313 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:17:46.313 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:17:46.313 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=15, tag_tid=2, entry_tid=12647], [object_number=14, tag_tid=2, entry_tid=12646], [object_number=13, tag_tid=2, entry_tid=12645], [object_number=12, tag_tid=2, entry_tid=12644]]], state=connected] 2024-09-11T06:17:46.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:46.318 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=15, tag_tid=2, entry_tid=12647' 2024-09-11T06:17:46.318 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=21, tag_tid=3, entry_tid=1' -a 'object_number=21, tag_tid=3, entry_tid=1' = 'object_number=15, tag_tid=2, entry_tid=12647' 2024-09-11T06:17:46.318 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=15, tag_tid=2, entry_tid=12647' '!=' 'object_number=14, tag_tid=2, entry_tid=12538' 2024-09-11T06:17:46.318 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:17:46.318 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=15, tag_tid=2, entry_tid=12647' = 'object_number=14, tag_tid=2, entry_tid=12538' 2024-09-11T06:17:46.318 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=15, tag_tid=2, entry_tid=12647' 2024-09-11T06:17:46.318 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:46.319 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:17:46.319 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:46.319 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=3 2024-09-11T06:17:46.320 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=15, tag_tid=2, entry_tid=12647' 2024-09-11T06:17:46.320 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:17:46.320 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:46.321 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=2 2024-09-11T06:17:46.322 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:46.322 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:17:46.322 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:46.322 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:17:46.323 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=15, tag_tid=2, entry_tid=12647' 2024-09-11T06:17:46.323 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:17:46.323 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:46.325 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=12647 2024-09-11T06:17:46.325 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 3 = 2 -a 1 -le 12647 2024-09-11T06:17:46.325 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:17:46.325 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:17:46.325 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:17:46.526 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:17:46.526 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:17:46.526 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:17:46.526 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:17:46.527 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:17:46.527 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:17:46.527 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:17:46.527 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:17:46.527 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:17:46.527 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:17:46.527 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:17:46.527 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:17:46.527 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:17:46.527 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:17:46.527 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:17:46.528 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:17:46.528 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:17:46.531 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:17:46.531 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:17:46.531 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:17:46.531 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:17:46.531 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:17:46.531 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:17:46.531 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:17:46.531 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:17:46.531 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:17:46.531 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:17:46.531 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:17:46.532 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:17:46.532 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:17:46.532 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:17:46.532 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:17:46.532 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:17:46.532 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:17:46.532 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:17:46.532 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:17:46.532 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:17:46.532 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:17:46.532 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:17:46.532 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:17:46.532 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:17:46.533 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:17:46.533 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:17:46.533 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:17:46.533 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:17:46.533 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:17:46.533 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:17:46.533 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:17:46.533 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:17:46.557 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:46.557 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:46.557 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:46.558 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:17:46.689 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:17:46.689 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:17:46.690 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:46.690 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:46.690 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:46.690 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:17:46.690 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:46.690 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:46.690 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:46.690 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:17:46.690 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:17:46.690 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:17:46.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:17:46.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:46.693 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:46.693 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:17:46.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 3 2024-09-11T06:17:46.739 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:17:46.739 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:17:46.739 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:17:46.739 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=14, tag_tid=2, entry_tid=12766], [object_number=13, tag_tid=2, entry_tid=12765], [object_number=12, tag_tid=2, entry_tid=12764], [object_number=15, tag_tid=2, entry_tid=12763]]], state=connected] 2024-09-11T06:17:46.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:46.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:46.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:17:46.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:46.745 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:46.745 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:46.745 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:17:46.745 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:46.745 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:46.745 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:46.745 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:17:46.745 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:17:46.745 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:17:46.745 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:17:46.746 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:46.746 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:17:46.746 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:46.790 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 3 2024-09-11T06:17:46.790 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:17:46.791 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:17:46.791 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:17:46.791 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=14, tag_tid=2, entry_tid=12766], [object_number=13, tag_tid=2, entry_tid=12765], [object_number=12, tag_tid=2, entry_tid=12764], [object_number=15, tag_tid=2, entry_tid=12763]]], state=connected] 2024-09-11T06:17:46.794 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:46.796 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=14, tag_tid=2, entry_tid=12766' 2024-09-11T06:17:46.796 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=21, tag_tid=3, entry_tid=1' -a 'object_number=21, tag_tid=3, entry_tid=1' = 'object_number=14, tag_tid=2, entry_tid=12766' 2024-09-11T06:17:46.796 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=14, tag_tid=2, entry_tid=12766' '!=' 'object_number=15, tag_tid=2, entry_tid=12647' 2024-09-11T06:17:46.796 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:17:46.797 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=14, tag_tid=2, entry_tid=12766' = 'object_number=15, tag_tid=2, entry_tid=12647' 2024-09-11T06:17:46.797 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=14, tag_tid=2, entry_tid=12766' 2024-09-11T06:17:46.797 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:46.797 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:17:46.797 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:46.797 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=3 2024-09-11T06:17:46.798 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=14, tag_tid=2, entry_tid=12766' 2024-09-11T06:17:46.798 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:17:46.798 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:46.799 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=2 2024-09-11T06:17:46.800 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:46.800 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:17:46.800 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:46.801 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:17:46.801 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=14, tag_tid=2, entry_tid=12766' 2024-09-11T06:17:46.801 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:17:46.801 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:46.803 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=12766 2024-09-11T06:17:46.803 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 3 = 2 -a 1 -le 12766 2024-09-11T06:17:46.803 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:17:46.803 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:17:46.803 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:17:47.005 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:17:47.005 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:17:47.005 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:17:47.005 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:17:47.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:17:47.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:17:47.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:17:47.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:17:47.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:17:47.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:17:47.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:17:47.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:17:47.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:17:47.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:17:47.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:17:47.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:17:47.007 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:17:47.009 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:17:47.009 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:17:47.010 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:17:47.010 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:17:47.010 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:17:47.010 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:17:47.010 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:17:47.010 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:17:47.010 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:17:47.010 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:17:47.010 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:17:47.010 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:17:47.010 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:17:47.010 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:17:47.010 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:17:47.011 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:17:47.011 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:17:47.011 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:17:47.011 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:17:47.011 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:17:47.011 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:17:47.011 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:17:47.011 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:17:47.011 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:17:47.011 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:17:47.011 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:17:47.011 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:17:47.012 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:17:47.012 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:17:47.012 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:17:47.012 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:17:47.012 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:17:47.035 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:47.036 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:47.036 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:47.036 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:17:47.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:17:47.166 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:17:47.167 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:47.167 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:47.167 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:47.167 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:17:47.167 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:47.167 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:47.167 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:47.167 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:17:47.168 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:17:47.168 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:17:47.168 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:17:47.169 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:47.169 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:17:47.169 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:47.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 3 2024-09-11T06:17:47.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:17:47.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:17:47.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:17:47.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=12885], [object_number=12, tag_tid=2, entry_tid=12884], [object_number=15, tag_tid=2, entry_tid=12883], [object_number=14, tag_tid=2, entry_tid=12882]]], state=connected] 2024-09-11T06:17:47.223 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:47.225 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:47.226 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:17:47.226 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:47.226 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:47.226 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:47.226 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:17:47.226 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:47.226 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:47.226 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:47.227 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:17:47.227 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:17:47.227 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:17:47.227 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:17:47.227 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:47.227 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:17:47.227 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:47.273 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 3 2024-09-11T06:17:47.273 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:17:47.273 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:17:47.273 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:17:47.273 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=12885], [object_number=12, tag_tid=2, entry_tid=12884], [object_number=15, tag_tid=2, entry_tid=12883], [object_number=14, tag_tid=2, entry_tid=12882]]], state=connected] 2024-09-11T06:17:47.276 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:47.278 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=13, tag_tid=2, entry_tid=12885' 2024-09-11T06:17:47.279 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=21, tag_tid=3, entry_tid=1' -a 'object_number=21, tag_tid=3, entry_tid=1' = 'object_number=13, tag_tid=2, entry_tid=12885' 2024-09-11T06:17:47.279 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=13, tag_tid=2, entry_tid=12885' '!=' 'object_number=14, tag_tid=2, entry_tid=12766' 2024-09-11T06:17:47.279 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:17:47.279 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=13, tag_tid=2, entry_tid=12885' = 'object_number=14, tag_tid=2, entry_tid=12766' 2024-09-11T06:17:47.279 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=13, tag_tid=2, entry_tid=12885' 2024-09-11T06:17:47.279 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:47.279 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:17:47.279 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:47.280 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=3 2024-09-11T06:17:47.280 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=13, tag_tid=2, entry_tid=12885' 2024-09-11T06:17:47.280 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:17:47.281 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:47.281 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=2 2024-09-11T06:17:47.282 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:47.282 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:17:47.282 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:47.283 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:17:47.284 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=13, tag_tid=2, entry_tid=12885' 2024-09-11T06:17:47.284 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:17:47.284 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:47.285 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=12885 2024-09-11T06:17:47.285 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 3 = 2 -a 1 -le 12885 2024-09-11T06:17:47.285 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:17:47.285 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:17:47.285 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:17:47.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:17:47.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:17:47.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:17:47.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:17:47.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:17:47.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:17:47.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:17:47.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:17:47.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:17:47.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:17:47.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:17:47.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:17:47.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:17:47.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:17:47.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:17:47.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:17:47.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:17:47.492 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:17:47.492 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:17:47.492 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:17:47.492 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:17:47.492 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:17:47.493 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:17:47.493 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:17:47.493 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:17:47.493 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:17:47.493 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:17:47.493 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:17:47.493 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:17:47.493 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:17:47.493 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:17:47.493 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:17:47.493 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:17:47.493 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:17:47.493 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:17:47.493 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:17:47.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:17:47.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:17:47.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:17:47.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:17:47.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:17:47.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:17:47.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:17:47.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:17:47.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:17:47.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:17:47.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:17:47.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:17:47.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:17:47.518 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:47.518 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:47.518 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:47.518 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:17:47.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:17:47.655 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:17:47.656 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:47.656 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:47.656 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:47.656 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:17:47.656 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:47.656 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:47.656 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:47.656 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:17:47.656 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:17:47.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:17:47.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:17:47.658 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:47.658 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:17:47.658 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:47.702 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 3 2024-09-11T06:17:47.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:17:47.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:17:47.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:17:47.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=13009], [object_number=12, tag_tid=2, entry_tid=13008], [object_number=15, tag_tid=2, entry_tid=13007], [object_number=14, tag_tid=2, entry_tid=13006]]], state=connected] 2024-09-11T06:17:47.706 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:47.707 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:47.708 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:17:47.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:47.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:47.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:47.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:17:47.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:47.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:47.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:47.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:17:47.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:17:47.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:17:47.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:17:47.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:47.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:17:47.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:47.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 3 2024-09-11T06:17:47.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:17:47.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:17:47.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:17:47.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=13009], [object_number=12, tag_tid=2, entry_tid=13008], [object_number=15, tag_tid=2, entry_tid=13007], [object_number=14, tag_tid=2, entry_tid=13006]]], state=connected] 2024-09-11T06:17:47.758 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:47.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=13, tag_tid=2, entry_tid=13009' 2024-09-11T06:17:47.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=21, tag_tid=3, entry_tid=1' -a 'object_number=21, tag_tid=3, entry_tid=1' = 'object_number=13, tag_tid=2, entry_tid=13009' 2024-09-11T06:17:47.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=13, tag_tid=2, entry_tid=13009' '!=' 'object_number=13, tag_tid=2, entry_tid=12885' 2024-09-11T06:17:47.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:17:47.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=13, tag_tid=2, entry_tid=13009' = 'object_number=13, tag_tid=2, entry_tid=12885' 2024-09-11T06:17:47.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=13, tag_tid=2, entry_tid=13009' 2024-09-11T06:17:47.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:47.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:17:47.764 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:47.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=3 2024-09-11T06:17:47.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=13, tag_tid=2, entry_tid=13009' 2024-09-11T06:17:47.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:17:47.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:47.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=2 2024-09-11T06:17:47.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:47.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:17:47.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:47.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:17:47.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=13, tag_tid=2, entry_tid=13009' 2024-09-11T06:17:47.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:17:47.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:47.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=13009 2024-09-11T06:17:47.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 3 = 2 -a 1 -le 13009 2024-09-11T06:17:47.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:17:47.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:17:47.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:17:47.969 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:17:47.969 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:17:47.969 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:17:47.969 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:17:47.969 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:17:47.970 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:17:47.970 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:17:47.970 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:17:47.970 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:17:47.970 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:17:47.970 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:17:47.970 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:17:47.970 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:17:47.970 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:17:47.970 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:17:47.971 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:17:47.971 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:17:47.973 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:17:47.973 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:17:47.973 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:17:47.973 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:17:47.973 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:17:47.973 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:17:47.973 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:17:47.973 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:17:47.973 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:17:47.973 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:17:47.973 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:17:47.973 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:17:47.974 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:17:47.974 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:17:47.974 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:17:47.974 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:17:47.974 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:17:47.974 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:17:47.974 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:17:47.974 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:17:47.974 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:17:47.974 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:17:47.974 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:17:47.974 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:17:47.974 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:17:47.974 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:17:47.974 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:17:47.975 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:17:47.975 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:17:47.975 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:17:47.975 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:17:47.975 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:17:48.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:48.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:48.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:48.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:17:48.130 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:17:48.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:17:48.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:48.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:48.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:48.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:17:48.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:48.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:48.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:48.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:17:48.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:17:48.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:17:48.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:17:48.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:48.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:17:48.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:48.186 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 3 2024-09-11T06:17:48.186 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:17:48.186 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:17:48.186 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:17:48.186 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=15, tag_tid=2, entry_tid=13127], [object_number=14, tag_tid=2, entry_tid=13126], [object_number=13, tag_tid=2, entry_tid=13125], [object_number=12, tag_tid=2, entry_tid=13124]]], state=connected] 2024-09-11T06:17:48.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:48.190 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:48.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:17:48.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:48.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:48.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:48.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:17:48.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:48.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:48.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:48.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:17:48.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:17:48.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:17:48.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:17:48.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:48.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:17:48.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:48.238 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 3 2024-09-11T06:17:48.238 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:17:48.238 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:17:48.238 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:17:48.239 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=15, tag_tid=2, entry_tid=13127], [object_number=14, tag_tid=2, entry_tid=13126], [object_number=13, tag_tid=2, entry_tid=13125], [object_number=12, tag_tid=2, entry_tid=13124]]], state=connected] 2024-09-11T06:17:48.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:48.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=15, tag_tid=2, entry_tid=13127' 2024-09-11T06:17:48.244 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=21, tag_tid=3, entry_tid=1' -a 'object_number=21, tag_tid=3, entry_tid=1' = 'object_number=15, tag_tid=2, entry_tid=13127' 2024-09-11T06:17:48.244 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=15, tag_tid=2, entry_tid=13127' '!=' 'object_number=13, tag_tid=2, entry_tid=13009' 2024-09-11T06:17:48.244 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:17:48.244 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=15, tag_tid=2, entry_tid=13127' = 'object_number=13, tag_tid=2, entry_tid=13009' 2024-09-11T06:17:48.244 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=15, tag_tid=2, entry_tid=13127' 2024-09-11T06:17:48.245 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:48.245 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:17:48.245 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:48.245 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=3 2024-09-11T06:17:48.246 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=15, tag_tid=2, entry_tid=13127' 2024-09-11T06:17:48.246 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:17:48.246 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:48.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=2 2024-09-11T06:17:48.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:48.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:17:48.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:48.248 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:17:48.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=15, tag_tid=2, entry_tid=13127' 2024-09-11T06:17:48.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:17:48.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:48.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=13127 2024-09-11T06:17:48.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 3 = 2 -a 1 -le 13127 2024-09-11T06:17:48.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:17:48.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:17:48.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:17:48.452 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:17:48.452 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:17:48.452 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:17:48.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:17:48.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:17:48.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:17:48.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:17:48.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:17:48.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:17:48.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:17:48.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:17:48.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:17:48.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:17:48.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:17:48.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:17:48.454 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:17:48.454 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:17:48.456 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:17:48.457 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:17:48.457 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:17:48.457 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:17:48.457 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:17:48.457 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:17:48.457 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:17:48.457 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:17:48.457 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:17:48.457 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:17:48.457 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:17:48.457 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:17:48.457 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:17:48.457 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:17:48.458 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:17:48.458 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:17:48.458 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:17:48.458 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:17:48.458 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:17:48.458 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:17:48.458 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:17:48.458 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:17:48.458 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:17:48.458 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:17:48.458 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:17:48.458 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:17:48.458 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:17:48.459 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:17:48.459 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:17:48.459 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:17:48.459 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:17:48.459 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:17:48.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:48.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:48.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:48.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:17:48.608 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:17:48.610 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:17:48.610 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:48.610 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:48.610 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:48.610 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:17:48.610 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:48.611 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:48.611 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:48.611 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:17:48.611 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:17:48.611 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:17:48.611 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:17:48.612 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:48.613 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:17:48.613 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:48.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 3 2024-09-11T06:17:48.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:17:48.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:17:48.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:17:48.658 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=15, tag_tid=2, entry_tid=13247], [object_number=14, tag_tid=2, entry_tid=13246], [object_number=13, tag_tid=2, entry_tid=13245], [object_number=12, tag_tid=2, entry_tid=13244]]], state=connected] 2024-09-11T06:17:48.661 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:48.662 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:48.663 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:17:48.664 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:48.664 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:48.664 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:48.664 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:17:48.664 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:48.664 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:48.664 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:48.664 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:17:48.664 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:17:48.664 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:17:48.664 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:17:48.665 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:48.665 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:17:48.665 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:48.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 3 2024-09-11T06:17:48.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:17:48.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:17:48.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:17:48.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=15, tag_tid=2, entry_tid=13247], [object_number=14, tag_tid=2, entry_tid=13246], [object_number=13, tag_tid=2, entry_tid=13245], [object_number=12, tag_tid=2, entry_tid=13244]]], state=connected] 2024-09-11T06:17:48.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:48.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=15, tag_tid=2, entry_tid=13247' 2024-09-11T06:17:48.716 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=21, tag_tid=3, entry_tid=1' -a 'object_number=21, tag_tid=3, entry_tid=1' = 'object_number=15, tag_tid=2, entry_tid=13247' 2024-09-11T06:17:48.716 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=15, tag_tid=2, entry_tid=13247' '!=' 'object_number=15, tag_tid=2, entry_tid=13127' 2024-09-11T06:17:48.716 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:17:48.716 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=15, tag_tid=2, entry_tid=13247' = 'object_number=15, tag_tid=2, entry_tid=13127' 2024-09-11T06:17:48.716 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=15, tag_tid=2, entry_tid=13247' 2024-09-11T06:17:48.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:48.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:17:48.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:48.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=3 2024-09-11T06:17:48.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=15, tag_tid=2, entry_tid=13247' 2024-09-11T06:17:48.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:17:48.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:48.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=2 2024-09-11T06:17:48.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:48.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:17:48.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:48.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:17:48.721 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=15, tag_tid=2, entry_tid=13247' 2024-09-11T06:17:48.721 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:17:48.721 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:48.722 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=13247 2024-09-11T06:17:48.723 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 3 = 2 -a 1 -le 13247 2024-09-11T06:17:48.723 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:17:48.723 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:17:48.723 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:17:48.924 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:17:48.925 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:17:48.925 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:17:48.925 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:17:48.925 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:17:48.925 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:17:48.925 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:17:48.925 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:17:48.925 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:17:48.925 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:17:48.925 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:17:48.925 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:17:48.926 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:17:48.926 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:17:48.926 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:17:48.926 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:17:48.926 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:17:48.930 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:17:48.930 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:17:48.930 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:17:48.930 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:17:48.930 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:17:48.930 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:17:48.930 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:17:48.930 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:17:48.930 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:17:48.931 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:17:48.931 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:17:48.931 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:17:48.931 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:17:48.931 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:17:48.931 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:17:48.931 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:17:48.931 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:17:48.931 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:17:48.931 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:17:48.931 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:17:48.931 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:17:48.931 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:17:48.932 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:17:48.932 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:17:48.932 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:17:48.932 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:17:48.932 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:17:48.932 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:17:48.932 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:17:48.932 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:17:48.932 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:17:48.932 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:17:48.954 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:48.954 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:48.954 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:48.955 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:17:49.082 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:17:49.083 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:17:49.083 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:49.084 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:49.084 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:49.084 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:17:49.084 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:49.084 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:49.084 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:49.084 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:17:49.085 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:17:49.085 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:17:49.085 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:17:49.086 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:49.086 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:17:49.086 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:49.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 3 2024-09-11T06:17:49.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:17:49.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:17:49.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:17:49.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=14, tag_tid=2, entry_tid=13366], [object_number=13, tag_tid=2, entry_tid=13365], [object_number=12, tag_tid=2, entry_tid=13364], [object_number=15, tag_tid=2, entry_tid=13363]]], state=connected] 2024-09-11T06:17:49.145 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:49.147 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:49.147 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:17:49.147 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:49.147 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:49.147 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:49.147 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:17:49.147 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:49.147 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:49.147 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:49.147 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:17:49.147 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:17:49.148 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:17:49.148 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:17:49.148 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:49.148 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:17:49.149 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:49.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 3 2024-09-11T06:17:49.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:17:49.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:17:49.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:17:49.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=14, tag_tid=2, entry_tid=13366], [object_number=13, tag_tid=2, entry_tid=13365], [object_number=12, tag_tid=2, entry_tid=13364], [object_number=15, tag_tid=2, entry_tid=13363]]], state=connected] 2024-09-11T06:17:49.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:49.200 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=14, tag_tid=2, entry_tid=13366' 2024-09-11T06:17:49.200 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=21, tag_tid=3, entry_tid=1' -a 'object_number=21, tag_tid=3, entry_tid=1' = 'object_number=14, tag_tid=2, entry_tid=13366' 2024-09-11T06:17:49.200 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=14, tag_tid=2, entry_tid=13366' '!=' 'object_number=15, tag_tid=2, entry_tid=13247' 2024-09-11T06:17:49.200 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:17:49.200 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=14, tag_tid=2, entry_tid=13366' = 'object_number=15, tag_tid=2, entry_tid=13247' 2024-09-11T06:17:49.200 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=14, tag_tid=2, entry_tid=13366' 2024-09-11T06:17:49.200 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:49.201 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:17:49.201 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:49.201 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=3 2024-09-11T06:17:49.202 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=14, tag_tid=2, entry_tid=13366' 2024-09-11T06:17:49.202 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:17:49.202 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:49.202 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=2 2024-09-11T06:17:49.203 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:49.203 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:17:49.203 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:49.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:17:49.205 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=14, tag_tid=2, entry_tid=13366' 2024-09-11T06:17:49.205 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:17:49.205 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:49.206 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=13366 2024-09-11T06:17:49.206 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 3 = 2 -a 1 -le 13366 2024-09-11T06:17:49.206 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:17:49.206 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:17:49.207 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:17:49.408 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:17:49.408 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:17:49.408 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:17:49.408 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:17:49.408 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:17:49.408 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:17:49.408 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:17:49.408 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:17:49.408 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:17:49.409 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:17:49.409 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:17:49.409 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:17:49.409 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:17:49.409 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:17:49.409 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:17:49.409 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:17:49.409 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:17:49.413 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:17:49.413 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:17:49.413 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:17:49.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:17:49.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:17:49.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:17:49.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:17:49.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:17:49.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:17:49.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:17:49.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:17:49.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:17:49.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:17:49.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:17:49.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:17:49.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:17:49.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:17:49.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:17:49.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:17:49.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:17:49.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:17:49.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:17:49.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:17:49.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:17:49.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:17:49.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:17:49.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:17:49.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:17:49.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:17:49.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:17:49.416 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:17:49.416 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:17:49.438 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:49.439 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:49.439 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:49.439 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:17:49.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:17:49.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:17:49.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:49.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:49.569 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:49.569 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:17:49.569 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:49.569 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:49.569 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:49.569 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:17:49.569 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:17:49.569 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:17:49.570 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:17:49.572 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:49.572 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:17:49.573 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:49.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 3 2024-09-11T06:17:49.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:17:49.623 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:17:49.623 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:17:49.623 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=12, tag_tid=2, entry_tid=13488], [object_number=15, tag_tid=2, entry_tid=13487], [object_number=14, tag_tid=2, entry_tid=13486], [object_number=13, tag_tid=2, entry_tid=13485]]], state=connected] 2024-09-11T06:17:49.626 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:49.627 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:49.629 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:17:49.629 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:49.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:49.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:49.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:17:49.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:49.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:49.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:49.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:17:49.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:17:49.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:17:49.631 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:17:49.631 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:49.631 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:17:49.631 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:49.676 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 3 2024-09-11T06:17:49.676 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:17:49.676 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:17:49.676 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:17:49.676 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=12, tag_tid=2, entry_tid=13488], [object_number=15, tag_tid=2, entry_tid=13487], [object_number=14, tag_tid=2, entry_tid=13486], [object_number=13, tag_tid=2, entry_tid=13485]]], state=connected] 2024-09-11T06:17:49.679 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:49.681 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=12, tag_tid=2, entry_tid=13488' 2024-09-11T06:17:49.681 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=21, tag_tid=3, entry_tid=1' -a 'object_number=21, tag_tid=3, entry_tid=1' = 'object_number=12, tag_tid=2, entry_tid=13488' 2024-09-11T06:17:49.682 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=12, tag_tid=2, entry_tid=13488' '!=' 'object_number=14, tag_tid=2, entry_tid=13366' 2024-09-11T06:17:49.682 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:17:49.682 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=12, tag_tid=2, entry_tid=13488' = 'object_number=14, tag_tid=2, entry_tid=13366' 2024-09-11T06:17:49.682 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=12, tag_tid=2, entry_tid=13488' 2024-09-11T06:17:49.682 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:49.682 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:17:49.682 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:49.682 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=3 2024-09-11T06:17:49.683 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=12, tag_tid=2, entry_tid=13488' 2024-09-11T06:17:49.683 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:17:49.683 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:49.684 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=2 2024-09-11T06:17:49.685 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:49.685 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:17:49.685 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:49.686 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:17:49.686 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=12, tag_tid=2, entry_tid=13488' 2024-09-11T06:17:49.686 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:17:49.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:49.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=13488 2024-09-11T06:17:49.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 3 = 2 -a 1 -le 13488 2024-09-11T06:17:49.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:17:49.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:17:49.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:17:49.890 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:17:49.890 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:17:49.890 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:17:49.890 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:17:49.890 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:17:49.890 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:17:49.890 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:17:49.890 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:17:49.890 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:17:49.891 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:17:49.891 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:17:49.891 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:17:49.891 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:17:49.891 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:17:49.891 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:17:49.891 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:17:49.891 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:17:49.895 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:17:49.895 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:17:49.895 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:17:49.895 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:17:49.895 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:17:49.895 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:17:49.895 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:17:49.895 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:17:49.896 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:17:49.896 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:17:49.896 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:17:49.896 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:17:49.896 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:17:49.896 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:17:49.896 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:17:49.896 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:17:49.896 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:17:49.896 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:17:49.896 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:17:49.896 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:17:49.896 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:17:49.896 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:17:49.897 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:17:49.897 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:17:49.897 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:17:49.897 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:17:49.897 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:17:49.897 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:17:49.897 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:17:49.897 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:17:49.897 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:17:49.897 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:17:49.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:49.921 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:49.921 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:49.921 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:17:50.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:17:50.056 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:17:50.056 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:50.056 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:50.056 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:50.056 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:17:50.056 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:50.056 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:50.057 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:50.057 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:17:50.057 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:17:50.057 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:17:50.057 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:17:50.058 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:50.059 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:17:50.059 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:50.104 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 3 2024-09-11T06:17:50.104 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:17:50.104 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:17:50.104 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:17:50.105 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=14, tag_tid=2, entry_tid=13606], [object_number=13, tag_tid=2, entry_tid=13605], [object_number=12, tag_tid=2, entry_tid=13604], [object_number=15, tag_tid=2, entry_tid=13603]]], state=connected] 2024-09-11T06:17:50.108 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:50.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:50.110 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:17:50.110 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:50.110 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:50.110 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:50.110 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:17:50.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:50.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:50.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:50.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:17:50.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:17:50.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:17:50.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:17:50.112 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:50.112 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:17:50.112 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:50.157 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 3 2024-09-11T06:17:50.157 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:17:50.157 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:17:50.157 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:17:50.157 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=14, tag_tid=2, entry_tid=13606], [object_number=13, tag_tid=2, entry_tid=13605], [object_number=12, tag_tid=2, entry_tid=13604], [object_number=15, tag_tid=2, entry_tid=13603]]], state=connected] 2024-09-11T06:17:50.160 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:50.162 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=14, tag_tid=2, entry_tid=13606' 2024-09-11T06:17:50.162 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=21, tag_tid=3, entry_tid=1' -a 'object_number=21, tag_tid=3, entry_tid=1' = 'object_number=14, tag_tid=2, entry_tid=13606' 2024-09-11T06:17:50.162 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=14, tag_tid=2, entry_tid=13606' '!=' 'object_number=12, tag_tid=2, entry_tid=13488' 2024-09-11T06:17:50.162 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:17:50.162 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=14, tag_tid=2, entry_tid=13606' = 'object_number=12, tag_tid=2, entry_tid=13488' 2024-09-11T06:17:50.162 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=14, tag_tid=2, entry_tid=13606' 2024-09-11T06:17:50.163 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:50.163 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:17:50.163 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:50.163 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=3 2024-09-11T06:17:50.164 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=14, tag_tid=2, entry_tid=13606' 2024-09-11T06:17:50.164 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:17:50.164 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:50.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=2 2024-09-11T06:17:50.166 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:50.166 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:17:50.166 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:50.167 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:17:50.167 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=14, tag_tid=2, entry_tid=13606' 2024-09-11T06:17:50.167 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:17:50.168 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:50.169 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=13606 2024-09-11T06:17:50.169 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 3 = 2 -a 1 -le 13606 2024-09-11T06:17:50.169 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:17:50.169 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:17:50.169 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:17:50.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:17:50.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:17:50.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:17:50.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:17:50.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:17:50.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:17:50.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:17:50.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:17:50.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:17:50.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:17:50.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:17:50.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:17:50.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:17:50.372 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:17:50.372 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:17:50.372 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:17:50.372 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:17:50.376 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:17:50.376 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:17:50.376 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:17:50.376 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:17:50.376 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:17:50.376 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:17:50.376 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:17:50.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:17:50.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:17:50.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:17:50.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:17:50.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:17:50.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:17:50.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:17:50.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:17:50.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:17:50.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:17:50.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:17:50.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:17:50.378 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:17:50.378 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:17:50.378 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:17:50.378 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:17:50.378 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:17:50.378 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:17:50.378 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:17:50.378 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:17:50.378 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:17:50.378 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:17:50.378 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:17:50.378 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:17:50.379 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:17:50.403 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:50.403 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:50.403 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:50.403 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:17:50.533 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:17:50.534 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:17:50.534 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:50.535 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:50.535 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:50.535 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:17:50.535 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:50.535 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:50.535 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:50.535 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:17:50.536 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:17:50.536 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:17:50.536 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:17:50.537 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:50.537 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:17:50.537 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:50.585 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 3 2024-09-11T06:17:50.585 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:17:50.585 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:17:50.585 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:17:50.585 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=13713], [object_number=12, tag_tid=2, entry_tid=13712], [object_number=15, tag_tid=2, entry_tid=13711], [object_number=14, tag_tid=2, entry_tid=13710]]], state=connected] 2024-09-11T06:17:50.588 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:50.590 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:50.591 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:17:50.591 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:50.591 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:50.591 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:50.591 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:17:50.591 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:50.591 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:50.591 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:50.591 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:17:50.591 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:17:50.592 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:17:50.592 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:17:50.592 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:50.592 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:17:50.593 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:50.636 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 3 2024-09-11T06:17:50.637 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:17:50.637 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:17:50.637 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:17:50.637 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=13713], [object_number=12, tag_tid=2, entry_tid=13712], [object_number=15, tag_tid=2, entry_tid=13711], [object_number=14, tag_tid=2, entry_tid=13710]]], state=connected] 2024-09-11T06:17:50.640 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:50.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=13, tag_tid=2, entry_tid=13713' 2024-09-11T06:17:50.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=21, tag_tid=3, entry_tid=1' -a 'object_number=21, tag_tid=3, entry_tid=1' = 'object_number=13, tag_tid=2, entry_tid=13713' 2024-09-11T06:17:50.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=13, tag_tid=2, entry_tid=13713' '!=' 'object_number=14, tag_tid=2, entry_tid=13606' 2024-09-11T06:17:50.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:17:50.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=13, tag_tid=2, entry_tid=13713' = 'object_number=14, tag_tid=2, entry_tid=13606' 2024-09-11T06:17:50.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=13, tag_tid=2, entry_tid=13713' 2024-09-11T06:17:50.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:50.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:17:50.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:50.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=3 2024-09-11T06:17:50.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=13, tag_tid=2, entry_tid=13713' 2024-09-11T06:17:50.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:17:50.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:50.645 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=2 2024-09-11T06:17:50.645 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:50.645 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:17:50.646 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:50.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:17:50.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=13, tag_tid=2, entry_tid=13713' 2024-09-11T06:17:50.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:17:50.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:50.649 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=13713 2024-09-11T06:17:50.649 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 3 = 2 -a 1 -le 13713 2024-09-11T06:17:50.649 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:17:50.649 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:17:50.649 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:17:50.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:17:50.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:17:50.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:17:50.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:17:50.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:17:50.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:17:50.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:17:50.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:17:50.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:17:50.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:17:50.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:17:50.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:17:50.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:17:50.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:17:50.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:17:50.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:17:50.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:17:50.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:17:50.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:17:50.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:17:50.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:17:50.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:17:50.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:17:50.857 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:17:50.857 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:17:50.857 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:17:50.857 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:17:50.857 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:17:50.857 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:17:50.857 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:17:50.857 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:17:50.857 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:17:50.857 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:17:50.857 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:17:50.857 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:17:50.858 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:17:50.858 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:17:50.858 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:17:50.858 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:17:50.858 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:17:50.858 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:17:50.858 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:17:50.858 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:17:50.858 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:17:50.858 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:17:50.858 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:17:50.858 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:17:50.858 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:17:50.859 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:17:50.881 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:50.882 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:50.882 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:50.882 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:17:51.018 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:17:51.019 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:17:51.019 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:51.019 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:51.019 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:51.019 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:17:51.019 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:51.019 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:51.019 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:51.019 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:17:51.020 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:17:51.020 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:17:51.020 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:17:51.022 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:51.023 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:17:51.023 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:51.073 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 3 2024-09-11T06:17:51.073 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:17:51.073 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:17:51.073 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:17:51.074 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=15, tag_tid=2, entry_tid=13827], [object_number=14, tag_tid=2, entry_tid=13826], [object_number=13, tag_tid=2, entry_tid=13825], [object_number=12, tag_tid=2, entry_tid=13824]]], state=connected] 2024-09-11T06:17:51.077 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:51.078 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:51.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:17:51.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:51.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:51.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:51.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:17:51.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:51.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:51.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:51.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:17:51.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:17:51.081 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:17:51.081 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:17:51.081 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:51.081 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:17:51.082 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:51.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 3 2024-09-11T06:17:51.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:17:51.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:17:51.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:17:51.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=15, tag_tid=2, entry_tid=13827], [object_number=14, tag_tid=2, entry_tid=13826], [object_number=13, tag_tid=2, entry_tid=13825], [object_number=12, tag_tid=2, entry_tid=13824]]], state=connected] 2024-09-11T06:17:51.129 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:51.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=15, tag_tid=2, entry_tid=13827' 2024-09-11T06:17:51.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=21, tag_tid=3, entry_tid=1' -a 'object_number=21, tag_tid=3, entry_tid=1' = 'object_number=15, tag_tid=2, entry_tid=13827' 2024-09-11T06:17:51.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=15, tag_tid=2, entry_tid=13827' '!=' 'object_number=13, tag_tid=2, entry_tid=13713' 2024-09-11T06:17:51.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:17:51.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=15, tag_tid=2, entry_tid=13827' = 'object_number=13, tag_tid=2, entry_tid=13713' 2024-09-11T06:17:51.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=15, tag_tid=2, entry_tid=13827' 2024-09-11T06:17:51.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:51.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:17:51.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:51.133 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=3 2024-09-11T06:17:51.133 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=15, tag_tid=2, entry_tid=13827' 2024-09-11T06:17:51.133 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:17:51.133 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:51.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=2 2024-09-11T06:17:51.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:51.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:17:51.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:51.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:17:51.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=15, tag_tid=2, entry_tid=13827' 2024-09-11T06:17:51.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:17:51.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:51.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=13827 2024-09-11T06:17:51.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 3 = 2 -a 1 -le 13827 2024-09-11T06:17:51.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:17:51.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:17:51.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:17:51.340 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:17:51.340 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:17:51.341 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:17:51.341 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:17:51.341 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:17:51.341 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:17:51.341 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:17:51.341 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:17:51.341 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:17:51.341 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:17:51.341 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:17:51.341 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:17:51.341 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:17:51.341 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:17:51.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:17:51.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:17:51.345 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:17:51.345 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:17:51.345 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:17:51.345 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:17:51.345 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:17:51.345 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:17:51.345 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:17:51.345 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:17:51.345 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:17:51.345 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:17:51.346 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:17:51.346 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:17:51.346 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:17:51.346 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:17:51.346 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:17:51.346 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:17:51.346 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:17:51.346 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:17:51.346 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:17:51.346 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:17:51.346 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:17:51.346 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:17:51.347 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:17:51.347 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:17:51.347 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:17:51.347 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:17:51.347 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:17:51.347 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:17:51.347 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:17:51.347 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:17:51.347 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:17:51.347 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:17:51.347 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:17:51.369 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:51.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:51.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:51.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:17:51.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:17:51.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:17:51.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:51.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:51.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:51.498 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:17:51.498 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:51.498 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:51.498 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:51.498 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:17:51.498 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:17:51.499 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:17:51.499 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:17:51.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:51.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:17:51.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:51.545 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 3 2024-09-11T06:17:51.545 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:17:51.545 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:17:51.545 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:17:51.545 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=14, tag_tid=2, entry_tid=13938], [object_number=13, tag_tid=2, entry_tid=13937], [object_number=12, tag_tid=2, entry_tid=13936], [object_number=15, tag_tid=2, entry_tid=13935]]], state=connected] 2024-09-11T06:17:51.548 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:51.550 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:51.551 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:17:51.551 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:51.551 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:51.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:51.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:17:51.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:51.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:51.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:51.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:17:51.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:17:51.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:17:51.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:17:51.553 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:51.553 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:17:51.553 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:51.597 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 3 2024-09-11T06:17:51.597 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:17:51.597 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:17:51.597 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:17:51.597 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=14, tag_tid=2, entry_tid=13938], [object_number=13, tag_tid=2, entry_tid=13937], [object_number=12, tag_tid=2, entry_tid=13936], [object_number=15, tag_tid=2, entry_tid=13935]]], state=connected] 2024-09-11T06:17:51.600 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:51.602 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=14, tag_tid=2, entry_tid=13938' 2024-09-11T06:17:51.602 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=21, tag_tid=3, entry_tid=1' -a 'object_number=21, tag_tid=3, entry_tid=1' = 'object_number=14, tag_tid=2, entry_tid=13938' 2024-09-11T06:17:51.602 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=14, tag_tid=2, entry_tid=13938' '!=' 'object_number=15, tag_tid=2, entry_tid=13827' 2024-09-11T06:17:51.602 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:17:51.602 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=14, tag_tid=2, entry_tid=13938' = 'object_number=15, tag_tid=2, entry_tid=13827' 2024-09-11T06:17:51.602 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=14, tag_tid=2, entry_tid=13938' 2024-09-11T06:17:51.602 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:51.603 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:17:51.603 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:51.603 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=3 2024-09-11T06:17:51.604 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=14, tag_tid=2, entry_tid=13938' 2024-09-11T06:17:51.604 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:17:51.604 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:51.605 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=2 2024-09-11T06:17:51.606 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:51.606 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:17:51.606 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:51.607 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:17:51.607 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=14, tag_tid=2, entry_tid=13938' 2024-09-11T06:17:51.607 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:17:51.608 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:51.609 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=13938 2024-09-11T06:17:51.609 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 3 = 2 -a 1 -le 13938 2024-09-11T06:17:51.609 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:17:51.609 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:17:51.609 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:17:51.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:17:51.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:17:51.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:17:51.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:17:51.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:17:51.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:17:51.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:17:51.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:17:51.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:17:51.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:17:51.812 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:17:51.812 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:17:51.812 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:17:51.812 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:17:51.812 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:17:51.812 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:17:51.813 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:17:51.816 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:17:51.816 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:17:51.816 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:17:51.816 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:17:51.816 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:17:51.817 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:17:51.817 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:17:51.817 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:17:51.817 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:17:51.817 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:17:51.817 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:17:51.817 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:17:51.817 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:17:51.817 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:17:51.817 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:17:51.817 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:17:51.817 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:17:51.817 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:17:51.818 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:17:51.818 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:17:51.818 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:17:51.818 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:17:51.818 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:17:51.818 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:17:51.818 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:17:51.818 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:17:51.818 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:17:51.818 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:17:51.818 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:17:51.818 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:17:51.818 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:17:51.818 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:17:51.846 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:51.846 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:51.846 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:51.846 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:17:51.972 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:17:51.973 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:17:51.973 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:51.973 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:51.974 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:51.974 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:17:51.974 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:51.974 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:51.974 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:51.974 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:17:51.975 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:17:51.975 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:17:51.975 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:17:51.976 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:51.976 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:17:51.977 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:52.021 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 3 2024-09-11T06:17:52.021 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:17:52.021 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:17:52.021 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:17:52.021 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=12, tag_tid=2, entry_tid=14048], [object_number=15, tag_tid=2, entry_tid=14047], [object_number=14, tag_tid=2, entry_tid=14046], [object_number=13, tag_tid=2, entry_tid=14045]]], state=connected] 2024-09-11T06:17:52.024 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:52.026 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:52.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:17:52.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:52.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:52.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:52.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:17:52.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:52.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:52.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:52.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:17:52.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:17:52.028 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:17:52.028 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:17:52.028 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:52.028 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:17:52.028 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:52.072 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 3 2024-09-11T06:17:52.072 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:17:52.072 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:17:52.072 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:17:52.072 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=12, tag_tid=2, entry_tid=14048], [object_number=15, tag_tid=2, entry_tid=14047], [object_number=14, tag_tid=2, entry_tid=14046], [object_number=13, tag_tid=2, entry_tid=14045]]], state=connected] 2024-09-11T06:17:52.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:52.078 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=12, tag_tid=2, entry_tid=14048' 2024-09-11T06:17:52.078 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=21, tag_tid=3, entry_tid=1' -a 'object_number=21, tag_tid=3, entry_tid=1' = 'object_number=12, tag_tid=2, entry_tid=14048' 2024-09-11T06:17:52.078 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=12, tag_tid=2, entry_tid=14048' '!=' 'object_number=14, tag_tid=2, entry_tid=13938' 2024-09-11T06:17:52.078 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:17:52.078 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=12, tag_tid=2, entry_tid=14048' = 'object_number=14, tag_tid=2, entry_tid=13938' 2024-09-11T06:17:52.078 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=12, tag_tid=2, entry_tid=14048' 2024-09-11T06:17:52.078 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:52.078 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:17:52.078 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:52.078 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=3 2024-09-11T06:17:52.079 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=12, tag_tid=2, entry_tid=14048' 2024-09-11T06:17:52.079 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:17:52.079 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:52.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=2 2024-09-11T06:17:52.081 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:52.081 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:17:52.081 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:52.082 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:17:52.082 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=12, tag_tid=2, entry_tid=14048' 2024-09-11T06:17:52.083 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:17:52.083 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:52.084 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=14048 2024-09-11T06:17:52.084 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 3 = 2 -a 1 -le 14048 2024-09-11T06:17:52.084 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:17:52.084 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:17:52.084 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:17:52.286 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:17:52.286 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:17:52.286 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:17:52.286 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:17:52.286 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:17:52.286 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:17:52.286 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:17:52.286 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:17:52.286 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:17:52.286 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:17:52.286 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:17:52.287 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:17:52.287 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:17:52.287 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:17:52.287 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:17:52.287 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:17:52.287 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:17:52.290 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:17:52.290 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:17:52.290 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:17:52.290 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:17:52.290 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:17:52.290 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:17:52.290 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:17:52.291 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:17:52.291 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:17:52.291 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:17:52.291 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:17:52.291 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:17:52.291 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:17:52.291 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:17:52.291 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:17:52.291 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:17:52.291 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:17:52.291 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:17:52.291 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:17:52.291 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:17:52.291 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:17:52.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:17:52.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:17:52.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:17:52.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:17:52.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:17:52.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:17:52.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:17:52.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:17:52.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:17:52.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:17:52.293 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:17:52.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:52.317 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:52.317 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:52.317 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:17:52.461 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:17:52.462 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:17:52.462 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:52.462 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:52.463 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:52.463 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:17:52.463 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:52.463 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:52.463 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:52.463 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:17:52.463 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:17:52.464 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:17:52.464 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:17:52.466 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:52.466 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:17:52.466 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:52.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 3 2024-09-11T06:17:52.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:17:52.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:17:52.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:17:52.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=12, tag_tid=2, entry_tid=14136], [object_number=15, tag_tid=2, entry_tid=14135], [object_number=14, tag_tid=2, entry_tid=14134], [object_number=13, tag_tid=2, entry_tid=14133]]], state=connected] 2024-09-11T06:17:52.515 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:52.518 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:52.518 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:17:52.518 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:52.518 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:52.518 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:52.518 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:17:52.518 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:52.518 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:52.519 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:52.519 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:17:52.519 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:17:52.519 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:17:52.519 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:17:52.519 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:52.519 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:17:52.520 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:52.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 3 2024-09-11T06:17:52.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:17:52.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:17:52.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:17:52.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=12, tag_tid=2, entry_tid=14136], [object_number=15, tag_tid=2, entry_tid=14135], [object_number=14, tag_tid=2, entry_tid=14134], [object_number=13, tag_tid=2, entry_tid=14133]]], state=connected] 2024-09-11T06:17:52.570 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:52.573 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=12, tag_tid=2, entry_tid=14136' 2024-09-11T06:17:52.573 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=21, tag_tid=3, entry_tid=1' -a 'object_number=21, tag_tid=3, entry_tid=1' = 'object_number=12, tag_tid=2, entry_tid=14136' 2024-09-11T06:17:52.573 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=12, tag_tid=2, entry_tid=14136' '!=' 'object_number=12, tag_tid=2, entry_tid=14048' 2024-09-11T06:17:52.573 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:17:52.573 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=12, tag_tid=2, entry_tid=14136' = 'object_number=12, tag_tid=2, entry_tid=14048' 2024-09-11T06:17:52.573 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=12, tag_tid=2, entry_tid=14136' 2024-09-11T06:17:52.573 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:52.573 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:17:52.573 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:52.574 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=3 2024-09-11T06:17:52.574 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=12, tag_tid=2, entry_tid=14136' 2024-09-11T06:17:52.574 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:17:52.574 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:52.575 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=2 2024-09-11T06:17:52.576 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:52.576 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:17:52.576 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:52.577 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:17:52.577 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=12, tag_tid=2, entry_tid=14136' 2024-09-11T06:17:52.578 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:17:52.578 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:52.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=14136 2024-09-11T06:17:52.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 3 = 2 -a 1 -le 14136 2024-09-11T06:17:52.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:17:52.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:17:52.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:17:52.781 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:17:52.781 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:17:52.781 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:17:52.781 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:17:52.781 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:17:52.781 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:17:52.781 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:17:52.781 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:17:52.781 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:17:52.781 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:17:52.781 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:17:52.782 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:17:52.782 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:17:52.782 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:17:52.782 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:17:52.782 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:17:52.782 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:17:52.785 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:17:52.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:17:52.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:17:52.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:17:52.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:17:52.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:17:52.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:17:52.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:17:52.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:17:52.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:17:52.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:17:52.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:17:52.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:17:52.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:17:52.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:17:52.787 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:17:52.787 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:17:52.787 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:17:52.787 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:17:52.787 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:17:52.787 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:17:52.787 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:17:52.787 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:17:52.787 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:17:52.787 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:17:52.787 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:17:52.787 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:17:52.787 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:17:52.787 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:17:52.788 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:17:52.788 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:17:52.788 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:17:52.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:52.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:52.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:52.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:17:52.952 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:17:52.952 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:17:52.953 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:52.953 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:52.953 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:52.953 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:17:52.953 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:52.953 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:52.953 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:52.953 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:17:52.954 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:17:52.954 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:17:52.954 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:17:52.957 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:52.957 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:52.958 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:17:53.008 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 4 2024-09-11T06:17:53.008 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:17:53.008 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:17:53.008 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:17:53.008 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=14189], [object_number=16, tag_tid=2, entry_tid=14188], [object_number=19, tag_tid=2, entry_tid=14187], [object_number=18, tag_tid=2, entry_tid=14186]]], state=connected] 2024-09-11T06:17:53.011 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:53.014 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:53.014 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:17:53.014 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:53.014 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:53.014 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:53.014 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:17:53.014 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:53.014 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:53.014 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:53.015 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:17:53.015 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:17:53.015 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:17:53.015 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:17:53.015 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:53.015 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:17:53.016 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:53.060 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 4 2024-09-11T06:17:53.060 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:17:53.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:17:53.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:17:53.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=14189], [object_number=16, tag_tid=2, entry_tid=14188], [object_number=19, tag_tid=2, entry_tid=14187], [object_number=18, tag_tid=2, entry_tid=14186]]], state=connected] 2024-09-11T06:17:53.064 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:53.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=17, tag_tid=2, entry_tid=14189' 2024-09-11T06:17:53.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=21, tag_tid=3, entry_tid=1' -a 'object_number=21, tag_tid=3, entry_tid=1' = 'object_number=17, tag_tid=2, entry_tid=14189' 2024-09-11T06:17:53.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=17, tag_tid=2, entry_tid=14189' '!=' 'object_number=12, tag_tid=2, entry_tid=14136' 2024-09-11T06:17:53.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:17:53.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=17, tag_tid=2, entry_tid=14189' = 'object_number=12, tag_tid=2, entry_tid=14136' 2024-09-11T06:17:53.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=17, tag_tid=2, entry_tid=14189' 2024-09-11T06:17:53.067 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:53.067 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:17:53.067 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:53.067 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=3 2024-09-11T06:17:53.068 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=17, tag_tid=2, entry_tid=14189' 2024-09-11T06:17:53.068 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:17:53.068 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:53.069 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=2 2024-09-11T06:17:53.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:53.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:17:53.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:53.071 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:17:53.071 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=17, tag_tid=2, entry_tid=14189' 2024-09-11T06:17:53.071 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:17:53.071 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:53.073 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=14189 2024-09-11T06:17:53.073 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 3 = 2 -a 1 -le 14189 2024-09-11T06:17:53.073 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:17:53.073 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:17:53.073 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:17:53.274 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:17:53.274 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:17:53.274 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:17:53.275 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:17:53.275 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:17:53.275 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:17:53.275 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:17:53.275 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:17:53.275 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:17:53.275 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:17:53.275 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:17:53.275 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:17:53.275 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:17:53.275 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:17:53.275 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:17:53.276 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:17:53.276 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:17:53.280 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:17:53.280 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:17:53.280 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:17:53.280 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:17:53.280 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:17:53.280 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:17:53.280 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:17:53.280 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:17:53.281 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:17:53.281 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:17:53.281 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:17:53.281 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:17:53.281 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:17:53.281 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:17:53.281 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:17:53.281 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:17:53.281 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:17:53.281 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:17:53.281 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:17:53.281 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:17:53.281 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:17:53.281 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:17:53.282 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:17:53.282 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:17:53.282 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:17:53.282 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:17:53.282 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:17:53.282 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:17:53.282 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:17:53.282 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:17:53.282 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:17:53.282 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:17:53.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:53.305 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:53.305 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:53.305 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:17:53.434 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:17:53.435 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:17:53.435 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:53.435 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:53.435 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:53.435 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:17:53.435 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:53.435 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:53.435 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:53.435 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:17:53.435 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:17:53.436 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:17:53.436 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:17:53.437 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:53.437 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:17:53.437 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:53.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 4 2024-09-11T06:17:53.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:17:53.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:17:53.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:17:53.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=18, tag_tid=2, entry_tid=14298], [object_number=17, tag_tid=2, entry_tid=14297], [object_number=16, tag_tid=2, entry_tid=14296], [object_number=19, tag_tid=2, entry_tid=14295]]], state=connected] 2024-09-11T06:17:53.493 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:53.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:53.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:17:53.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:53.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:53.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:53.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:17:53.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:53.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:53.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:53.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:17:53.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:17:53.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:17:53.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:17:53.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:53.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:17:53.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:53.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 4 2024-09-11T06:17:53.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:17:53.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:17:53.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:17:53.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=18, tag_tid=2, entry_tid=14298], [object_number=17, tag_tid=2, entry_tid=14297], [object_number=16, tag_tid=2, entry_tid=14296], [object_number=19, tag_tid=2, entry_tid=14295]]], state=connected] 2024-09-11T06:17:53.545 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:53.547 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=18, tag_tid=2, entry_tid=14298' 2024-09-11T06:17:53.547 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=21, tag_tid=3, entry_tid=1' -a 'object_number=21, tag_tid=3, entry_tid=1' = 'object_number=18, tag_tid=2, entry_tid=14298' 2024-09-11T06:17:53.547 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=18, tag_tid=2, entry_tid=14298' '!=' 'object_number=17, tag_tid=2, entry_tid=14189' 2024-09-11T06:17:53.547 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:17:53.547 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=18, tag_tid=2, entry_tid=14298' = 'object_number=17, tag_tid=2, entry_tid=14189' 2024-09-11T06:17:53.547 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=18, tag_tid=2, entry_tid=14298' 2024-09-11T06:17:53.548 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:53.548 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:17:53.548 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:53.549 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=3 2024-09-11T06:17:53.549 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=18, tag_tid=2, entry_tid=14298' 2024-09-11T06:17:53.549 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:17:53.549 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:53.550 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=2 2024-09-11T06:17:53.551 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:53.551 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:17:53.551 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:53.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:17:53.553 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=18, tag_tid=2, entry_tid=14298' 2024-09-11T06:17:53.553 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:17:53.553 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:53.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=14298 2024-09-11T06:17:53.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 3 = 2 -a 1 -le 14298 2024-09-11T06:17:53.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:17:53.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:17:53.555 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:17:53.756 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:17:53.756 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:17:53.756 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:17:53.756 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:17:53.756 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:17:53.756 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:17:53.756 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:17:53.757 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:17:53.757 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:17:53.757 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:17:53.757 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:17:53.757 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:17:53.757 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:17:53.757 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:17:53.757 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:17:53.757 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:17:53.758 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:17:53.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:17:53.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:17:53.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:17:53.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:17:53.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:17:53.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:17:53.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:17:53.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:17:53.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:17:53.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:17:53.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:17:53.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:17:53.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:17:53.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:17:53.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:17:53.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:17:53.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:17:53.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:17:53.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:17:53.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:17:53.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:17:53.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:17:53.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:17:53.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:17:53.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:17:53.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:17:53.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:17:53.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:17:53.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:17:53.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:17:53.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:17:53.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:17:53.787 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:53.788 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:53.788 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:53.788 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:17:53.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:17:53.915 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:17:53.916 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:53.916 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:53.916 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:53.916 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:17:53.916 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:53.916 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:53.916 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:53.916 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:17:53.916 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:17:53.917 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:17:53.917 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:17:53.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:53.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:17:53.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:53.974 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 4 2024-09-11T06:17:53.974 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:17:53.974 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:17:53.974 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:17:53.974 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=16, tag_tid=2, entry_tid=14416], [object_number=19, tag_tid=2, entry_tid=14415], [object_number=18, tag_tid=2, entry_tid=14414], [object_number=17, tag_tid=2, entry_tid=14413]]], state=connected] 2024-09-11T06:17:53.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:53.979 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:53.980 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:17:53.980 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:53.980 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:53.980 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:53.980 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:17:53.981 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:53.981 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:53.981 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:53.981 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:17:53.981 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:17:53.981 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:17:53.981 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:17:53.982 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:53.982 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:17:53.982 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:54.028 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 4 2024-09-11T06:17:54.028 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:17:54.028 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:17:54.028 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:17:54.028 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=16, tag_tid=2, entry_tid=14416], [object_number=19, tag_tid=2, entry_tid=14415], [object_number=18, tag_tid=2, entry_tid=14414], [object_number=17, tag_tid=2, entry_tid=14413]]], state=connected] 2024-09-11T06:17:54.032 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:54.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=16, tag_tid=2, entry_tid=14416' 2024-09-11T06:17:54.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=21, tag_tid=3, entry_tid=1' -a 'object_number=21, tag_tid=3, entry_tid=1' = 'object_number=16, tag_tid=2, entry_tid=14416' 2024-09-11T06:17:54.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=16, tag_tid=2, entry_tid=14416' '!=' 'object_number=18, tag_tid=2, entry_tid=14298' 2024-09-11T06:17:54.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:17:54.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=16, tag_tid=2, entry_tid=14416' = 'object_number=18, tag_tid=2, entry_tid=14298' 2024-09-11T06:17:54.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=16, tag_tid=2, entry_tid=14416' 2024-09-11T06:17:54.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:54.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:17:54.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:54.035 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=3 2024-09-11T06:17:54.035 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=16, tag_tid=2, entry_tid=14416' 2024-09-11T06:17:54.036 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:17:54.036 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:54.037 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=2 2024-09-11T06:17:54.037 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:54.037 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:17:54.037 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:54.038 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:17:54.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=16, tag_tid=2, entry_tid=14416' 2024-09-11T06:17:54.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:17:54.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:54.040 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=14416 2024-09-11T06:17:54.040 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 3 = 2 -a 1 -le 14416 2024-09-11T06:17:54.041 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:17:54.041 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:17:54.041 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:17:54.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:17:54.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:17:54.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:17:54.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:17:54.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:17:54.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:17:54.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:17:54.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:17:54.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:17:54.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:17:54.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:17:54.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:17:54.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:17:54.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:17:54.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:17:54.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:17:54.244 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:17:54.246 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:17:54.246 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:17:54.246 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:17:54.246 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:17:54.246 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:17:54.246 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:17:54.246 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:17:54.246 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:17:54.246 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:17:54.246 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:17:54.246 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:17:54.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:17:54.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:17:54.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:17:54.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:17:54.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:17:54.248 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:17:54.248 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:17:54.248 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:17:54.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:17:54.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:17:54.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:17:54.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:17:54.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:17:54.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:17:54.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:17:54.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:17:54.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:17:54.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:17:54.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:17:54.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:17:54.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:17:54.272 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:54.273 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:54.273 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:54.273 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:17:54.403 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:17:54.404 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:17:54.404 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:54.404 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:54.405 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:54.405 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:17:54.405 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:54.405 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:54.405 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:54.405 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:17:54.405 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:17:54.405 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:17:54.405 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:17:54.408 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:54.408 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:17:54.408 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:54.454 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 4 2024-09-11T06:17:54.454 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:17:54.454 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:17:54.454 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:17:54.455 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=18, tag_tid=2, entry_tid=14518], [object_number=17, tag_tid=2, entry_tid=14517], [object_number=16, tag_tid=2, entry_tid=14516], [object_number=19, tag_tid=2, entry_tid=14515]]], state=connected] 2024-09-11T06:17:54.457 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:54.458 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:54.459 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:17:54.459 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:54.459 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:54.459 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:54.459 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:17:54.460 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:54.460 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:54.460 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:54.460 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:17:54.460 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:17:54.460 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:17:54.460 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:17:54.461 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:54.461 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:17:54.461 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:54.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 4 2024-09-11T06:17:54.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:17:54.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:17:54.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:17:54.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=18, tag_tid=2, entry_tid=14518], [object_number=17, tag_tid=2, entry_tid=14517], [object_number=16, tag_tid=2, entry_tid=14516], [object_number=19, tag_tid=2, entry_tid=14515]]], state=connected] 2024-09-11T06:17:54.509 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:54.511 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=18, tag_tid=2, entry_tid=14518' 2024-09-11T06:17:54.511 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=21, tag_tid=3, entry_tid=1' -a 'object_number=21, tag_tid=3, entry_tid=1' = 'object_number=18, tag_tid=2, entry_tid=14518' 2024-09-11T06:17:54.511 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=18, tag_tid=2, entry_tid=14518' '!=' 'object_number=16, tag_tid=2, entry_tid=14416' 2024-09-11T06:17:54.511 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:17:54.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=18, tag_tid=2, entry_tid=14518' = 'object_number=16, tag_tid=2, entry_tid=14416' 2024-09-11T06:17:54.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=18, tag_tid=2, entry_tid=14518' 2024-09-11T06:17:54.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:54.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:17:54.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:54.513 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=3 2024-09-11T06:17:54.513 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=18, tag_tid=2, entry_tid=14518' 2024-09-11T06:17:54.513 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:17:54.513 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:54.515 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=2 2024-09-11T06:17:54.515 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:54.515 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:17:54.515 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:54.516 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:17:54.517 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=18, tag_tid=2, entry_tid=14518' 2024-09-11T06:17:54.517 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:17:54.517 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:54.518 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=14518 2024-09-11T06:17:54.518 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 3 = 2 -a 1 -le 14518 2024-09-11T06:17:54.518 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:17:54.518 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:17:54.518 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:17:54.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:17:54.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:17:54.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:17:54.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:17:54.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:17:54.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:17:54.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:17:54.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:17:54.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:17:54.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:17:54.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:17:54.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:17:54.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:17:54.721 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:17:54.721 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:17:54.721 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:17:54.721 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:17:54.724 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:17:54.725 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:17:54.725 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:17:54.725 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:17:54.725 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:17:54.725 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:17:54.725 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:17:54.725 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:17:54.725 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:17:54.725 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:17:54.725 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:17:54.725 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:17:54.726 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:17:54.726 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:17:54.726 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:17:54.726 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:17:54.726 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:17:54.726 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:17:54.726 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:17:54.726 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:17:54.726 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:17:54.726 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:17:54.726 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:17:54.726 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:17:54.726 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:17:54.726 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:17:54.727 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:17:54.727 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:17:54.727 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:17:54.727 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:17:54.727 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:17:54.727 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:17:54.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:54.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:54.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:54.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:17:54.880 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:17:54.882 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:17:54.882 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:54.882 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:54.882 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:54.882 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:17:54.882 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:54.882 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:54.882 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:54.882 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:17:54.882 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:17:54.882 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:17:54.882 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:17:54.884 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:54.884 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:17:54.884 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:54.933 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 4 2024-09-11T06:17:54.933 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:17:54.933 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:17:54.934 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:17:54.934 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=14625], [object_number=16, tag_tid=2, entry_tid=14624], [object_number=19, tag_tid=2, entry_tid=14623], [object_number=18, tag_tid=2, entry_tid=14622]]], state=connected] 2024-09-11T06:17:54.937 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:54.938 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:54.939 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:17:54.940 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:54.940 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:54.940 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:54.940 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:17:54.940 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:54.940 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:54.940 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:54.940 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:17:54.940 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:17:54.940 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:17:54.940 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:17:54.941 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:54.941 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:17:54.941 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:54.986 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 4 2024-09-11T06:17:54.986 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:17:54.987 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:17:54.987 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:17:54.987 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=14625], [object_number=16, tag_tid=2, entry_tid=14624], [object_number=19, tag_tid=2, entry_tid=14623], [object_number=18, tag_tid=2, entry_tid=14622]]], state=connected] 2024-09-11T06:17:54.990 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:54.992 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=17, tag_tid=2, entry_tid=14625' 2024-09-11T06:17:54.992 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=21, tag_tid=3, entry_tid=1' -a 'object_number=21, tag_tid=3, entry_tid=1' = 'object_number=17, tag_tid=2, entry_tid=14625' 2024-09-11T06:17:54.992 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=17, tag_tid=2, entry_tid=14625' '!=' 'object_number=18, tag_tid=2, entry_tid=14518' 2024-09-11T06:17:54.992 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:17:54.992 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=17, tag_tid=2, entry_tid=14625' = 'object_number=18, tag_tid=2, entry_tid=14518' 2024-09-11T06:17:54.992 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=17, tag_tid=2, entry_tid=14625' 2024-09-11T06:17:54.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:54.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:17:54.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:54.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=3 2024-09-11T06:17:54.994 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=17, tag_tid=2, entry_tid=14625' 2024-09-11T06:17:54.994 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:17:54.994 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:54.995 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=2 2024-09-11T06:17:54.995 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:54.996 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:17:54.996 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:54.997 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:17:54.997 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=17, tag_tid=2, entry_tid=14625' 2024-09-11T06:17:54.997 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:17:54.997 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:54.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=14625 2024-09-11T06:17:54.999 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 3 = 2 -a 1 -le 14625 2024-09-11T06:17:54.999 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:17:54.999 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:17:54.999 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:17:55.200 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:17:55.201 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:17:55.201 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:17:55.201 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:17:55.201 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:17:55.201 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:17:55.201 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:17:55.201 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:17:55.201 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:17:55.201 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:17:55.201 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:17:55.201 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:17:55.201 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:17:55.202 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:17:55.202 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:17:55.202 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:17:55.202 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:17:55.203 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:17:55.203 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:17:55.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:17:55.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:17:55.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:17:55.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:17:55.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:17:55.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:17:55.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:17:55.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:17:55.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:17:55.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:17:55.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:17:55.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:17:55.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:17:55.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:17:55.206 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:17:55.206 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:17:55.206 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:17:55.206 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:17:55.206 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:17:55.206 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:17:55.206 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:17:55.206 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:17:55.206 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:17:55.206 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:17:55.206 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:17:55.206 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:17:55.206 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:17:55.207 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:17:55.207 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:17:55.207 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:17:55.231 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:55.232 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:55.232 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:55.232 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:17:55.358 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:17:55.359 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:17:55.359 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:55.359 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:55.359 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:55.359 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:17:55.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:55.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:55.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:55.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:17:55.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:17:55.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:17:55.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:17:55.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:55.362 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:17:55.362 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:55.412 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 4 2024-09-11T06:17:55.412 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:17:55.412 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:17:55.412 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:17:55.412 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=19, tag_tid=2, entry_tid=14731], [object_number=18, tag_tid=2, entry_tid=14730], [object_number=17, tag_tid=2, entry_tid=14729], [object_number=16, tag_tid=2, entry_tid=14728]]], state=connected] 2024-09-11T06:17:55.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:55.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:55.418 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:17:55.418 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:55.418 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:55.418 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:55.418 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:17:55.418 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:55.418 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:55.418 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:55.418 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:17:55.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:17:55.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:17:55.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:17:55.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:55.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:17:55.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:55.466 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 4 2024-09-11T06:17:55.466 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:17:55.466 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:17:55.466 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:17:55.466 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=19, tag_tid=2, entry_tid=14731], [object_number=18, tag_tid=2, entry_tid=14730], [object_number=17, tag_tid=2, entry_tid=14729], [object_number=16, tag_tid=2, entry_tid=14728]]], state=connected] 2024-09-11T06:17:55.469 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:55.471 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=19, tag_tid=2, entry_tid=14731' 2024-09-11T06:17:55.471 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=21, tag_tid=3, entry_tid=1' -a 'object_number=21, tag_tid=3, entry_tid=1' = 'object_number=19, tag_tid=2, entry_tid=14731' 2024-09-11T06:17:55.471 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=19, tag_tid=2, entry_tid=14731' '!=' 'object_number=17, tag_tid=2, entry_tid=14625' 2024-09-11T06:17:55.471 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:17:55.471 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=19, tag_tid=2, entry_tid=14731' = 'object_number=17, tag_tid=2, entry_tid=14625' 2024-09-11T06:17:55.472 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=19, tag_tid=2, entry_tid=14731' 2024-09-11T06:17:55.472 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:55.472 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:17:55.472 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:55.473 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=3 2024-09-11T06:17:55.473 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=19, tag_tid=2, entry_tid=14731' 2024-09-11T06:17:55.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:17:55.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:55.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=2 2024-09-11T06:17:55.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:55.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:17:55.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:55.476 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:17:55.477 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=19, tag_tid=2, entry_tid=14731' 2024-09-11T06:17:55.477 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:17:55.477 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:55.478 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=14731 2024-09-11T06:17:55.478 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 3 = 2 -a 1 -le 14731 2024-09-11T06:17:55.478 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:17:55.478 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:17:55.478 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:17:55.680 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:17:55.680 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:17:55.680 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:17:55.680 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:17:55.680 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:17:55.680 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:17:55.681 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:17:55.681 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:17:55.681 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:17:55.681 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:17:55.681 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:17:55.681 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:17:55.681 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:17:55.681 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:17:55.681 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:17:55.682 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:17:55.682 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:17:55.686 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:17:55.686 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:17:55.686 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:17:55.686 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:17:55.686 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:17:55.686 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:17:55.686 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:17:55.686 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:17:55.686 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:17:55.686 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:17:55.686 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:17:55.686 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:17:55.686 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:17:55.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:17:55.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:17:55.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:17:55.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:17:55.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:17:55.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:17:55.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:17:55.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:17:55.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:17:55.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:17:55.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:17:55.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:17:55.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:17:55.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:17:55.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:17:55.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:17:55.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:17:55.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:17:55.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:17:55.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:55.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:55.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:55.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:17:55.845 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:17:55.846 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:17:55.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:55.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:55.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:55.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:17:55.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:55.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:55.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:55.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:17:55.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:17:55.848 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:17:55.848 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:17:55.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:55.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:17:55.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:55.899 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 4 2024-09-11T06:17:55.899 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:17:55.899 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:17:55.899 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:17:55.899 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=16, tag_tid=2, entry_tid=14836], [object_number=19, tag_tid=2, entry_tid=14835], [object_number=18, tag_tid=2, entry_tid=14834], [object_number=17, tag_tid=2, entry_tid=14833]]], state=connected] 2024-09-11T06:17:55.902 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:55.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:55.905 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:17:55.905 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:55.905 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:55.905 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:55.905 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:17:55.905 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:55.905 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:55.905 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:55.905 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:17:55.906 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:17:55.906 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:17:55.906 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:17:55.906 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:55.906 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:17:55.907 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:55.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 4 2024-09-11T06:17:55.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:17:55.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:17:55.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:17:55.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=16, tag_tid=2, entry_tid=14836], [object_number=19, tag_tid=2, entry_tid=14835], [object_number=18, tag_tid=2, entry_tid=14834], [object_number=17, tag_tid=2, entry_tid=14833]]], state=connected] 2024-09-11T06:17:55.954 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:55.956 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=16, tag_tid=2, entry_tid=14836' 2024-09-11T06:17:55.956 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=21, tag_tid=3, entry_tid=1' -a 'object_number=21, tag_tid=3, entry_tid=1' = 'object_number=16, tag_tid=2, entry_tid=14836' 2024-09-11T06:17:55.957 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=16, tag_tid=2, entry_tid=14836' '!=' 'object_number=19, tag_tid=2, entry_tid=14731' 2024-09-11T06:17:55.957 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:17:55.957 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=16, tag_tid=2, entry_tid=14836' = 'object_number=19, tag_tid=2, entry_tid=14731' 2024-09-11T06:17:55.957 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=16, tag_tid=2, entry_tid=14836' 2024-09-11T06:17:55.957 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:55.957 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:17:55.957 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:55.958 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=3 2024-09-11T06:17:55.958 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=16, tag_tid=2, entry_tid=14836' 2024-09-11T06:17:55.958 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:17:55.958 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:55.959 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=2 2024-09-11T06:17:55.960 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:55.960 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:17:55.960 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:55.961 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:17:55.962 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=16, tag_tid=2, entry_tid=14836' 2024-09-11T06:17:55.962 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:17:55.962 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:55.963 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=14836 2024-09-11T06:17:55.963 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 3 = 2 -a 1 -le 14836 2024-09-11T06:17:55.963 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:17:55.963 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:17:55.963 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:17:56.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:17:56.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:17:56.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:17:56.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:17:56.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:17:56.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:17:56.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:17:56.166 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:17:56.166 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:17:56.166 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:17:56.166 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:17:56.166 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:17:56.166 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:17:56.166 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:17:56.166 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:17:56.167 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:17:56.167 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:17:56.170 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:17:56.171 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:17:56.171 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:17:56.171 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:17:56.171 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:17:56.171 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:17:56.171 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:17:56.171 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:17:56.171 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:17:56.171 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:17:56.171 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:17:56.171 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:17:56.171 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:17:56.171 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:17:56.172 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:17:56.172 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:17:56.172 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:17:56.172 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:17:56.172 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:17:56.172 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:17:56.172 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:17:56.172 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:17:56.172 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:17:56.172 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:17:56.172 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:17:56.172 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:17:56.172 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:17:56.172 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:17:56.172 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:17:56.173 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:17:56.173 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:17:56.173 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:17:56.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:56.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:56.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:56.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:17:56.328 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:17:56.329 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:17:56.329 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:56.329 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:56.329 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:56.329 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:17:56.330 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:56.330 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:56.330 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:56.330 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:17:56.330 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:17:56.330 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:17:56.331 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:17:56.332 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:56.332 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:17:56.332 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:56.379 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 4 2024-09-11T06:17:56.379 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:17:56.379 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:17:56.379 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:17:56.379 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=16, tag_tid=2, entry_tid=14948], [object_number=19, tag_tid=2, entry_tid=14947], [object_number=18, tag_tid=2, entry_tid=14946], [object_number=17, tag_tid=2, entry_tid=14945]]], state=connected] 2024-09-11T06:17:56.382 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:56.383 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:56.385 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:17:56.385 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:56.385 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:56.385 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:56.385 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:17:56.385 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:56.385 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:56.385 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:56.385 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:17:56.385 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:17:56.385 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:17:56.385 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:17:56.386 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:56.386 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:17:56.386 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:56.430 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 4 2024-09-11T06:17:56.431 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:17:56.431 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:17:56.431 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:17:56.431 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=16, tag_tid=2, entry_tid=14948], [object_number=19, tag_tid=2, entry_tid=14947], [object_number=18, tag_tid=2, entry_tid=14946], [object_number=17, tag_tid=2, entry_tid=14945]]], state=connected] 2024-09-11T06:17:56.434 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:56.436 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=16, tag_tid=2, entry_tid=14948' 2024-09-11T06:17:56.436 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=21, tag_tid=3, entry_tid=1' -a 'object_number=21, tag_tid=3, entry_tid=1' = 'object_number=16, tag_tid=2, entry_tid=14948' 2024-09-11T06:17:56.436 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=16, tag_tid=2, entry_tid=14948' '!=' 'object_number=16, tag_tid=2, entry_tid=14836' 2024-09-11T06:17:56.436 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:17:56.436 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=16, tag_tid=2, entry_tid=14948' = 'object_number=16, tag_tid=2, entry_tid=14836' 2024-09-11T06:17:56.437 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=16, tag_tid=2, entry_tid=14948' 2024-09-11T06:17:56.437 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:56.437 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:17:56.437 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:56.437 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=3 2024-09-11T06:17:56.438 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=16, tag_tid=2, entry_tid=14948' 2024-09-11T06:17:56.438 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:17:56.438 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:56.439 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=2 2024-09-11T06:17:56.440 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:56.440 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:17:56.440 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:56.441 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:17:56.441 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=16, tag_tid=2, entry_tid=14948' 2024-09-11T06:17:56.441 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:17:56.441 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:56.443 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=14948 2024-09-11T06:17:56.443 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 3 = 2 -a 1 -le 14948 2024-09-11T06:17:56.443 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:17:56.443 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:17:56.443 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:17:56.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:17:56.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:17:56.645 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:17:56.645 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:17:56.645 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:17:56.645 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:17:56.645 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:17:56.645 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:17:56.645 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:17:56.645 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:17:56.645 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:17:56.645 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:17:56.645 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:17:56.645 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:17:56.646 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:17:56.646 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:17:56.646 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:17:56.650 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:17:56.650 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:17:56.650 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:17:56.650 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:17:56.650 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:17:56.650 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:17:56.650 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:17:56.650 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:17:56.650 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:17:56.650 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:17:56.650 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:17:56.650 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:17:56.651 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:17:56.651 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:17:56.651 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:17:56.651 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:17:56.651 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:17:56.651 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:17:56.651 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:17:56.651 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:17:56.651 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:17:56.651 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:17:56.651 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:17:56.651 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:17:56.651 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:17:56.652 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:17:56.652 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:17:56.652 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:17:56.652 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:17:56.652 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:17:56.652 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:17:56.652 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:17:56.676 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:56.676 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:56.676 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:56.676 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:17:56.804 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:17:56.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:17:56.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:56.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:56.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:56.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:17:56.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:56.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:56.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:56.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:17:56.806 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:17:56.806 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:17:56.806 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:17:56.808 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:56.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:17:56.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:56.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 4 2024-09-11T06:17:56.854 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:17:56.854 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:17:56.854 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:17:56.854 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=19, tag_tid=2, entry_tid=15047], [object_number=18, tag_tid=2, entry_tid=15046], [object_number=17, tag_tid=2, entry_tid=15045], [object_number=16, tag_tid=2, entry_tid=15044]]], state=connected] 2024-09-11T06:17:56.857 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:56.859 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:56.860 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:17:56.860 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:56.860 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:56.860 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:56.860 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:17:56.860 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:56.860 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:56.860 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:56.860 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:17:56.860 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:17:56.860 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:17:56.860 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:17:56.861 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:56.861 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:17:56.861 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:56.905 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 4 2024-09-11T06:17:56.905 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:17:56.905 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:17:56.906 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:17:56.906 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=19, tag_tid=2, entry_tid=15047], [object_number=18, tag_tid=2, entry_tid=15046], [object_number=17, tag_tid=2, entry_tid=15045], [object_number=16, tag_tid=2, entry_tid=15044]]], state=connected] 2024-09-11T06:17:56.909 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:56.910 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=19, tag_tid=2, entry_tid=15047' 2024-09-11T06:17:56.911 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=21, tag_tid=3, entry_tid=1' -a 'object_number=21, tag_tid=3, entry_tid=1' = 'object_number=19, tag_tid=2, entry_tid=15047' 2024-09-11T06:17:56.911 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=19, tag_tid=2, entry_tid=15047' '!=' 'object_number=16, tag_tid=2, entry_tid=14948' 2024-09-11T06:17:56.911 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:17:56.911 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=19, tag_tid=2, entry_tid=15047' = 'object_number=16, tag_tid=2, entry_tid=14948' 2024-09-11T06:17:56.911 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=19, tag_tid=2, entry_tid=15047' 2024-09-11T06:17:56.911 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:56.911 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:17:56.911 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:56.911 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=3 2024-09-11T06:17:56.912 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=19, tag_tid=2, entry_tid=15047' 2024-09-11T06:17:56.912 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:17:56.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:56.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=2 2024-09-11T06:17:56.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:56.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:17:56.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:56.915 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:17:56.916 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=19, tag_tid=2, entry_tid=15047' 2024-09-11T06:17:56.916 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:17:56.916 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:56.917 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=15047 2024-09-11T06:17:56.917 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 3 = 2 -a 1 -le 15047 2024-09-11T06:17:56.917 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:17:56.917 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:17:56.917 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:17:57.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:17:57.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:17:57.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:17:57.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:17:57.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:17:57.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:17:57.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:17:57.120 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:17:57.120 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:17:57.120 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:17:57.120 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:17:57.120 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:17:57.120 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:17:57.120 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:17:57.120 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:17:57.121 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:17:57.121 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:17:57.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:17:57.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:17:57.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:17:57.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:17:57.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:17:57.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:17:57.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:17:57.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:17:57.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:17:57.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:17:57.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:17:57.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:17:57.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:17:57.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:17:57.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:17:57.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:17:57.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:17:57.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:17:57.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:17:57.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:17:57.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:17:57.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:17:57.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:17:57.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:17:57.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:17:57.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:17:57.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:17:57.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:17:57.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:17:57.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:17:57.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:17:57.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:17:57.150 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:57.150 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:57.150 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:57.150 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:17:57.277 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:17:57.279 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:17:57.279 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:57.279 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:57.279 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:57.279 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:17:57.279 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:57.279 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:57.279 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:57.279 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:17:57.280 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:17:57.280 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:17:57.280 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:17:57.281 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:57.281 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:17:57.281 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:57.335 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 4 2024-09-11T06:17:57.335 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:17:57.335 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:17:57.335 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:17:57.335 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=19, tag_tid=2, entry_tid=15155], [object_number=18, tag_tid=2, entry_tid=15154], [object_number=17, tag_tid=2, entry_tid=15153], [object_number=16, tag_tid=2, entry_tid=15152]]], state=connected] 2024-09-11T06:17:57.338 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:57.339 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:57.340 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:17:57.340 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:57.341 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:57.341 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:57.341 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:17:57.341 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:57.341 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:57.341 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:57.341 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:17:57.341 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:17:57.341 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:17:57.341 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:17:57.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:57.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:17:57.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:57.387 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 4 2024-09-11T06:17:57.387 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:17:57.387 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:17:57.387 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:17:57.387 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=19, tag_tid=2, entry_tid=15155], [object_number=18, tag_tid=2, entry_tid=15154], [object_number=17, tag_tid=2, entry_tid=15153], [object_number=16, tag_tid=2, entry_tid=15152]]], state=connected] 2024-09-11T06:17:57.391 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:57.392 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=19, tag_tid=2, entry_tid=15155' 2024-09-11T06:17:57.392 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=21, tag_tid=3, entry_tid=1' -a 'object_number=21, tag_tid=3, entry_tid=1' = 'object_number=19, tag_tid=2, entry_tid=15155' 2024-09-11T06:17:57.393 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=19, tag_tid=2, entry_tid=15155' '!=' 'object_number=19, tag_tid=2, entry_tid=15047' 2024-09-11T06:17:57.393 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:17:57.393 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=19, tag_tid=2, entry_tid=15155' = 'object_number=19, tag_tid=2, entry_tid=15047' 2024-09-11T06:17:57.393 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=19, tag_tid=2, entry_tid=15155' 2024-09-11T06:17:57.393 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:57.393 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:17:57.394 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:57.394 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=3 2024-09-11T06:17:57.394 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=19, tag_tid=2, entry_tid=15155' 2024-09-11T06:17:57.395 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:17:57.395 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:57.395 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=2 2024-09-11T06:17:57.396 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:57.396 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:17:57.396 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:57.397 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:17:57.398 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=19, tag_tid=2, entry_tid=15155' 2024-09-11T06:17:57.398 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:17:57.398 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:57.399 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=15155 2024-09-11T06:17:57.399 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 3 = 2 -a 1 -le 15155 2024-09-11T06:17:57.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:17:57.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:17:57.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:17:57.601 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:17:57.601 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:17:57.601 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:17:57.602 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:17:57.602 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:17:57.602 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:17:57.602 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:17:57.602 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:17:57.602 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:17:57.602 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:17:57.602 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:17:57.602 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:17:57.602 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:17:57.602 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:17:57.602 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:17:57.603 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:17:57.603 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:17:57.606 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:17:57.606 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:17:57.606 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:17:57.606 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:17:57.607 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:17:57.607 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:17:57.607 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:17:57.607 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:17:57.607 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:17:57.607 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:17:57.607 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:17:57.607 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:17:57.607 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:17:57.607 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:17:57.607 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:17:57.607 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:17:57.607 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:17:57.607 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:17:57.608 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:17:57.608 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:17:57.608 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:17:57.608 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:17:57.608 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:17:57.608 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:17:57.608 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:17:57.608 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:17:57.608 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:17:57.608 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:17:57.608 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:17:57.608 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:17:57.608 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:17:57.609 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:17:57.634 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:57.635 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:57.635 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:57.635 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:17:57.774 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:17:57.775 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:17:57.775 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:57.776 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:57.776 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:57.776 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:17:57.776 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:57.776 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:57.776 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:57.776 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:17:57.776 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:17:57.777 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:17:57.777 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:17:57.779 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:57.779 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:17:57.779 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:57.823 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 4 2024-09-11T06:17:57.823 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:17:57.823 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:17:57.823 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:17:57.823 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=16, tag_tid=2, entry_tid=15264], [object_number=19, tag_tid=2, entry_tid=15263], [object_number=18, tag_tid=2, entry_tid=15262], [object_number=17, tag_tid=2, entry_tid=15261]]], state=connected] 2024-09-11T06:17:57.826 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:57.828 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:57.829 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:17:57.829 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:57.829 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:57.829 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:57.829 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:17:57.830 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:57.830 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:57.830 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:57.830 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:17:57.830 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:17:57.830 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:17:57.830 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:17:57.831 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:57.831 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:17:57.831 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:57.875 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 4 2024-09-11T06:17:57.875 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:17:57.875 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:17:57.875 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:17:57.875 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=16, tag_tid=2, entry_tid=15264], [object_number=19, tag_tid=2, entry_tid=15263], [object_number=18, tag_tid=2, entry_tid=15262], [object_number=17, tag_tid=2, entry_tid=15261]]], state=connected] 2024-09-11T06:17:57.878 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:57.880 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=16, tag_tid=2, entry_tid=15264' 2024-09-11T06:17:57.880 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=21, tag_tid=3, entry_tid=1' -a 'object_number=21, tag_tid=3, entry_tid=1' = 'object_number=16, tag_tid=2, entry_tid=15264' 2024-09-11T06:17:57.880 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=16, tag_tid=2, entry_tid=15264' '!=' 'object_number=19, tag_tid=2, entry_tid=15155' 2024-09-11T06:17:57.880 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:17:57.880 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=16, tag_tid=2, entry_tid=15264' = 'object_number=19, tag_tid=2, entry_tid=15155' 2024-09-11T06:17:57.880 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=16, tag_tid=2, entry_tid=15264' 2024-09-11T06:17:57.881 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:57.881 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:17:57.881 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:57.881 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=3 2024-09-11T06:17:57.882 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=16, tag_tid=2, entry_tid=15264' 2024-09-11T06:17:57.882 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:17:57.882 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:57.883 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=2 2024-09-11T06:17:57.883 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:57.883 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:17:57.884 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:57.884 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:17:57.885 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=16, tag_tid=2, entry_tid=15264' 2024-09-11T06:17:57.885 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:17:57.885 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:57.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=15264 2024-09-11T06:17:57.887 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 3 = 2 -a 1 -le 15264 2024-09-11T06:17:57.887 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:17:57.887 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:17:57.887 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:17:58.088 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:17:58.088 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:17:58.088 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:17:58.089 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:17:58.089 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:17:58.089 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:17:58.089 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:17:58.089 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:17:58.089 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:17:58.089 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:17:58.089 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:17:58.089 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:17:58.089 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:17:58.089 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:17:58.089 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:17:58.090 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:17:58.090 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:17:58.093 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:17:58.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:17:58.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:17:58.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:17:58.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:17:58.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:17:58.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:17:58.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:17:58.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:17:58.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:17:58.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:17:58.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:17:58.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:17:58.095 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:17:58.095 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:17:58.095 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:17:58.095 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:17:58.095 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:17:58.095 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:17:58.095 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:17:58.095 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:17:58.095 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:17:58.095 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:17:58.095 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:17:58.095 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:17:58.095 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:17:58.095 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:17:58.096 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:17:58.096 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:17:58.096 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:17:58.096 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:17:58.096 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:17:58.120 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:58.120 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:58.121 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:58.121 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:17:58.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:17:58.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:17:58.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:58.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:58.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:58.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:17:58.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:58.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:58.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:58.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:17:58.250 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:17:58.250 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:17:58.250 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:17:58.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:58.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:17:58.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:58.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 4 2024-09-11T06:17:58.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:17:58.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:17:58.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:17:58.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=16, tag_tid=2, entry_tid=15380], [object_number=19, tag_tid=2, entry_tid=15379], [object_number=18, tag_tid=2, entry_tid=15378], [object_number=17, tag_tid=2, entry_tid=15377]]], state=connected] 2024-09-11T06:17:58.305 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:58.307 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:58.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:17:58.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:58.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:58.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:58.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:17:58.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:58.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:58.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:58.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:17:58.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:17:58.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:17:58.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:17:58.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:58.310 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:17:58.310 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:58.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 4 2024-09-11T06:17:58.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:17:58.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:17:58.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:17:58.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=16, tag_tid=2, entry_tid=15380], [object_number=19, tag_tid=2, entry_tid=15379], [object_number=18, tag_tid=2, entry_tid=15378], [object_number=17, tag_tid=2, entry_tid=15377]]], state=connected] 2024-09-11T06:17:58.359 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:58.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=16, tag_tid=2, entry_tid=15380' 2024-09-11T06:17:58.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=21, tag_tid=3, entry_tid=1' -a 'object_number=21, tag_tid=3, entry_tid=1' = 'object_number=16, tag_tid=2, entry_tid=15380' 2024-09-11T06:17:58.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=16, tag_tid=2, entry_tid=15380' '!=' 'object_number=16, tag_tid=2, entry_tid=15264' 2024-09-11T06:17:58.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:17:58.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=16, tag_tid=2, entry_tid=15380' = 'object_number=16, tag_tid=2, entry_tid=15264' 2024-09-11T06:17:58.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=16, tag_tid=2, entry_tid=15380' 2024-09-11T06:17:58.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:58.362 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:17:58.362 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:58.362 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=3 2024-09-11T06:17:58.363 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=16, tag_tid=2, entry_tid=15380' 2024-09-11T06:17:58.363 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:17:58.363 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:58.364 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=2 2024-09-11T06:17:58.364 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:58.364 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:17:58.365 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:58.365 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:17:58.366 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=16, tag_tid=2, entry_tid=15380' 2024-09-11T06:17:58.366 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:17:58.366 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:58.367 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=15380 2024-09-11T06:17:58.368 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 3 = 2 -a 1 -le 15380 2024-09-11T06:17:58.368 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:17:58.368 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:17:58.368 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:17:58.569 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:17:58.570 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:17:58.570 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:17:58.570 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:17:58.570 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:17:58.570 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:17:58.570 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:17:58.570 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:17:58.570 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:17:58.570 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:17:58.570 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:17:58.570 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:17:58.570 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:17:58.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:17:58.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:17:58.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:17:58.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:17:58.574 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:17:58.574 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:17:58.574 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:17:58.574 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:17:58.574 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:17:58.574 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:17:58.574 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:17:58.574 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:17:58.574 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:17:58.574 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:17:58.575 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:17:58.575 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:17:58.575 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:17:58.575 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:17:58.575 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:17:58.575 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:17:58.575 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:17:58.575 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:17:58.575 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:17:58.575 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:17:58.575 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:17:58.575 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:17:58.575 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:17:58.575 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:17:58.576 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:17:58.576 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:17:58.576 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:17:58.576 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:17:58.576 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:17:58.576 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:17:58.576 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:17:58.576 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:17:58.600 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:58.600 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:58.601 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:58.601 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:17:58.730 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:17:58.731 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:17:58.731 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:58.731 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:58.731 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:58.731 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:17:58.731 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:58.731 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:58.731 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:58.732 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:17:58.732 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:17:58.732 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:17:58.732 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:17:58.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:58.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:17:58.734 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:58.779 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 4 2024-09-11T06:17:58.779 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:17:58.779 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:17:58.779 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:17:58.779 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=19, tag_tid=2, entry_tid=15499], [object_number=18, tag_tid=2, entry_tid=15498], [object_number=17, tag_tid=2, entry_tid=15497], [object_number=16, tag_tid=2, entry_tid=15496]]], state=connected] 2024-09-11T06:17:58.831 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:58.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:58.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:17:58.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:58.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:58.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:58.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:17:58.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:58.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:58.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:58.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:17:58.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:17:58.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:17:58.835 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:17:58.836 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:58.836 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:17:58.836 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:58.887 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 4 2024-09-11T06:17:58.887 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:17:58.887 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:17:58.887 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:17:58.887 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=19, tag_tid=2, entry_tid=15499], [object_number=18, tag_tid=2, entry_tid=15498], [object_number=17, tag_tid=2, entry_tid=15497], [object_number=16, tag_tid=2, entry_tid=15496]]], state=connected] 2024-09-11T06:17:58.925 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:58.926 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=19, tag_tid=2, entry_tid=15499' 2024-09-11T06:17:58.926 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=21, tag_tid=3, entry_tid=1' -a 'object_number=21, tag_tid=3, entry_tid=1' = 'object_number=19, tag_tid=2, entry_tid=15499' 2024-09-11T06:17:58.926 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=19, tag_tid=2, entry_tid=15499' '!=' 'object_number=16, tag_tid=2, entry_tid=15380' 2024-09-11T06:17:58.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:17:58.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=19, tag_tid=2, entry_tid=15499' = 'object_number=16, tag_tid=2, entry_tid=15380' 2024-09-11T06:17:58.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=19, tag_tid=2, entry_tid=15499' 2024-09-11T06:17:58.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:58.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:17:58.928 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:58.929 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=3 2024-09-11T06:17:58.929 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=19, tag_tid=2, entry_tid=15499' 2024-09-11T06:17:58.929 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:17:58.929 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:58.931 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=2 2024-09-11T06:17:58.931 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:58.931 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:17:58.931 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:58.932 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:17:58.933 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=19, tag_tid=2, entry_tid=15499' 2024-09-11T06:17:58.933 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:17:58.933 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:58.935 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=15499 2024-09-11T06:17:58.935 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 3 = 2 -a 1 -le 15499 2024-09-11T06:17:58.935 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:17:58.935 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:17:58.935 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:17:59.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:17:59.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:17:59.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:17:59.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:17:59.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:17:59.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:17:59.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:17:59.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:17:59.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:17:59.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:17:59.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:17:59.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:17:59.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:17:59.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:17:59.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:17:59.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:17:59.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:17:59.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:17:59.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:17:59.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:17:59.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:17:59.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:17:59.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:17:59.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:17:59.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:17:59.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:17:59.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:17:59.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:17:59.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:17:59.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:17:59.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:17:59.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:17:59.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:17:59.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:17:59.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:17:59.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:17:59.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:17:59.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:17:59.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:17:59.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:17:59.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:17:59.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:17:59.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:17:59.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:17:59.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:17:59.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:17:59.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:17:59.143 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:17:59.143 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:17:59.168 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:59.168 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:59.168 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:59.168 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:17:59.293 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:17:59.294 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:17:59.294 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:59.294 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:59.294 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:59.294 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:17:59.294 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:59.294 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:59.294 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:59.294 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:17:59.295 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:17:59.295 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:17:59.295 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:17:59.296 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:59.297 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:17:59.297 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:59.347 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 4 2024-09-11T06:17:59.347 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:17:59.347 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:17:59.347 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:17:59.347 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=15633], [object_number=16, tag_tid=2, entry_tid=15632], [object_number=19, tag_tid=2, entry_tid=15631], [object_number=18, tag_tid=2, entry_tid=15630]]], state=connected] 2024-09-11T06:17:59.351 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:59.352 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:59.353 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:17:59.353 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:59.353 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:59.353 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:59.353 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:17:59.353 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:59.354 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:59.354 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:59.354 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:17:59.354 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:17:59.354 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:17:59.354 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:17:59.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:59.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:17:59.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:59.602 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 4 2024-09-11T06:17:59.603 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:17:59.603 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:17:59.603 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:17:59.603 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=15633], [object_number=16, tag_tid=2, entry_tid=15632], [object_number=19, tag_tid=2, entry_tid=15631], [object_number=18, tag_tid=2, entry_tid=15630]]], state=connected] 2024-09-11T06:17:59.610 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:59.612 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=17, tag_tid=2, entry_tid=15633' 2024-09-11T06:17:59.612 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=21, tag_tid=3, entry_tid=1' -a 'object_number=21, tag_tid=3, entry_tid=1' = 'object_number=17, tag_tid=2, entry_tid=15633' 2024-09-11T06:17:59.612 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=17, tag_tid=2, entry_tid=15633' '!=' 'object_number=19, tag_tid=2, entry_tid=15499' 2024-09-11T06:17:59.612 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:17:59.612 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=17, tag_tid=2, entry_tid=15633' = 'object_number=19, tag_tid=2, entry_tid=15499' 2024-09-11T06:17:59.613 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=17, tag_tid=2, entry_tid=15633' 2024-09-11T06:17:59.613 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:59.613 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:17:59.613 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:59.616 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=3 2024-09-11T06:17:59.617 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=17, tag_tid=2, entry_tid=15633' 2024-09-11T06:17:59.617 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:17:59.617 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:59.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=2 2024-09-11T06:17:59.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:17:59.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:17:59.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:59.621 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:17:59.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=17, tag_tid=2, entry_tid=15633' 2024-09-11T06:17:59.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:17:59.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:17:59.625 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=15633 2024-09-11T06:17:59.625 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 3 = 2 -a 1 -le 15633 2024-09-11T06:17:59.625 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:17:59.625 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:17:59.625 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:17:59.826 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:17:59.826 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:17:59.827 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:17:59.827 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:17:59.827 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:17:59.827 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:17:59.827 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:17:59.827 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:17:59.827 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:17:59.827 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:17:59.827 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:17:59.827 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:17:59.827 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:17:59.827 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:17:59.828 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:17:59.828 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:17:59.828 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:17:59.831 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:17:59.831 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:17:59.831 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:17:59.831 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:17:59.831 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:17:59.831 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:17:59.831 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:17:59.831 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:17:59.831 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:17:59.831 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:17:59.831 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:17:59.831 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:17:59.831 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:17:59.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:17:59.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:17:59.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:17:59.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:17:59.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:17:59.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:17:59.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:17:59.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:17:59.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:17:59.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:17:59.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:17:59.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:17:59.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:17:59.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:17:59.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:17:59.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:17:59.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:17:59.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:17:59.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:17:59.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:59.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:59.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:17:59.857 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:17:59.986 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:17:59.988 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:17:59.988 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:59.988 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:59.988 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:59.988 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:17:59.988 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:17:59.988 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:17:59.988 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:17:59.988 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:17:59.989 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:17:59.989 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:17:59.989 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:17:59.990 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:17:59.990 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:17:59.991 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:00.036 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 4 2024-09-11T06:18:00.036 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:18:00.036 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:18:00.036 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:18:00.036 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=15797], [object_number=16, tag_tid=2, entry_tid=15796], [object_number=19, tag_tid=2, entry_tid=15795], [object_number=18, tag_tid=2, entry_tid=15794]]], state=connected] 2024-09-11T06:18:00.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:00.041 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:18:00.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:18:00.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:00.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:00.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:00.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:18:00.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:00.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:00.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:00.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:18:00.043 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:18:00.043 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:18:00.043 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:18:00.043 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:00.043 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:18:00.043 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:00.089 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 4 2024-09-11T06:18:00.089 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:18:00.089 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:18:00.089 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:18:00.089 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=15797], [object_number=16, tag_tid=2, entry_tid=15796], [object_number=19, tag_tid=2, entry_tid=15795], [object_number=18, tag_tid=2, entry_tid=15794]]], state=connected] 2024-09-11T06:18:00.092 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:00.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=17, tag_tid=2, entry_tid=15797' 2024-09-11T06:18:00.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=21, tag_tid=3, entry_tid=1' -a 'object_number=21, tag_tid=3, entry_tid=1' = 'object_number=17, tag_tid=2, entry_tid=15797' 2024-09-11T06:18:00.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=17, tag_tid=2, entry_tid=15797' '!=' 'object_number=17, tag_tid=2, entry_tid=15633' 2024-09-11T06:18:00.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:18:00.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=17, tag_tid=2, entry_tid=15797' = 'object_number=17, tag_tid=2, entry_tid=15633' 2024-09-11T06:18:00.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=17, tag_tid=2, entry_tid=15797' 2024-09-11T06:18:00.095 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:18:00.095 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:18:00.095 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:00.095 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=3 2024-09-11T06:18:00.096 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=17, tag_tid=2, entry_tid=15797' 2024-09-11T06:18:00.096 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:18:00.096 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:00.097 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=2 2024-09-11T06:18:00.098 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:18:00.098 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:18:00.098 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:00.099 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:18:00.099 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=17, tag_tid=2, entry_tid=15797' 2024-09-11T06:18:00.099 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:18:00.100 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:00.101 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=15797 2024-09-11T06:18:00.101 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 3 = 2 -a 1 -le 15797 2024-09-11T06:18:00.101 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:18:00.101 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:18:00.101 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:18:00.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:18:00.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:18:00.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:18:00.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:18:00.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:18:00.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:18:00.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:18:00.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:18:00.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:18:00.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:18:00.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:18:00.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:18:00.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:18:00.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:18:00.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:18:00.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:18:00.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:18:00.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:18:00.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:18:00.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:18:00.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:18:00.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:18:00.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:18:00.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:18:00.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:18:00.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:18:00.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:18:00.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:18:00.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:18:00.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:18:00.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:18:00.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:18:00.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:18:00.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:18:00.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:18:00.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:18:00.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:18:00.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:18:00.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:18:00.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:18:00.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:18:00.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:18:00.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:18:00.310 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:18:00.310 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:18:00.310 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:18:00.310 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:18:00.310 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:18:00.310 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:18:00.333 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:18:00.333 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:18:00.333 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:18:00.334 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:18:00.461 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:18:00.462 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:18:00.463 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:00.463 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:00.463 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:00.463 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:18:00.463 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:00.463 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:00.463 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:00.463 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:18:00.464 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:18:00.464 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:18:00.464 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:18:00.465 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:00.465 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:18:00.465 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:00.510 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 4 2024-09-11T06:18:00.510 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:18:00.510 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:18:00.510 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:18:00.510 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=19, tag_tid=2, entry_tid=15919], [object_number=18, tag_tid=2, entry_tid=15918], [object_number=17, tag_tid=2, entry_tid=15917], [object_number=16, tag_tid=2, entry_tid=15916]]], state=connected] 2024-09-11T06:18:00.514 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:00.515 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:18:00.516 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:18:00.516 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:00.516 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:00.516 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:00.517 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:18:00.517 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:00.517 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:00.517 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:00.517 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:18:00.517 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:18:00.517 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:18:00.517 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:18:00.518 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:00.518 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:18:00.518 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:00.562 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 4 2024-09-11T06:18:00.562 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:18:00.562 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:18:00.562 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:18:00.563 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=19, tag_tid=2, entry_tid=15919], [object_number=18, tag_tid=2, entry_tid=15918], [object_number=17, tag_tid=2, entry_tid=15917], [object_number=16, tag_tid=2, entry_tid=15916]]], state=connected] 2024-09-11T06:18:00.566 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:00.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=19, tag_tid=2, entry_tid=15919' 2024-09-11T06:18:00.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=21, tag_tid=3, entry_tid=1' -a 'object_number=21, tag_tid=3, entry_tid=1' = 'object_number=19, tag_tid=2, entry_tid=15919' 2024-09-11T06:18:00.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=19, tag_tid=2, entry_tid=15919' '!=' 'object_number=17, tag_tid=2, entry_tid=15797' 2024-09-11T06:18:00.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:18:00.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=19, tag_tid=2, entry_tid=15919' = 'object_number=17, tag_tid=2, entry_tid=15797' 2024-09-11T06:18:00.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=19, tag_tid=2, entry_tid=15919' 2024-09-11T06:18:00.569 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:18:00.569 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:18:00.569 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:00.569 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=3 2024-09-11T06:18:00.570 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=19, tag_tid=2, entry_tid=15919' 2024-09-11T06:18:00.570 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:18:00.570 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:00.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=2 2024-09-11T06:18:00.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:18:00.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:18:00.572 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:00.572 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:18:00.573 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=19, tag_tid=2, entry_tid=15919' 2024-09-11T06:18:00.573 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:18:00.573 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:00.574 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=15919 2024-09-11T06:18:00.574 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 3 = 2 -a 1 -le 15919 2024-09-11T06:18:00.574 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:18:00.574 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:18:00.575 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:18:00.776 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:18:00.777 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:18:00.777 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:18:00.777 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:18:00.777 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:18:00.777 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:18:00.777 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:18:00.777 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:18:00.777 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:18:00.777 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:18:00.777 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:18:00.777 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:18:00.777 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:18:00.778 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:18:00.778 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:18:00.778 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:18:00.778 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:18:00.781 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:18:00.781 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:18:00.781 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:18:00.781 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:18:00.781 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:18:00.781 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:18:00.781 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:18:00.781 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:18:00.781 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:18:00.781 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:18:00.781 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:18:00.781 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:18:00.782 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:18:00.782 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:18:00.782 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:18:00.782 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:18:00.782 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:18:00.782 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:18:00.782 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:18:00.782 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:18:00.782 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:18:00.782 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:18:00.782 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:18:00.782 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:18:00.782 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:18:00.782 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:18:00.783 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:18:00.783 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:18:00.783 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:18:00.783 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:18:00.783 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:18:00.783 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:18:00.802 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:18:00.803 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:18:00.803 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:18:00.803 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:18:00.934 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:18:00.936 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:18:00.936 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:00.936 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:00.936 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:00.936 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:18:00.936 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:00.936 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:00.936 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:00.936 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:18:00.936 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:18:00.937 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:18:00.937 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:18:00.939 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:00.939 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:18:00.940 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:00.985 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 4 2024-09-11T06:18:00.986 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:18:00.986 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:18:00.986 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:18:00.986 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=19, tag_tid=2, entry_tid=16023], [object_number=18, tag_tid=2, entry_tid=16022], [object_number=17, tag_tid=2, entry_tid=16021], [object_number=16, tag_tid=2, entry_tid=16020]]], state=connected] 2024-09-11T06:18:00.989 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:00.990 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:18:00.991 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:18:00.991 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:00.992 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:00.992 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:00.992 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:18:00.992 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:00.992 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:00.992 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:00.992 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:18:00.992 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:18:00.992 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:18:00.992 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:18:00.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:00.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:18:00.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:01.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 4 2024-09-11T06:18:01.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:18:01.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:18:01.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:18:01.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=19, tag_tid=2, entry_tid=16023], [object_number=18, tag_tid=2, entry_tid=16022], [object_number=17, tag_tid=2, entry_tid=16021], [object_number=16, tag_tid=2, entry_tid=16020]]], state=connected] 2024-09-11T06:18:01.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:01.044 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=19, tag_tid=2, entry_tid=16023' 2024-09-11T06:18:01.045 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=21, tag_tid=3, entry_tid=1' -a 'object_number=21, tag_tid=3, entry_tid=1' = 'object_number=19, tag_tid=2, entry_tid=16023' 2024-09-11T06:18:01.045 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=19, tag_tid=2, entry_tid=16023' '!=' 'object_number=19, tag_tid=2, entry_tid=15919' 2024-09-11T06:18:01.045 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:18:01.045 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=19, tag_tid=2, entry_tid=16023' = 'object_number=19, tag_tid=2, entry_tid=15919' 2024-09-11T06:18:01.045 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=19, tag_tid=2, entry_tid=16023' 2024-09-11T06:18:01.045 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:18:01.045 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:18:01.045 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:01.046 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=3 2024-09-11T06:18:01.046 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=19, tag_tid=2, entry_tid=16023' 2024-09-11T06:18:01.046 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:18:01.046 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:01.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=2 2024-09-11T06:18:01.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:18:01.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:18:01.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:01.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:18:01.050 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=19, tag_tid=2, entry_tid=16023' 2024-09-11T06:18:01.050 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:18:01.050 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:01.051 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=16023 2024-09-11T06:18:01.051 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 3 = 2 -a 1 -le 16023 2024-09-11T06:18:01.051 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:18:01.051 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:18:01.051 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:18:01.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:18:01.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:18:01.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:18:01.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:18:01.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:18:01.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:18:01.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:18:01.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:18:01.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:18:01.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:18:01.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:18:01.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:18:01.254 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:18:01.254 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:18:01.254 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:18:01.254 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:18:01.254 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:18:01.257 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:18:01.257 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:18:01.257 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:18:01.258 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:18:01.258 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:18:01.258 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:18:01.258 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:18:01.258 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:18:01.258 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:18:01.258 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:18:01.258 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:18:01.258 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:18:01.258 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:18:01.258 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:18:01.258 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:18:01.258 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:18:01.258 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:18:01.258 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:18:01.259 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:18:01.259 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:18:01.259 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:18:01.259 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:18:01.259 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:18:01.259 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:18:01.259 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:18:01.259 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:18:01.259 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:18:01.259 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:18:01.259 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:18:01.259 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:18:01.259 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:18:01.259 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:18:01.283 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:18:01.284 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:18:01.284 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:18:01.284 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:18:01.425 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:18:01.425 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:18:01.425 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:01.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:01.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:01.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:18:01.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:01.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:01.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:01.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:18:01.427 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:18:01.427 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:18:01.427 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:18:01.429 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:01.429 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:18:01.429 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:01.476 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 4 2024-09-11T06:18:01.476 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:18:01.476 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:18:01.476 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:18:01.476 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=16137], [object_number=16, tag_tid=2, entry_tid=16136], [object_number=19, tag_tid=2, entry_tid=16135], [object_number=18, tag_tid=2, entry_tid=16134]]], state=connected] 2024-09-11T06:18:01.479 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:01.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:18:01.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:18:01.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:01.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:01.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:01.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:18:01.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:01.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:01.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:01.483 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:18:01.483 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:18:01.483 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:18:01.483 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:18:01.483 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:01.483 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:18:01.483 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:01.528 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 4 2024-09-11T06:18:01.528 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:18:01.528 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:18:01.528 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:18:01.528 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=16137], [object_number=16, tag_tid=2, entry_tid=16136], [object_number=19, tag_tid=2, entry_tid=16135], [object_number=18, tag_tid=2, entry_tid=16134]]], state=connected] 2024-09-11T06:18:01.531 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:01.533 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=17, tag_tid=2, entry_tid=16137' 2024-09-11T06:18:01.533 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=21, tag_tid=3, entry_tid=1' -a 'object_number=21, tag_tid=3, entry_tid=1' = 'object_number=17, tag_tid=2, entry_tid=16137' 2024-09-11T06:18:01.533 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=17, tag_tid=2, entry_tid=16137' '!=' 'object_number=19, tag_tid=2, entry_tid=16023' 2024-09-11T06:18:01.533 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:18:01.534 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=17, tag_tid=2, entry_tid=16137' = 'object_number=19, tag_tid=2, entry_tid=16023' 2024-09-11T06:18:01.534 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=17, tag_tid=2, entry_tid=16137' 2024-09-11T06:18:01.534 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:18:01.534 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:18:01.534 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:01.534 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=3 2024-09-11T06:18:01.535 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=17, tag_tid=2, entry_tid=16137' 2024-09-11T06:18:01.535 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:18:01.535 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:01.536 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=2 2024-09-11T06:18:01.537 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:18:01.537 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:18:01.537 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:01.538 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:18:01.538 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=17, tag_tid=2, entry_tid=16137' 2024-09-11T06:18:01.538 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:18:01.539 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:01.540 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=16137 2024-09-11T06:18:01.540 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 3 = 2 -a 1 -le 16137 2024-09-11T06:18:01.540 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:18:01.540 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:18:01.540 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:18:01.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:18:01.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:18:01.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:18:01.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:18:01.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:18:01.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:18:01.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:18:01.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:18:01.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:18:01.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:18:01.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:18:01.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:18:01.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:18:01.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:18:01.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:18:01.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:18:01.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:18:01.745 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:18:01.746 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:18:01.746 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:18:01.746 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:18:01.746 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:18:01.746 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:18:01.746 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:18:01.746 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:18:01.746 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:18:01.746 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:18:01.746 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:18:01.746 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:18:01.746 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:18:01.747 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:18:01.747 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:18:01.747 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:18:01.747 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:18:01.747 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:18:01.748 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:18:01.748 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:18:01.748 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:18:01.748 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:18:01.748 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:18:01.748 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:18:01.748 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:18:01.748 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:18:01.748 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:18:01.748 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:18:01.748 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:18:01.748 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:18:01.748 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:18:01.749 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:18:01.772 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:18:01.772 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:18:01.773 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:18:01.773 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:18:01.900 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:18:01.901 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:18:01.901 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:01.902 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:01.902 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:01.902 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:18:01.902 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:01.902 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:01.902 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:01.902 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:18:01.902 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:18:01.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:18:01.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:18:01.905 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:01.905 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:18:01.905 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:01.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 4 2024-09-11T06:18:01.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:18:01.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:18:01.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:18:01.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=18, tag_tid=2, entry_tid=16254], [object_number=17, tag_tid=2, entry_tid=16253], [object_number=16, tag_tid=2, entry_tid=16252], [object_number=19, tag_tid=2, entry_tid=16251]]], state=connected] 2024-09-11T06:18:01.953 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:01.956 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:18:01.956 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:18:01.956 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:01.956 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:01.956 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:01.956 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:18:01.957 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:01.957 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:01.957 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:01.957 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:18:01.957 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:18:01.957 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:18:01.957 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:18:01.958 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:01.958 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:18:01.958 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:02.002 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 4 2024-09-11T06:18:02.002 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:18:02.002 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:18:02.003 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:18:02.003 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=18, tag_tid=2, entry_tid=16254], [object_number=17, tag_tid=2, entry_tid=16253], [object_number=16, tag_tid=2, entry_tid=16252], [object_number=19, tag_tid=2, entry_tid=16251]]], state=connected] 2024-09-11T06:18:02.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:02.008 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=18, tag_tid=2, entry_tid=16254' 2024-09-11T06:18:02.008 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=21, tag_tid=3, entry_tid=1' -a 'object_number=21, tag_tid=3, entry_tid=1' = 'object_number=18, tag_tid=2, entry_tid=16254' 2024-09-11T06:18:02.008 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=18, tag_tid=2, entry_tid=16254' '!=' 'object_number=17, tag_tid=2, entry_tid=16137' 2024-09-11T06:18:02.008 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:18:02.008 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=18, tag_tid=2, entry_tid=16254' = 'object_number=17, tag_tid=2, entry_tid=16137' 2024-09-11T06:18:02.008 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=18, tag_tid=2, entry_tid=16254' 2024-09-11T06:18:02.008 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:18:02.009 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:18:02.009 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:02.009 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=3 2024-09-11T06:18:02.010 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=18, tag_tid=2, entry_tid=16254' 2024-09-11T06:18:02.010 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:18:02.010 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:02.011 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=2 2024-09-11T06:18:02.012 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:18:02.012 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:18:02.012 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:02.013 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:18:02.013 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=18, tag_tid=2, entry_tid=16254' 2024-09-11T06:18:02.013 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:18:02.013 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:02.015 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=16254 2024-09-11T06:18:02.015 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 3 = 2 -a 1 -le 16254 2024-09-11T06:18:02.015 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:18:02.015 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:18:02.015 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:18:02.216 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:18:02.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:18:02.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:18:02.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:18:02.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:18:02.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:18:02.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:18:02.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:18:02.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:18:02.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:18:02.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:18:02.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:18:02.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:18:02.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:18:02.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:18:02.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:18:02.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:18:02.221 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:18:02.221 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:18:02.221 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:18:02.221 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:18:02.221 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:18:02.221 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:18:02.221 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:18:02.221 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:18:02.221 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:18:02.221 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:18:02.222 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:18:02.222 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:18:02.222 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:18:02.222 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:18:02.222 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:18:02.222 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:18:02.223 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:18:02.223 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:18:02.223 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:18:02.223 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:18:02.223 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:18:02.223 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:18:02.224 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:18:02.224 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:18:02.224 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:18:02.224 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:18:02.224 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:18:02.224 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:18:02.224 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:18:02.224 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:18:02.224 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:18:02.224 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:18:02.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:18:02.250 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:18:02.250 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:18:02.250 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:18:02.380 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:18:02.381 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:18:02.381 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:02.381 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:02.382 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:02.382 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:18:02.382 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:02.382 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:02.382 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:02.382 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:18:02.382 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:18:02.383 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:18:02.383 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:18:02.384 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:02.385 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:18:02.385 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:02.432 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 4 2024-09-11T06:18:02.432 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:18:02.432 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:18:02.432 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:18:02.432 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=19, tag_tid=2, entry_tid=16367], [object_number=18, tag_tid=2, entry_tid=16366], [object_number=17, tag_tid=2, entry_tid=16365], [object_number=16, tag_tid=2, entry_tid=16364]]], state=connected] 2024-09-11T06:18:02.436 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:02.437 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:18:02.438 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:18:02.438 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:02.438 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:02.438 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:02.438 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:18:02.438 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:02.438 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:02.438 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:02.438 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:18:02.438 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:18:02.438 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:18:02.439 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:18:02.439 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:02.439 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:18:02.439 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:02.484 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 4 2024-09-11T06:18:02.484 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:18:02.484 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:18:02.485 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:18:02.485 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=19, tag_tid=2, entry_tid=16367], [object_number=18, tag_tid=2, entry_tid=16366], [object_number=17, tag_tid=2, entry_tid=16365], [object_number=16, tag_tid=2, entry_tid=16364]]], state=connected] 2024-09-11T06:18:02.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:02.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=19, tag_tid=2, entry_tid=16367' 2024-09-11T06:18:02.490 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=21, tag_tid=3, entry_tid=1' -a 'object_number=21, tag_tid=3, entry_tid=1' = 'object_number=19, tag_tid=2, entry_tid=16367' 2024-09-11T06:18:02.490 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=19, tag_tid=2, entry_tid=16367' '!=' 'object_number=18, tag_tid=2, entry_tid=16254' 2024-09-11T06:18:02.490 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:18:02.490 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=19, tag_tid=2, entry_tid=16367' = 'object_number=18, tag_tid=2, entry_tid=16254' 2024-09-11T06:18:02.490 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=19, tag_tid=2, entry_tid=16367' 2024-09-11T06:18:02.490 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:18:02.490 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:18:02.490 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:02.491 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=3 2024-09-11T06:18:02.492 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=19, tag_tid=2, entry_tid=16367' 2024-09-11T06:18:02.492 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:18:02.492 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:02.493 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=2 2024-09-11T06:18:02.493 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:18:02.493 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:18:02.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:02.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:18:02.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=19, tag_tid=2, entry_tid=16367' 2024-09-11T06:18:02.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:18:02.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:02.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=16367 2024-09-11T06:18:02.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 3 = 2 -a 1 -le 16367 2024-09-11T06:18:02.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:18:02.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:18:02.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:18:02.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:18:02.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:18:02.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:18:02.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:18:02.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:18:02.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:18:02.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:18:02.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:18:02.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:18:02.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:18:02.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:18:02.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:18:02.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:18:02.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:18:02.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:18:02.700 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:18:02.700 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:18:02.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:18:02.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:18:02.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:18:02.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:18:02.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:18:02.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:18:02.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:18:02.706 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:18:02.706 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:18:02.706 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:18:02.706 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:18:02.706 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:18:02.706 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:18:02.707 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:18:02.707 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:18:02.707 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:18:02.707 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:18:02.707 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:18:02.707 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:18:02.708 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:18:02.708 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:18:02.708 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:18:02.708 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:18:02.708 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:18:02.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:18:02.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:18:02.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:18:02.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:18:02.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:18:02.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:18:02.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:18:02.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:18:02.729 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:18:02.729 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:18:02.729 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:18:02.729 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:18:02.860 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:18:02.862 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:18:02.862 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:02.862 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:02.862 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:02.862 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:18:02.862 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:02.862 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:02.862 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:02.863 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:18:02.863 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:18:02.863 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:18:02.863 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:18:02.865 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:02.865 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:18:02.865 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:02.911 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 4 2024-09-11T06:18:02.911 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:18:02.911 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:18:02.911 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:18:02.911 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=18, tag_tid=2, entry_tid=16482], [object_number=17, tag_tid=2, entry_tid=16481], [object_number=16, tag_tid=2, entry_tid=16480], [object_number=19, tag_tid=2, entry_tid=16479]]], state=connected] 2024-09-11T06:18:02.915 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:02.916 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:18:02.917 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:18:02.917 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:02.917 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:02.917 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:02.917 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:18:02.917 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:02.917 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:02.917 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:02.917 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:18:02.918 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:18:02.918 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:18:02.918 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:18:02.918 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:02.918 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:18:02.918 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:02.963 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 4 2024-09-11T06:18:02.964 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:18:02.964 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:18:02.964 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:18:02.964 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=18, tag_tid=2, entry_tid=16482], [object_number=17, tag_tid=2, entry_tid=16481], [object_number=16, tag_tid=2, entry_tid=16480], [object_number=19, tag_tid=2, entry_tid=16479]]], state=connected] 2024-09-11T06:18:02.968 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:02.969 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=18, tag_tid=2, entry_tid=16482' 2024-09-11T06:18:02.969 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=21, tag_tid=3, entry_tid=1' -a 'object_number=21, tag_tid=3, entry_tid=1' = 'object_number=18, tag_tid=2, entry_tid=16482' 2024-09-11T06:18:02.969 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=18, tag_tid=2, entry_tid=16482' '!=' 'object_number=19, tag_tid=2, entry_tid=16367' 2024-09-11T06:18:02.970 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:18:02.970 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=18, tag_tid=2, entry_tid=16482' = 'object_number=19, tag_tid=2, entry_tid=16367' 2024-09-11T06:18:02.970 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=18, tag_tid=2, entry_tid=16482' 2024-09-11T06:18:02.970 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:18:02.970 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:18:02.970 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:02.971 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=3 2024-09-11T06:18:02.972 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=18, tag_tid=2, entry_tid=16482' 2024-09-11T06:18:02.972 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:18:02.972 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:02.973 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=2 2024-09-11T06:18:02.973 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:18:02.973 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:18:02.973 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:02.974 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:18:02.975 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=18, tag_tid=2, entry_tid=16482' 2024-09-11T06:18:02.975 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:18:02.975 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:02.976 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=16482 2024-09-11T06:18:02.976 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 3 = 2 -a 1 -le 16482 2024-09-11T06:18:02.976 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:18:02.977 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:18:02.977 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:18:03.178 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:18:03.178 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:18:03.178 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:18:03.178 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:18:03.178 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:18:03.178 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:18:03.178 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:18:03.179 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:18:03.179 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:18:03.179 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:18:03.179 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:18:03.179 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:18:03.179 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:18:03.179 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:18:03.179 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:18:03.180 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:18:03.180 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:18:03.182 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:18:03.182 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:18:03.183 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:18:03.183 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:18:03.183 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:18:03.183 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:18:03.183 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:18:03.183 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:18:03.183 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:18:03.183 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:18:03.184 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:18:03.184 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:18:03.184 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:18:03.184 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:18:03.184 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:18:03.184 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:18:03.184 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:18:03.184 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:18:03.184 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:18:03.184 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:18:03.184 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:18:03.184 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:18:03.184 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:18:03.184 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:18:03.185 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:18:03.185 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:18:03.185 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:18:03.185 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:18:03.185 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:18:03.185 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:18:03.185 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:18:03.185 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:18:03.208 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:18:03.209 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:18:03.209 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:18:03.209 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:18:03.339 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:18:03.340 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:18:03.340 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:03.340 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:03.340 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:03.340 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:18:03.341 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:03.341 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:03.341 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:03.341 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:18:03.341 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:18:03.341 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:18:03.341 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:18:03.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:03.343 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:18:03.343 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:03.395 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 4 2024-09-11T06:18:03.395 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:18:03.395 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:18:03.395 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:18:03.395 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=16597], [object_number=16, tag_tid=2, entry_tid=16596], [object_number=19, tag_tid=2, entry_tid=16595], [object_number=18, tag_tid=2, entry_tid=16594]]], state=connected] 2024-09-11T06:18:03.398 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:03.399 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:18:03.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:18:03.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:03.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:03.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:03.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:18:03.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:03.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:03.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:03.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:18:03.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:18:03.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:18:03.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:18:03.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:03.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:18:03.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:03.448 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 4 2024-09-11T06:18:03.448 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:18:03.448 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:18:03.448 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:18:03.448 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=16597], [object_number=16, tag_tid=2, entry_tid=16596], [object_number=19, tag_tid=2, entry_tid=16595], [object_number=18, tag_tid=2, entry_tid=16594]]], state=connected] 2024-09-11T06:18:03.451 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:03.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=17, tag_tid=2, entry_tid=16597' 2024-09-11T06:18:03.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=21, tag_tid=3, entry_tid=1' -a 'object_number=21, tag_tid=3, entry_tid=1' = 'object_number=17, tag_tid=2, entry_tid=16597' 2024-09-11T06:18:03.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=17, tag_tid=2, entry_tid=16597' '!=' 'object_number=18, tag_tid=2, entry_tid=16482' 2024-09-11T06:18:03.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:18:03.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=17, tag_tid=2, entry_tid=16597' = 'object_number=18, tag_tid=2, entry_tid=16482' 2024-09-11T06:18:03.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=17, tag_tid=2, entry_tid=16597' 2024-09-11T06:18:03.454 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:18:03.454 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:18:03.454 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:03.454 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=3 2024-09-11T06:18:03.455 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=17, tag_tid=2, entry_tid=16597' 2024-09-11T06:18:03.455 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:18:03.455 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:03.456 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=2 2024-09-11T06:18:03.457 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:18:03.457 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:18:03.457 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:03.458 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:18:03.458 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=17, tag_tid=2, entry_tid=16597' 2024-09-11T06:18:03.458 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:18:03.458 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:03.460 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=16597 2024-09-11T06:18:03.460 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 3 = 2 -a 1 -le 16597 2024-09-11T06:18:03.460 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:18:03.460 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:18:03.460 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:18:03.661 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:18:03.661 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:18:03.661 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:18:03.661 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:18:03.661 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:18:03.662 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:18:03.662 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:18:03.662 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:18:03.662 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:18:03.662 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:18:03.662 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:18:03.662 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:18:03.662 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:18:03.662 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:18:03.662 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:18:03.662 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:18:03.663 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:18:03.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:18:03.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:18:03.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:18:03.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:18:03.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:18:03.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:18:03.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:18:03.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:18:03.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:18:03.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:18:03.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:18:03.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:18:03.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:18:03.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:18:03.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:18:03.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:18:03.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:18:03.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:18:03.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:18:03.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:18:03.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:18:03.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:18:03.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:18:03.668 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:18:03.668 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:18:03.668 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:18:03.668 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:18:03.668 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:18:03.668 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:18:03.668 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:18:03.668 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:18:03.668 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:18:03.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:18:03.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:18:03.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:18:03.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:18:03.819 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:18:03.820 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:18:03.820 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:03.820 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:03.820 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:03.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:18:03.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:03.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:03.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:03.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:18:03.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:18:03.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:18:03.822 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:18:03.823 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:03.823 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:18:03.823 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:03.874 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 4 2024-09-11T06:18:03.874 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:18:03.874 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:18:03.874 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:18:03.874 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=19, tag_tid=2, entry_tid=16715], [object_number=18, tag_tid=2, entry_tid=16714], [object_number=17, tag_tid=2, entry_tid=16713], [object_number=16, tag_tid=2, entry_tid=16712]]], state=connected] 2024-09-11T06:18:03.877 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:03.879 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:18:03.880 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:18:03.880 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:03.880 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:03.880 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:03.880 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:18:03.880 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:03.880 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:03.880 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:03.881 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:18:03.881 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:18:03.881 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:18:03.881 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:18:03.881 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:03.881 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:18:03.881 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:03.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 4 2024-09-11T06:18:03.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:18:03.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:18:03.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:18:03.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=19, tag_tid=2, entry_tid=16715], [object_number=18, tag_tid=2, entry_tid=16714], [object_number=17, tag_tid=2, entry_tid=16713], [object_number=16, tag_tid=2, entry_tid=16712]]], state=connected] 2024-09-11T06:18:03.930 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:03.932 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=19, tag_tid=2, entry_tid=16715' 2024-09-11T06:18:03.932 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=21, tag_tid=3, entry_tid=1' -a 'object_number=21, tag_tid=3, entry_tid=1' = 'object_number=19, tag_tid=2, entry_tid=16715' 2024-09-11T06:18:03.932 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=19, tag_tid=2, entry_tid=16715' '!=' 'object_number=17, tag_tid=2, entry_tid=16597' 2024-09-11T06:18:03.932 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:18:03.932 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=19, tag_tid=2, entry_tid=16715' = 'object_number=17, tag_tid=2, entry_tid=16597' 2024-09-11T06:18:03.932 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=19, tag_tid=2, entry_tid=16715' 2024-09-11T06:18:03.933 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:18:03.933 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:18:03.933 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:03.933 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=3 2024-09-11T06:18:03.934 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=19, tag_tid=2, entry_tid=16715' 2024-09-11T06:18:03.934 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:18:03.934 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:03.935 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=2 2024-09-11T06:18:03.936 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:18:03.936 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:18:03.936 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:03.937 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:18:03.938 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=19, tag_tid=2, entry_tid=16715' 2024-09-11T06:18:03.938 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:18:03.938 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:03.939 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=16715 2024-09-11T06:18:03.939 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 3 = 2 -a 1 -le 16715 2024-09-11T06:18:03.939 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:18:03.939 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:18:03.939 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:18:04.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:18:04.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:18:04.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:18:04.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:18:04.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:18:04.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:18:04.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:18:04.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:18:04.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:18:04.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:18:04.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:18:04.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:18:04.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:18:04.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:18:04.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:18:04.143 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:18:04.143 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:18:04.146 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:18:04.146 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:18:04.146 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:18:04.146 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:18:04.146 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:18:04.147 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:18:04.147 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:18:04.147 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:18:04.147 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:18:04.147 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:18:04.147 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:18:04.147 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:18:04.147 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:18:04.147 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:18:04.147 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:18:04.147 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:18:04.147 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:18:04.147 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:18:04.147 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:18:04.147 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:18:04.148 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:18:04.148 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:18:04.148 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:18:04.148 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:18:04.148 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:18:04.148 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:18:04.148 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:18:04.148 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:18:04.148 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:18:04.148 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:18:04.148 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:18:04.148 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:18:04.171 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:18:04.172 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:18:04.172 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:18:04.172 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:18:04.301 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:18:04.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:18:04.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:04.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:04.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:04.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:18:04.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:04.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:04.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:04.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:18:04.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:18:04.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:18:04.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:18:04.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:04.305 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:18:04.305 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:04.351 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 4 2024-09-11T06:18:04.351 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:18:04.351 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:18:04.351 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:18:04.351 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=18, tag_tid=2, entry_tid=16830], [object_number=17, tag_tid=2, entry_tid=16829], [object_number=16, tag_tid=2, entry_tid=16828], [object_number=19, tag_tid=2, entry_tid=16827]]], state=connected] 2024-09-11T06:18:04.354 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:04.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:18:04.356 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:18:04.357 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:04.357 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:04.357 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:04.357 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:18:04.357 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:04.357 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:04.357 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:04.357 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:18:04.357 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:18:04.357 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:18:04.357 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:18:04.358 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:04.358 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:18:04.358 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:04.403 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 4 2024-09-11T06:18:04.404 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:18:04.404 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:18:04.404 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:18:04.404 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=18, tag_tid=2, entry_tid=16830], [object_number=17, tag_tid=2, entry_tid=16829], [object_number=16, tag_tid=2, entry_tid=16828], [object_number=19, tag_tid=2, entry_tid=16827]]], state=connected] 2024-09-11T06:18:04.407 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:04.409 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=18, tag_tid=2, entry_tid=16830' 2024-09-11T06:18:04.409 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=21, tag_tid=3, entry_tid=1' -a 'object_number=21, tag_tid=3, entry_tid=1' = 'object_number=18, tag_tid=2, entry_tid=16830' 2024-09-11T06:18:04.409 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=18, tag_tid=2, entry_tid=16830' '!=' 'object_number=19, tag_tid=2, entry_tid=16715' 2024-09-11T06:18:04.409 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:18:04.409 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=18, tag_tid=2, entry_tid=16830' = 'object_number=19, tag_tid=2, entry_tid=16715' 2024-09-11T06:18:04.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=18, tag_tid=2, entry_tid=16830' 2024-09-11T06:18:04.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:18:04.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:18:04.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:04.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=3 2024-09-11T06:18:04.411 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=18, tag_tid=2, entry_tid=16830' 2024-09-11T06:18:04.411 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:18:04.411 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:04.412 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=2 2024-09-11T06:18:04.412 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:18:04.413 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:18:04.413 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:04.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:18:04.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=18, tag_tid=2, entry_tid=16830' 2024-09-11T06:18:04.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:18:04.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:04.416 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=16830 2024-09-11T06:18:04.416 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 3 = 2 -a 1 -le 16830 2024-09-11T06:18:04.416 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:18:04.416 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:18:04.416 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:18:04.617 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:18:04.617 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:18:04.617 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:18:04.618 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:18:04.618 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:18:04.618 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:18:04.618 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:18:04.618 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:18:04.618 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:18:04.618 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:18:04.618 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:18:04.618 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:18:04.618 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:18:04.618 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:18:04.618 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:18:04.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:18:04.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:18:04.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:18:04.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:18:04.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:18:04.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:18:04.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:18:04.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:18:04.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:18:04.623 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:18:04.623 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:18:04.623 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:18:04.623 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:18:04.623 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:18:04.623 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:18:04.623 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:18:04.623 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:18:04.623 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:18:04.623 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:18:04.623 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:18:04.623 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:18:04.623 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:18:04.624 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:18:04.624 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:18:04.624 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:18:04.624 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:18:04.624 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:18:04.624 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:18:04.624 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:18:04.624 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:18:04.624 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:18:04.624 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:18:04.624 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:18:04.624 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:18:04.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:18:04.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:18:04.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:18:04.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:18:04.775 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:18:04.776 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:18:04.776 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:04.776 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:04.776 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:04.776 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:18:04.776 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:04.776 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:04.776 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:04.776 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:18:04.777 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:18:04.777 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:18:04.777 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:18:04.780 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:04.780 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:18:04.780 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:05.030 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 4 2024-09-11T06:18:05.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:18:05.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:18:05.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:18:05.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=18, tag_tid=2, entry_tid=16942], [object_number=17, tag_tid=2, entry_tid=16941], [object_number=16, tag_tid=2, entry_tid=16940], [object_number=19, tag_tid=2, entry_tid=16939]]], state=connected] 2024-09-11T06:18:05.063 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:05.063 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:18:05.063 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:18:05.063 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:05.063 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:05.063 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:05.063 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:18:05.063 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:05.063 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:05.063 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:05.063 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:18:05.063 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:18:05.064 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:18:05.064 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:18:05.064 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:05.064 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:18:05.064 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:05.095 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 4 2024-09-11T06:18:05.096 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:18:05.096 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:18:05.096 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:18:05.096 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=18, tag_tid=2, entry_tid=16942], [object_number=17, tag_tid=2, entry_tid=16941], [object_number=16, tag_tid=2, entry_tid=16940], [object_number=19, tag_tid=2, entry_tid=16939]]], state=connected] 2024-09-11T06:18:05.100 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:05.101 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=18, tag_tid=2, entry_tid=16942' 2024-09-11T06:18:05.101 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=21, tag_tid=3, entry_tid=1' -a 'object_number=21, tag_tid=3, entry_tid=1' = 'object_number=18, tag_tid=2, entry_tid=16942' 2024-09-11T06:18:05.102 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=18, tag_tid=2, entry_tid=16942' '!=' 'object_number=18, tag_tid=2, entry_tid=16830' 2024-09-11T06:18:05.102 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:18:05.102 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=18, tag_tid=2, entry_tid=16942' = 'object_number=18, tag_tid=2, entry_tid=16830' 2024-09-11T06:18:05.102 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=18, tag_tid=2, entry_tid=16942' 2024-09-11T06:18:05.102 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:18:05.102 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:18:05.103 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:05.103 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=3 2024-09-11T06:18:05.104 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=18, tag_tid=2, entry_tid=16942' 2024-09-11T06:18:05.104 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:18:05.104 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:05.105 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=2 2024-09-11T06:18:05.105 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:18:05.106 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:18:05.106 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:05.107 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:18:05.107 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=18, tag_tid=2, entry_tid=16942' 2024-09-11T06:18:05.107 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:18:05.108 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:05.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=16942 2024-09-11T06:18:05.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 3 = 2 -a 1 -le 16942 2024-09-11T06:18:05.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:18:05.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:18:05.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:18:05.311 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:18:05.311 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:18:05.311 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:18:05.311 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:18:05.311 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:18:05.311 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:18:05.311 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:18:05.311 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:18:05.311 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:18:05.312 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:18:05.312 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:18:05.312 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:18:05.312 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:18:05.312 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:18:05.312 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:18:05.312 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:18:05.313 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:18:05.315 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:18:05.315 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:18:05.315 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:18:05.315 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:18:05.315 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:18:05.315 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:18:05.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:18:05.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:18:05.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:18:05.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:18:05.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:18:05.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:18:05.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:18:05.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:18:05.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:18:05.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:18:05.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:18:05.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:18:05.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:18:05.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:18:05.317 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:18:05.317 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:18:05.317 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:18:05.317 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:18:05.317 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:18:05.317 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:18:05.317 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:18:05.317 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:18:05.317 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:18:05.317 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:18:05.317 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:18:05.317 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:18:05.340 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:18:05.341 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:18:05.341 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:18:05.341 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:18:05.470 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:18:05.471 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:18:05.471 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:05.471 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:05.471 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:05.471 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:18:05.471 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:05.471 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:05.471 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:05.472 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:18:05.472 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:18:05.472 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:18:05.472 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:18:05.473 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:05.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:18:05.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:05.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 4 2024-09-11T06:18:05.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:18:05.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:18:05.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:18:05.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=16, tag_tid=2, entry_tid=17100], [object_number=19, tag_tid=2, entry_tid=17099], [object_number=18, tag_tid=2, entry_tid=17098], [object_number=17, tag_tid=2, entry_tid=17097]]], state=connected] 2024-09-11T06:18:05.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:05.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:18:05.745 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:18:05.746 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:05.746 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:05.746 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:05.746 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:18:05.746 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:05.746 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:05.746 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:05.746 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:18:05.746 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:18:05.746 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:18:05.746 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:18:05.747 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:05.747 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:18:05.747 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:05.796 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 4 2024-09-11T06:18:05.796 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:18:05.796 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:18:05.796 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:18:05.797 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=16, tag_tid=2, entry_tid=17100], [object_number=19, tag_tid=2, entry_tid=17099], [object_number=18, tag_tid=2, entry_tid=17098], [object_number=17, tag_tid=2, entry_tid=17097]]], state=connected] 2024-09-11T06:18:05.800 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:05.802 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=16, tag_tid=2, entry_tid=17100' 2024-09-11T06:18:05.802 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=21, tag_tid=3, entry_tid=1' -a 'object_number=21, tag_tid=3, entry_tid=1' = 'object_number=16, tag_tid=2, entry_tid=17100' 2024-09-11T06:18:05.802 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=16, tag_tid=2, entry_tid=17100' '!=' 'object_number=18, tag_tid=2, entry_tid=16942' 2024-09-11T06:18:05.802 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:18:05.802 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=16, tag_tid=2, entry_tid=17100' = 'object_number=18, tag_tid=2, entry_tid=16942' 2024-09-11T06:18:05.803 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=16, tag_tid=2, entry_tid=17100' 2024-09-11T06:18:05.803 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:18:05.803 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:18:05.803 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:05.803 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=3 2024-09-11T06:18:05.804 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=16, tag_tid=2, entry_tid=17100' 2024-09-11T06:18:05.804 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:18:05.804 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:05.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=2 2024-09-11T06:18:05.806 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:18:05.806 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:18:05.806 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:05.807 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:18:05.808 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=16, tag_tid=2, entry_tid=17100' 2024-09-11T06:18:05.808 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:18:05.808 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:05.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=17100 2024-09-11T06:18:05.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 3 = 2 -a 1 -le 17100 2024-09-11T06:18:05.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:18:05.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:18:05.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:18:06.011 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:18:06.011 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:18:06.011 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:18:06.011 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:18:06.011 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:18:06.011 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:18:06.011 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:18:06.011 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:18:06.012 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:18:06.012 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:18:06.012 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:18:06.012 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:18:06.012 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:18:06.012 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:18:06.012 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:18:06.012 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:18:06.013 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:18:06.016 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:18:06.016 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:18:06.016 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:18:06.016 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:18:06.016 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:18:06.016 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:18:06.016 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:18:06.016 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:18:06.016 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:18:06.016 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:18:06.017 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:18:06.017 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:18:06.017 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:18:06.017 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:18:06.017 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:18:06.017 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:18:06.018 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:18:06.018 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:18:06.018 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:18:06.018 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:18:06.018 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:18:06.018 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:18:06.018 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:18:06.018 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:18:06.019 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:18:06.019 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:18:06.019 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:18:06.019 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:18:06.019 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:18:06.019 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:18:06.019 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:18:06.019 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:18:06.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:18:06.043 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:18:06.043 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:18:06.043 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:18:06.179 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:18:06.179 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:18:06.180 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:06.180 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:06.180 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:06.180 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:18:06.180 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:06.180 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:06.180 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:06.180 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:18:06.181 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:18:06.181 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:18:06.181 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:18:06.183 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:06.183 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:18:06.184 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:06.228 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 4 2024-09-11T06:18:06.228 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:18:06.228 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:18:06.228 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:18:06.228 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=19, tag_tid=2, entry_tid=17263], [object_number=18, tag_tid=2, entry_tid=17262], [object_number=17, tag_tid=2, entry_tid=17261], [object_number=16, tag_tid=2, entry_tid=17260]]], state=connected] 2024-09-11T06:18:06.231 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:06.233 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:18:06.234 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:18:06.234 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:06.234 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:06.234 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:06.234 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:18:06.234 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:06.234 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:06.234 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:06.234 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:18:06.235 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:18:06.235 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:18:06.235 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:18:06.235 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:06.235 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:18:06.235 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:06.279 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 4 2024-09-11T06:18:06.279 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:18:06.279 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:18:06.280 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:18:06.280 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=19, tag_tid=2, entry_tid=17263], [object_number=18, tag_tid=2, entry_tid=17262], [object_number=17, tag_tid=2, entry_tid=17261], [object_number=16, tag_tid=2, entry_tid=17260]]], state=connected] 2024-09-11T06:18:06.283 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:06.285 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=19, tag_tid=2, entry_tid=17263' 2024-09-11T06:18:06.285 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=21, tag_tid=3, entry_tid=1' -a 'object_number=21, tag_tid=3, entry_tid=1' = 'object_number=19, tag_tid=2, entry_tid=17263' 2024-09-11T06:18:06.285 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=19, tag_tid=2, entry_tid=17263' '!=' 'object_number=16, tag_tid=2, entry_tid=17100' 2024-09-11T06:18:06.285 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:18:06.285 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=19, tag_tid=2, entry_tid=17263' = 'object_number=16, tag_tid=2, entry_tid=17100' 2024-09-11T06:18:06.285 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=19, tag_tid=2, entry_tid=17263' 2024-09-11T06:18:06.286 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:18:06.286 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:18:06.286 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:06.286 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=3 2024-09-11T06:18:06.287 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=19, tag_tid=2, entry_tid=17263' 2024-09-11T06:18:06.287 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:18:06.287 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:06.288 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=2 2024-09-11T06:18:06.288 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:18:06.289 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:18:06.289 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:06.289 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:18:06.290 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=19, tag_tid=2, entry_tid=17263' 2024-09-11T06:18:06.290 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:18:06.290 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:06.291 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=17263 2024-09-11T06:18:06.291 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 3 = 2 -a 1 -le 17263 2024-09-11T06:18:06.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:18:06.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:18:06.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:18:06.493 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:18:06.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:18:06.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:18:06.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:18:06.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:18:06.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:18:06.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:18:06.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:18:06.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:18:06.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:18:06.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:18:06.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:18:06.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:18:06.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:18:06.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:18:06.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:18:06.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:18:06.498 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:18:06.498 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:18:06.498 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:18:06.498 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:18:06.498 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:18:06.498 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:18:06.499 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:18:06.499 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:18:06.499 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:18:06.499 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:18:06.499 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:18:06.499 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:18:06.499 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:18:06.499 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:18:06.499 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:18:06.499 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:18:06.499 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:18:06.499 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:18:06.499 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:18:06.499 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:18:06.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:18:06.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:18:06.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:18:06.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:18:06.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:18:06.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:18:06.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:18:06.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:18:06.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:18:06.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:18:06.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:18:06.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:18:06.525 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:18:06.526 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:18:06.526 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:18:06.526 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:18:06.655 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:18:06.656 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:18:06.656 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:06.656 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:06.656 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:06.656 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:18:06.656 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:06.656 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:06.656 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:06.656 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:18:06.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:18:06.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:18:06.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:18:06.659 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:06.659 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:18:06.659 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:06.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 4 2024-09-11T06:18:06.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:18:06.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:18:06.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:18:06.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=18, tag_tid=2, entry_tid=17382], [object_number=17, tag_tid=2, entry_tid=17381], [object_number=16, tag_tid=2, entry_tid=17380], [object_number=19, tag_tid=2, entry_tid=17379]]], state=connected] 2024-09-11T06:18:06.708 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:06.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:18:06.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:18:06.711 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:06.711 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:06.711 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:06.711 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:18:06.711 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:06.711 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:06.711 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:06.711 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:18:06.711 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:18:06.711 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:18:06.711 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:18:06.712 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:06.712 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:18:06.712 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:06.757 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 4 2024-09-11T06:18:06.758 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:18:06.758 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:18:06.758 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:18:06.758 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=18, tag_tid=2, entry_tid=17382], [object_number=17, tag_tid=2, entry_tid=17381], [object_number=16, tag_tid=2, entry_tid=17380], [object_number=19, tag_tid=2, entry_tid=17379]]], state=connected] 2024-09-11T06:18:06.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:06.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=18, tag_tid=2, entry_tid=17382' 2024-09-11T06:18:06.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=21, tag_tid=3, entry_tid=1' -a 'object_number=21, tag_tid=3, entry_tid=1' = 'object_number=18, tag_tid=2, entry_tid=17382' 2024-09-11T06:18:06.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=18, tag_tid=2, entry_tid=17382' '!=' 'object_number=19, tag_tid=2, entry_tid=17263' 2024-09-11T06:18:06.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:18:06.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=18, tag_tid=2, entry_tid=17382' = 'object_number=19, tag_tid=2, entry_tid=17263' 2024-09-11T06:18:06.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=18, tag_tid=2, entry_tid=17382' 2024-09-11T06:18:06.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:18:06.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:18:06.764 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:06.764 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=3 2024-09-11T06:18:06.765 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=18, tag_tid=2, entry_tid=17382' 2024-09-11T06:18:06.765 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:18:06.765 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:06.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=2 2024-09-11T06:18:06.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:18:06.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:18:06.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:06.768 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:18:06.768 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=18, tag_tid=2, entry_tid=17382' 2024-09-11T06:18:06.768 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:18:06.768 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:06.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=17382 2024-09-11T06:18:06.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 3 = 2 -a 1 -le 17382 2024-09-11T06:18:06.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:18:06.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:18:06.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:18:06.972 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:18:06.972 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:18:06.972 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:18:06.972 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:18:06.972 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:18:06.972 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:18:06.972 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:18:06.972 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:18:06.972 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:18:06.973 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:18:06.973 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:18:06.973 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:18:06.973 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:18:06.973 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:18:06.973 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:18:06.973 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:18:06.973 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:18:06.976 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:18:06.976 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:18:06.976 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:18:06.976 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:18:06.976 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:18:06.976 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:18:06.976 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:18:06.976 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:18:06.976 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:18:06.977 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:18:06.977 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:18:06.977 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:18:06.977 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:18:06.977 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:18:06.977 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:18:06.977 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:18:06.977 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:18:06.977 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:18:06.977 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:18:06.977 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:18:06.977 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:18:06.977 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:18:06.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:18:06.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:18:06.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:18:06.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:18:06.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:18:06.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:18:06.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:18:06.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:18:06.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:18:06.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:18:07.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:18:07.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:18:07.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:18:07.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:18:07.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:18:07.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:18:07.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:07.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:07.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:07.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:18:07.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:07.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:07.128 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:07.128 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:18:07.128 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:18:07.128 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:18:07.128 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:18:07.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:07.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:18:07.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:07.184 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 4 2024-09-11T06:18:07.184 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:18:07.185 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:18:07.185 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:18:07.185 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=19, tag_tid=2, entry_tid=17495], [object_number=18, tag_tid=2, entry_tid=17494], [object_number=17, tag_tid=2, entry_tid=17493], [object_number=16, tag_tid=2, entry_tid=17492]]], state=connected] 2024-09-11T06:18:07.188 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:07.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:18:07.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:18:07.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:07.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:07.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:07.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:18:07.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:07.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:07.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:07.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:18:07.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:18:07.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:18:07.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:18:07.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:07.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:18:07.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:07.238 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 4 2024-09-11T06:18:07.238 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:18:07.238 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:18:07.238 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:18:07.238 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=19, tag_tid=2, entry_tid=17495], [object_number=18, tag_tid=2, entry_tid=17494], [object_number=17, tag_tid=2, entry_tid=17493], [object_number=16, tag_tid=2, entry_tid=17492]]], state=connected] 2024-09-11T06:18:07.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:07.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=19, tag_tid=2, entry_tid=17495' 2024-09-11T06:18:07.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=21, tag_tid=3, entry_tid=1' -a 'object_number=21, tag_tid=3, entry_tid=1' = 'object_number=19, tag_tid=2, entry_tid=17495' 2024-09-11T06:18:07.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=19, tag_tid=2, entry_tid=17495' '!=' 'object_number=18, tag_tid=2, entry_tid=17382' 2024-09-11T06:18:07.244 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:18:07.244 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=19, tag_tid=2, entry_tid=17495' = 'object_number=18, tag_tid=2, entry_tid=17382' 2024-09-11T06:18:07.244 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=19, tag_tid=2, entry_tid=17495' 2024-09-11T06:18:07.244 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:18:07.244 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:18:07.244 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:07.244 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=3 2024-09-11T06:18:07.245 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=19, tag_tid=2, entry_tid=17495' 2024-09-11T06:18:07.245 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:18:07.245 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:07.246 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=2 2024-09-11T06:18:07.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:18:07.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:18:07.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:07.248 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:18:07.248 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=19, tag_tid=2, entry_tid=17495' 2024-09-11T06:18:07.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:18:07.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:07.250 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=17495 2024-09-11T06:18:07.250 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 3 = 2 -a 1 -le 17495 2024-09-11T06:18:07.250 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:18:07.250 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:18:07.250 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:18:07.452 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:18:07.452 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:18:07.452 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:18:07.452 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:18:07.452 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:18:07.452 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:18:07.452 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:18:07.452 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:18:07.452 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:18:07.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:18:07.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:18:07.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:18:07.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:18:07.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:18:07.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:18:07.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:18:07.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:18:07.455 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:18:07.456 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:18:07.456 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:18:07.456 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:18:07.456 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:18:07.456 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:18:07.456 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:18:07.456 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:18:07.456 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:18:07.456 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:18:07.456 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:18:07.456 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:18:07.456 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:18:07.456 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:18:07.456 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:18:07.457 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:18:07.457 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:18:07.457 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:18:07.457 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:18:07.457 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:18:07.457 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:18:07.457 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:18:07.457 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:18:07.457 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:18:07.457 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:18:07.457 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:18:07.457 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:18:07.457 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:18:07.457 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:18:07.458 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:18:07.458 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:18:07.458 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:18:07.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:18:07.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:18:07.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:18:07.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:18:07.616 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:18:07.617 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:18:07.617 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:07.617 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:07.617 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:07.617 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:18:07.617 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:07.617 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:07.617 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:07.617 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:18:07.618 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:18:07.618 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:18:07.618 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:18:07.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:07.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:18:07.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:07.664 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 4 2024-09-11T06:18:07.664 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:18:07.664 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:18:07.664 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:18:07.664 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=18, tag_tid=2, entry_tid=17614], [object_number=17, tag_tid=2, entry_tid=17613], [object_number=16, tag_tid=2, entry_tid=17612], [object_number=19, tag_tid=2, entry_tid=17611]]], state=connected] 2024-09-11T06:18:07.668 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:07.669 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:18:07.670 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:18:07.670 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:07.670 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:07.670 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:07.670 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:18:07.671 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:07.671 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:07.671 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:07.671 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:18:07.671 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:18:07.671 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:18:07.671 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:18:07.672 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:07.672 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:18:07.672 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:07.716 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 4 2024-09-11T06:18:07.716 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:18:07.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:18:07.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:18:07.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=18, tag_tid=2, entry_tid=17614], [object_number=17, tag_tid=2, entry_tid=17613], [object_number=16, tag_tid=2, entry_tid=17612], [object_number=19, tag_tid=2, entry_tid=17611]]], state=connected] 2024-09-11T06:18:07.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:07.722 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=18, tag_tid=2, entry_tid=17614' 2024-09-11T06:18:07.722 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=21, tag_tid=3, entry_tid=1' -a 'object_number=21, tag_tid=3, entry_tid=1' = 'object_number=18, tag_tid=2, entry_tid=17614' 2024-09-11T06:18:07.722 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=18, tag_tid=2, entry_tid=17614' '!=' 'object_number=19, tag_tid=2, entry_tid=17495' 2024-09-11T06:18:07.722 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:18:07.722 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=18, tag_tid=2, entry_tid=17614' = 'object_number=19, tag_tid=2, entry_tid=17495' 2024-09-11T06:18:07.722 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=18, tag_tid=2, entry_tid=17614' 2024-09-11T06:18:07.722 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:18:07.722 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:18:07.723 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:07.723 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=3 2024-09-11T06:18:07.724 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=18, tag_tid=2, entry_tid=17614' 2024-09-11T06:18:07.724 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:18:07.724 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:07.725 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=2 2024-09-11T06:18:07.725 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:18:07.726 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:18:07.726 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:07.727 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:18:07.727 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=18, tag_tid=2, entry_tid=17614' 2024-09-11T06:18:07.727 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:18:07.727 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:07.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=17614 2024-09-11T06:18:07.729 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 3 = 2 -a 1 -le 17614 2024-09-11T06:18:07.729 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:18:07.729 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:18:07.729 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:18:07.931 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:18:07.931 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:18:07.931 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:18:07.931 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:18:07.931 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:18:07.931 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:18:07.932 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:18:07.932 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:18:07.932 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:18:07.932 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:18:07.932 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:18:07.932 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:18:07.932 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:18:07.932 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:18:07.932 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:18:07.933 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:18:07.933 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:18:07.936 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:18:07.936 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:18:07.936 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:18:07.936 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:18:07.936 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:18:07.936 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:18:07.936 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:18:07.936 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:18:07.936 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:18:07.936 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:18:07.936 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:18:07.937 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:18:07.937 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:18:07.937 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:18:07.937 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:18:07.937 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:18:07.937 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:18:07.937 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:18:07.937 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:18:07.937 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:18:07.937 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:18:07.937 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:18:07.937 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:18:07.937 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:18:07.937 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:18:07.937 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:18:07.938 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:18:07.938 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:18:07.938 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:18:07.938 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:18:07.938 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:18:07.938 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:18:07.961 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:18:07.962 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:18:07.962 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:18:07.962 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:18:08.090 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:18:08.090 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:18:08.090 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:08.091 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:08.091 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:08.091 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:18:08.091 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:08.091 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:08.091 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:08.091 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:18:08.091 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:18:08.092 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:18:08.092 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:18:08.092 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:08.093 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:18:08.093 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:08.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 4 2024-09-11T06:18:08.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:18:08.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:18:08.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:18:08.143 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=16, tag_tid=2, entry_tid=17732], [object_number=19, tag_tid=2, entry_tid=17731], [object_number=18, tag_tid=2, entry_tid=17730], [object_number=17, tag_tid=2, entry_tid=17729]]], state=connected] 2024-09-11T06:18:08.146 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:08.147 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:18:08.148 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:18:08.148 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:08.149 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:08.149 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:08.149 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:18:08.149 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:08.149 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:08.149 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:08.149 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:18:08.149 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:18:08.149 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:18:08.149 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:18:08.150 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:08.150 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:18:08.150 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:08.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 4 2024-09-11T06:18:08.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:18:08.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:18:08.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:18:08.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=16, tag_tid=2, entry_tid=17732], [object_number=19, tag_tid=2, entry_tid=17731], [object_number=18, tag_tid=2, entry_tid=17730], [object_number=17, tag_tid=2, entry_tid=17729]]], state=connected] 2024-09-11T06:18:08.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:08.200 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=16, tag_tid=2, entry_tid=17732' 2024-09-11T06:18:08.201 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=21, tag_tid=3, entry_tid=1' -a 'object_number=21, tag_tid=3, entry_tid=1' = 'object_number=16, tag_tid=2, entry_tid=17732' 2024-09-11T06:18:08.201 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=16, tag_tid=2, entry_tid=17732' '!=' 'object_number=18, tag_tid=2, entry_tid=17614' 2024-09-11T06:18:08.201 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:18:08.201 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=16, tag_tid=2, entry_tid=17732' = 'object_number=18, tag_tid=2, entry_tid=17614' 2024-09-11T06:18:08.201 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=16, tag_tid=2, entry_tid=17732' 2024-09-11T06:18:08.201 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:18:08.201 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:18:08.201 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:08.201 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=3 2024-09-11T06:18:08.202 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=16, tag_tid=2, entry_tid=17732' 2024-09-11T06:18:08.202 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:18:08.202 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:08.203 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=2 2024-09-11T06:18:08.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:18:08.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:18:08.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:08.205 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:18:08.206 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=16, tag_tid=2, entry_tid=17732' 2024-09-11T06:18:08.206 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:18:08.206 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:08.207 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=17732 2024-09-11T06:18:08.207 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 3 = 2 -a 1 -le 17732 2024-09-11T06:18:08.207 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:18:08.207 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:18:08.207 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:18:08.408 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:18:08.409 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:18:08.409 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:18:08.409 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:18:08.409 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:18:08.409 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:18:08.409 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:18:08.409 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:18:08.409 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:18:08.409 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:18:08.409 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:18:08.409 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:18:08.409 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:18:08.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:18:08.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:18:08.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:18:08.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:18:08.413 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:18:08.413 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:18:08.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:18:08.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:18:08.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:18:08.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:18:08.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:18:08.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:18:08.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:18:08.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:18:08.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:18:08.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:18:08.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:18:08.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:18:08.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:18:08.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:18:08.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:18:08.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:18:08.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:18:08.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:18:08.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:18:08.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:18:08.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:18:08.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:18:08.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:18:08.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:18:08.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:18:08.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:18:08.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:18:08.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:18:08.416 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:18:08.416 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:18:08.438 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:18:08.439 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:18:08.439 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:18:08.439 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:18:08.565 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:18:08.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:18:08.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:08.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:08.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:08.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:18:08.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:08.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:08.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:08.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:18:08.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:18:08.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:18:08.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:18:08.569 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:08.569 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:18:08.569 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:08.615 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 4 2024-09-11T06:18:08.615 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:18:08.616 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:18:08.616 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:18:08.616 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=16, tag_tid=2, entry_tid=17844], [object_number=19, tag_tid=2, entry_tid=17843], [object_number=18, tag_tid=2, entry_tid=17842], [object_number=17, tag_tid=2, entry_tid=17841]]], state=connected] 2024-09-11T06:18:08.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:08.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:18:08.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:18:08.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:08.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:08.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:08.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:18:08.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:08.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:08.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:08.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:18:08.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:18:08.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:18:08.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:18:08.623 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:08.623 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:18:08.623 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:08.668 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 4 2024-09-11T06:18:08.668 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:18:08.669 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:18:08.669 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:18:08.669 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=16, tag_tid=2, entry_tid=17844], [object_number=19, tag_tid=2, entry_tid=17843], [object_number=18, tag_tid=2, entry_tid=17842], [object_number=17, tag_tid=2, entry_tid=17841]]], state=connected] 2024-09-11T06:18:08.672 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:08.674 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=16, tag_tid=2, entry_tid=17844' 2024-09-11T06:18:08.674 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=21, tag_tid=3, entry_tid=1' -a 'object_number=21, tag_tid=3, entry_tid=1' = 'object_number=16, tag_tid=2, entry_tid=17844' 2024-09-11T06:18:08.674 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=16, tag_tid=2, entry_tid=17844' '!=' 'object_number=16, tag_tid=2, entry_tid=17732' 2024-09-11T06:18:08.674 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:18:08.674 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=16, tag_tid=2, entry_tid=17844' = 'object_number=16, tag_tid=2, entry_tid=17732' 2024-09-11T06:18:08.674 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=16, tag_tid=2, entry_tid=17844' 2024-09-11T06:18:08.675 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:18:08.675 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:18:08.675 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:08.675 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=3 2024-09-11T06:18:08.676 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=16, tag_tid=2, entry_tid=17844' 2024-09-11T06:18:08.676 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:18:08.676 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:08.677 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=2 2024-09-11T06:18:08.677 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:18:08.678 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:18:08.678 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:08.678 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:18:08.679 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=16, tag_tid=2, entry_tid=17844' 2024-09-11T06:18:08.679 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:18:08.679 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:08.681 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=17844 2024-09-11T06:18:08.681 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 3 = 2 -a 1 -le 17844 2024-09-11T06:18:08.681 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:18:08.681 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:18:08.681 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:18:08.883 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:18:08.883 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:18:08.883 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:18:08.883 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:18:08.883 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:18:08.883 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:18:08.883 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:18:08.883 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:18:08.883 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:18:08.883 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:18:08.883 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:18:08.884 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:18:08.884 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:18:08.884 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:18:08.884 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:18:08.884 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:18:08.885 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:18:08.887 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:18:08.887 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:18:08.887 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:18:08.887 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:18:08.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:18:08.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:18:08.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:18:08.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:18:08.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:18:08.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:18:08.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:18:08.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:18:08.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:18:08.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:18:08.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:18:08.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:18:08.889 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:18:08.889 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:18:08.890 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:18:08.890 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:18:08.890 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:18:08.890 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:18:08.890 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:18:08.890 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:18:08.890 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:18:08.890 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:18:08.890 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:18:08.890 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:18:08.890 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:18:08.890 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:18:08.890 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:18:08.891 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:18:08.915 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:18:08.915 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:18:08.915 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:18:08.916 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:18:09.041 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:18:09.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:18:09.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:09.043 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:09.043 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:09.043 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:18:09.043 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:09.043 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:09.043 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:09.043 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:18:09.043 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:18:09.044 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:18:09.044 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:18:09.045 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:09.045 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:18:09.045 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:09.092 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 4 2024-09-11T06:18:09.092 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:18:09.092 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:18:09.092 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:18:09.092 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=16, tag_tid=2, entry_tid=17948], [object_number=19, tag_tid=2, entry_tid=17947], [object_number=18, tag_tid=2, entry_tid=17946], [object_number=17, tag_tid=2, entry_tid=17945]]], state=connected] 2024-09-11T06:18:09.095 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:09.098 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:18:09.098 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:18:09.098 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:09.098 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:09.098 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:09.098 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:18:09.098 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:09.098 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:09.098 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:09.098 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:18:09.098 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:18:09.099 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:18:09.099 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:18:09.099 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:09.099 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:18:09.099 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:09.147 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 4 2024-09-11T06:18:09.147 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:18:09.147 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:18:09.147 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:18:09.147 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=16, tag_tid=2, entry_tid=17948], [object_number=19, tag_tid=2, entry_tid=17947], [object_number=18, tag_tid=2, entry_tid=17946], [object_number=17, tag_tid=2, entry_tid=17945]]], state=connected] 2024-09-11T06:18:09.150 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:09.153 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=16, tag_tid=2, entry_tid=17948' 2024-09-11T06:18:09.153 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=21, tag_tid=3, entry_tid=1' -a 'object_number=21, tag_tid=3, entry_tid=1' = 'object_number=16, tag_tid=2, entry_tid=17948' 2024-09-11T06:18:09.153 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=16, tag_tid=2, entry_tid=17948' '!=' 'object_number=16, tag_tid=2, entry_tid=17844' 2024-09-11T06:18:09.153 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:18:09.153 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=16, tag_tid=2, entry_tid=17948' = 'object_number=16, tag_tid=2, entry_tid=17844' 2024-09-11T06:18:09.153 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=16, tag_tid=2, entry_tid=17948' 2024-09-11T06:18:09.153 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:18:09.153 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:18:09.153 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:09.154 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=3 2024-09-11T06:18:09.154 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=16, tag_tid=2, entry_tid=17948' 2024-09-11T06:18:09.154 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:18:09.154 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:09.155 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=2 2024-09-11T06:18:09.156 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:18:09.156 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:18:09.156 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:09.157 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:18:09.157 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=16, tag_tid=2, entry_tid=17948' 2024-09-11T06:18:09.157 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:18:09.157 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:09.159 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=17948 2024-09-11T06:18:09.159 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 3 = 2 -a 1 -le 17948 2024-09-11T06:18:09.159 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:18:09.159 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:18:09.159 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:18:09.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:18:09.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:18:09.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:18:09.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:18:09.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:18:09.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:18:09.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:18:09.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:18:09.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:18:09.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:18:09.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:18:09.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:18:09.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:18:09.362 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:18:09.362 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:18:09.362 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:18:09.362 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:18:09.365 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:18:09.365 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:18:09.365 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:18:09.365 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:18:09.365 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:18:09.365 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:18:09.365 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:18:09.365 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:18:09.365 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:18:09.365 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:18:09.365 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:18:09.365 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:18:09.365 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:18:09.366 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:18:09.366 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:18:09.366 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:18:09.367 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:18:09.367 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:18:09.367 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:18:09.367 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:18:09.368 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:18:09.368 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:18:09.368 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:18:09.368 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:18:09.368 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:18:09.368 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:18:09.368 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:18:09.368 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:18:09.368 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:18:09.368 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:18:09.368 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:18:09.368 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:18:09.391 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:18:09.391 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:18:09.391 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:18:09.391 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:18:09.528 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:18:09.530 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:18:09.530 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:09.531 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:09.531 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:09.531 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:18:09.531 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:09.531 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:09.531 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:09.531 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:18:09.532 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:18:09.532 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:18:09.532 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:18:09.533 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:09.533 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:18:09.534 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:09.581 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 5 2024-09-11T06:18:09.581 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:18:09.581 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:18:09.581 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:18:09.581 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=22, tag_tid=2, entry_tid=18010], [object_number=21, tag_tid=2, entry_tid=18009], [object_number=20, tag_tid=2, entry_tid=18008], [object_number=23, tag_tid=2, entry_tid=18007]]], state=connected] 2024-09-11T06:18:09.585 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:09.587 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:18:09.587 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:18:09.587 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:09.587 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:09.587 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:09.587 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:18:09.587 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:09.587 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:09.587 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:09.588 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:18:09.588 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:18:09.588 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:18:09.588 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:18:09.588 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:09.588 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:18:09.589 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:09.633 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 5 2024-09-11T06:18:09.633 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:18:09.633 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:18:09.633 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:18:09.633 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=22, tag_tid=2, entry_tid=18010], [object_number=21, tag_tid=2, entry_tid=18009], [object_number=20, tag_tid=2, entry_tid=18008], [object_number=23, tag_tid=2, entry_tid=18007]]], state=connected] 2024-09-11T06:18:09.636 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:09.638 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=22, tag_tid=2, entry_tid=18010' 2024-09-11T06:18:09.638 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=21, tag_tid=3, entry_tid=1' -a 'object_number=21, tag_tid=3, entry_tid=1' = 'object_number=22, tag_tid=2, entry_tid=18010' 2024-09-11T06:18:09.638 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=22, tag_tid=2, entry_tid=18010' '!=' 'object_number=16, tag_tid=2, entry_tid=17948' 2024-09-11T06:18:09.638 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:18:09.638 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=22, tag_tid=2, entry_tid=18010' = 'object_number=16, tag_tid=2, entry_tid=17948' 2024-09-11T06:18:09.638 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=22, tag_tid=2, entry_tid=18010' 2024-09-11T06:18:09.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:18:09.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:18:09.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:09.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=3 2024-09-11T06:18:09.640 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=22, tag_tid=2, entry_tid=18010' 2024-09-11T06:18:09.640 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:18:09.640 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:09.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=2 2024-09-11T06:18:09.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:18:09.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:18:09.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:09.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:18:09.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=22, tag_tid=2, entry_tid=18010' 2024-09-11T06:18:09.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:18:09.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:09.645 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=18010 2024-09-11T06:18:09.645 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 3 = 2 -a 1 -le 18010 2024-09-11T06:18:09.645 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:18:09.645 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:18:09.645 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:18:09.846 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:18:09.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:18:09.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:18:09.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:18:09.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:18:09.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:18:09.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:18:09.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:18:09.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:18:09.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:18:09.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:18:09.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:18:09.848 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:18:09.848 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:18:09.848 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:18:09.848 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:18:09.848 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:18:09.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:18:09.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:18:09.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:18:09.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:18:09.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:18:09.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:18:09.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:18:09.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:18:09.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:18:09.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:18:09.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:18:09.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:18:09.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:18:09.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:18:09.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:18:09.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:18:09.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:18:09.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:18:09.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:18:09.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:18:09.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:18:09.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:18:09.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:18:09.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:18:09.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:18:09.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:18:09.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:18:09.854 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:18:09.854 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:18:09.854 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:18:09.854 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:18:09.854 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:18:09.873 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:18:09.873 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:18:09.874 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:18:09.874 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:18:10.009 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:18:10.011 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:18:10.011 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:10.011 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:10.011 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:10.011 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:18:10.011 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:10.011 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:10.011 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:10.011 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:18:10.011 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:18:10.012 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:18:10.012 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:18:10.013 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:10.013 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:10.013 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:18:10.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 5 2024-09-11T06:18:10.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:18:10.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:18:10.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:18:10.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=23, tag_tid=2, entry_tid=18119], [object_number=22, tag_tid=2, entry_tid=18118], [object_number=21, tag_tid=2, entry_tid=18117], [object_number=20, tag_tid=2, entry_tid=18116]]], state=connected] 2024-09-11T06:18:10.069 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:10.071 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:18:10.072 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:18:10.072 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:10.072 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:10.072 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:10.072 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:18:10.072 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:10.072 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:10.072 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:10.072 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:18:10.072 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:18:10.073 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:18:10.073 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:18:10.073 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:10.073 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:18:10.073 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:10.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 5 2024-09-11T06:18:10.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:18:10.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:18:10.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:18:10.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=23, tag_tid=2, entry_tid=18119], [object_number=22, tag_tid=2, entry_tid=18118], [object_number=21, tag_tid=2, entry_tid=18117], [object_number=20, tag_tid=2, entry_tid=18116]]], state=connected] 2024-09-11T06:18:10.122 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:10.124 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=23, tag_tid=2, entry_tid=18119' 2024-09-11T06:18:10.124 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=21, tag_tid=3, entry_tid=1' -a 'object_number=21, tag_tid=3, entry_tid=1' = 'object_number=23, tag_tid=2, entry_tid=18119' 2024-09-11T06:18:10.124 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=23, tag_tid=2, entry_tid=18119' '!=' 'object_number=22, tag_tid=2, entry_tid=18010' 2024-09-11T06:18:10.124 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:18:10.124 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=23, tag_tid=2, entry_tid=18119' = 'object_number=22, tag_tid=2, entry_tid=18010' 2024-09-11T06:18:10.124 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=23, tag_tid=2, entry_tid=18119' 2024-09-11T06:18:10.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:18:10.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:18:10.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:10.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=3 2024-09-11T06:18:10.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=23, tag_tid=2, entry_tid=18119' 2024-09-11T06:18:10.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:18:10.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:10.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=2 2024-09-11T06:18:10.128 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:18:10.128 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:18:10.128 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:10.129 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:18:10.129 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=23, tag_tid=2, entry_tid=18119' 2024-09-11T06:18:10.129 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:18:10.129 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=18119 2024-09-11T06:18:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 3 = 2 -a 1 -le 18119 2024-09-11T06:18:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:18:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:18:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:18:10.333 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:18:10.333 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:18:10.333 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:18:10.333 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:18:10.333 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:18:10.333 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:18:10.333 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:18:10.333 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:18:10.333 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:18:10.334 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:18:10.334 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:18:10.334 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:18:10.334 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:18:10.334 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:18:10.334 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:18:10.334 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:18:10.334 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:18:10.338 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:18:10.338 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:18:10.338 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:18:10.338 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:18:10.338 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:18:10.339 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:18:10.339 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:18:10.339 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:18:10.339 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:18:10.339 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:18:10.339 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:18:10.339 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:18:10.339 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:18:10.339 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:18:10.339 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:18:10.339 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:18:10.339 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:18:10.339 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:18:10.339 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:18:10.340 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:18:10.340 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:18:10.340 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:18:10.340 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:18:10.340 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:18:10.340 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:18:10.340 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:18:10.340 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:18:10.340 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:18:10.340 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:18:10.340 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:18:10.340 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:18:10.340 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:18:10.362 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:18:10.363 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:18:10.363 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:18:10.363 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:18:10.499 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:18:10.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:18:10.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:10.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:10.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:10.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:18:10.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:10.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:10.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:10.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:18:10.501 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:18:10.501 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:18:10.501 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:18:10.503 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:10.504 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:18:10.504 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:10.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 5 2024-09-11T06:18:10.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:18:10.555 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:18:10.555 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:18:10.555 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=21, tag_tid=2, entry_tid=18233], [object_number=20, tag_tid=2, entry_tid=18232], [object_number=23, tag_tid=2, entry_tid=18231], [object_number=22, tag_tid=2, entry_tid=18230]]], state=connected] 2024-09-11T06:18:10.558 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:10.559 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:18:10.560 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:18:10.560 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:10.560 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:10.560 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:10.561 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:18:10.561 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:10.561 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:10.561 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:10.561 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:18:10.561 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:18:10.561 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:18:10.561 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:18:10.562 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:10.562 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:18:10.562 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:10.607 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 5 2024-09-11T06:18:10.607 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:18:10.607 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:18:10.607 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:18:10.607 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=21, tag_tid=2, entry_tid=18233], [object_number=20, tag_tid=2, entry_tid=18232], [object_number=23, tag_tid=2, entry_tid=18231], [object_number=22, tag_tid=2, entry_tid=18230]]], state=connected] 2024-09-11T06:18:10.610 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:10.612 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=21, tag_tid=2, entry_tid=18233' 2024-09-11T06:18:10.612 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=21, tag_tid=3, entry_tid=1' -a 'object_number=21, tag_tid=3, entry_tid=1' = 'object_number=21, tag_tid=2, entry_tid=18233' 2024-09-11T06:18:10.612 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=21, tag_tid=2, entry_tid=18233' '!=' 'object_number=23, tag_tid=2, entry_tid=18119' 2024-09-11T06:18:10.612 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:18:10.612 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=21, tag_tid=2, entry_tid=18233' = 'object_number=23, tag_tid=2, entry_tid=18119' 2024-09-11T06:18:10.612 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=21, tag_tid=2, entry_tid=18233' 2024-09-11T06:18:10.613 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:18:10.613 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:18:10.613 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:10.614 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=3 2024-09-11T06:18:10.614 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=2, entry_tid=18233' 2024-09-11T06:18:10.614 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:18:10.615 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:10.615 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=2 2024-09-11T06:18:10.616 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:18:10.616 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:18:10.616 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:10.617 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:18:10.618 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=2, entry_tid=18233' 2024-09-11T06:18:10.618 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:18:10.618 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:10.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=18233 2024-09-11T06:18:10.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 3 = 2 -a 1 -le 18233 2024-09-11T06:18:10.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:18:10.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:18:10.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:18:10.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:18:10.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:18:10.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:18:10.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:18:10.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:18:10.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:18:10.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:18:10.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:18:10.822 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:18:10.822 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:18:10.822 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:18:10.822 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:18:10.822 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:18:10.822 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:18:10.822 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:18:10.822 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:18:10.823 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:18:10.826 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:18:10.826 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:18:10.826 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:18:10.826 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:18:10.826 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:18:10.826 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:18:10.826 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:18:10.826 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:18:10.826 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:18:10.826 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:18:10.826 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:18:10.826 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:18:10.827 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:18:10.827 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:18:10.827 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:18:10.827 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:18:10.827 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:18:10.827 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:18:10.827 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:18:10.827 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:18:10.827 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:18:10.827 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:18:10.827 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:18:10.827 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:18:10.827 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:18:10.828 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:18:10.828 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:18:10.828 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:18:10.828 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:18:10.828 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:18:10.828 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:18:10.828 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:18:10.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:18:10.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:18:10.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:18:10.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:18:10.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:18:10.985 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:18:10.985 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:10.986 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:10.986 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:10.986 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:18:10.986 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:10.986 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:10.986 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:10.986 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:18:10.987 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:18:10.987 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:18:10.987 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:18:10.989 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:10.989 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:18:10.989 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:11.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 5 2024-09-11T06:18:11.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:18:11.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:18:11.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:18:11.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=20, tag_tid=2, entry_tid=18348], [object_number=23, tag_tid=2, entry_tid=18347], [object_number=22, tag_tid=2, entry_tid=18346], [object_number=21, tag_tid=2, entry_tid=18345]]], state=connected] 2024-09-11T06:18:11.037 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:11.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:18:11.040 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:18:11.040 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:11.040 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:11.040 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:11.040 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:18:11.040 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:11.040 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:11.040 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:11.040 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:18:11.041 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:18:11.041 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:18:11.041 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:18:11.041 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:11.041 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:18:11.041 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:11.088 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 5 2024-09-11T06:18:11.088 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:18:11.088 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:18:11.088 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:18:11.088 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=20, tag_tid=2, entry_tid=18348], [object_number=23, tag_tid=2, entry_tid=18347], [object_number=22, tag_tid=2, entry_tid=18346], [object_number=21, tag_tid=2, entry_tid=18345]]], state=connected] 2024-09-11T06:18:11.092 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:11.093 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=20, tag_tid=2, entry_tid=18348' 2024-09-11T06:18:11.093 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=21, tag_tid=3, entry_tid=1' -a 'object_number=21, tag_tid=3, entry_tid=1' = 'object_number=20, tag_tid=2, entry_tid=18348' 2024-09-11T06:18:11.093 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=20, tag_tid=2, entry_tid=18348' '!=' 'object_number=21, tag_tid=2, entry_tid=18233' 2024-09-11T06:18:11.093 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:18:11.093 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=20, tag_tid=2, entry_tid=18348' = 'object_number=21, tag_tid=2, entry_tid=18233' 2024-09-11T06:18:11.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=20, tag_tid=2, entry_tid=18348' 2024-09-11T06:18:11.096 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:18:11.096 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:18:11.096 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:11.096 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=3 2024-09-11T06:18:11.097 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=20, tag_tid=2, entry_tid=18348' 2024-09-11T06:18:11.097 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:18:11.097 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:11.097 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=2 2024-09-11T06:18:11.098 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:18:11.098 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:18:11.098 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:11.098 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:18:11.099 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=20, tag_tid=2, entry_tid=18348' 2024-09-11T06:18:11.099 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:18:11.099 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:11.100 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=18348 2024-09-11T06:18:11.100 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 3 = 2 -a 1 -le 18348 2024-09-11T06:18:11.100 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:18:11.100 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:18:11.100 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:18:11.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:18:11.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:18:11.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:18:11.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:18:11.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:18:11.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:18:11.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:18:11.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:18:11.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:18:11.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:18:11.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:18:11.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:18:11.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:18:11.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:18:11.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:18:11.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:18:11.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:18:11.307 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:18:11.307 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:18:11.307 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:18:11.307 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:18:11.307 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:18:11.307 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:18:11.307 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:18:11.307 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:18:11.307 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:18:11.307 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:18:11.307 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:18:11.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:18:11.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:18:11.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:18:11.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:18:11.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:18:11.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:18:11.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:18:11.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:18:11.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:18:11.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:18:11.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:18:11.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:18:11.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:18:11.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:18:11.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:18:11.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:18:11.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:18:11.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:18:11.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:18:11.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:18:11.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:18:11.331 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:18:11.332 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:18:11.332 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:18:11.332 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:18:11.646 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:18:11.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:18:11.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:11.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:11.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:11.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:18:11.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:11.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:11.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:11.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:18:11.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:18:11.649 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:18:11.649 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:18:11.652 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:11.652 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:18:11.652 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:11.701 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 5 2024-09-11T06:18:11.701 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:18:11.701 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:18:11.701 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:18:11.701 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=22, tag_tid=2, entry_tid=18458], [object_number=21, tag_tid=2, entry_tid=18457], [object_number=20, tag_tid=2, entry_tid=18456], [object_number=23, tag_tid=2, entry_tid=18455]]], state=connected] 2024-09-11T06:18:11.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:11.707 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:18:11.707 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:18:11.707 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:11.707 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:11.707 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:11.707 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:18:11.707 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:11.707 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:11.708 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:11.708 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:18:11.708 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:18:11.708 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:18:11.708 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:18:11.708 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:11.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:18:11.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:11.754 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 5 2024-09-11T06:18:11.754 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:18:11.754 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:18:11.754 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:18:11.754 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=22, tag_tid=2, entry_tid=18458], [object_number=21, tag_tid=2, entry_tid=18457], [object_number=20, tag_tid=2, entry_tid=18456], [object_number=23, tag_tid=2, entry_tid=18455]]], state=connected] 2024-09-11T06:18:11.758 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:11.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=22, tag_tid=2, entry_tid=18458' 2024-09-11T06:18:11.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=21, tag_tid=3, entry_tid=1' -a 'object_number=21, tag_tid=3, entry_tid=1' = 'object_number=22, tag_tid=2, entry_tid=18458' 2024-09-11T06:18:11.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=22, tag_tid=2, entry_tid=18458' '!=' 'object_number=20, tag_tid=2, entry_tid=18348' 2024-09-11T06:18:11.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:18:11.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=22, tag_tid=2, entry_tid=18458' = 'object_number=20, tag_tid=2, entry_tid=18348' 2024-09-11T06:18:11.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=22, tag_tid=2, entry_tid=18458' 2024-09-11T06:18:11.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:18:11.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:18:11.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:11.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=3 2024-09-11T06:18:11.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=22, tag_tid=2, entry_tid=18458' 2024-09-11T06:18:11.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:18:11.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:11.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=2 2024-09-11T06:18:11.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:18:11.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:18:11.764 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:11.764 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:18:11.765 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=22, tag_tid=2, entry_tid=18458' 2024-09-11T06:18:11.765 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:18:11.765 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:11.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=18458 2024-09-11T06:18:11.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 3 = 2 -a 1 -le 18458 2024-09-11T06:18:11.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:18:11.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:18:11.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:18:11.968 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:18:11.968 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:18:11.969 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:18:11.969 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:18:11.969 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:18:11.969 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:18:11.969 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:18:11.969 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:18:11.969 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:18:11.969 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:18:11.969 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:18:11.969 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:18:11.969 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:18:11.969 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:18:11.970 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:18:11.970 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:18:11.970 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:18:11.972 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:18:11.972 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:18:11.972 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:18:11.972 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:18:11.972 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:18:11.973 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:18:11.973 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:18:11.973 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:18:11.973 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:18:11.973 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:18:11.973 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:18:11.973 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:18:11.973 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:18:11.973 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:18:11.973 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:18:11.973 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:18:11.973 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:18:11.973 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:18:11.973 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:18:11.973 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:18:11.974 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:18:11.974 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:18:11.974 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:18:11.974 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:18:11.974 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:18:11.974 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:18:11.974 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:18:11.974 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:18:11.974 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:18:11.974 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:18:11.974 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:18:11.974 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:18:11.999 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:18:12.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:18:12.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:18:12.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:18:12.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:18:12.128 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:18:12.128 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:12.129 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:12.129 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:12.129 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:18:12.129 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:12.129 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:12.129 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:12.129 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:18:12.129 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:18:12.130 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:18:12.130 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:18:12.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:12.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:18:12.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:12.176 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 5 2024-09-11T06:18:12.176 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:18:12.176 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:18:12.177 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:18:12.177 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=21, tag_tid=2, entry_tid=18577], [object_number=20, tag_tid=2, entry_tid=18576], [object_number=23, tag_tid=2, entry_tid=18575], [object_number=22, tag_tid=2, entry_tid=18574]]], state=connected] 2024-09-11T06:18:12.180 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:12.181 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:18:12.182 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:18:12.182 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:12.182 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:12.182 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:12.183 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:18:12.183 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:12.183 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:12.183 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:12.183 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:18:12.183 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:18:12.183 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:18:12.183 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:18:12.184 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:12.184 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:18:12.184 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:12.229 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 5 2024-09-11T06:18:12.229 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:18:12.229 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:18:12.229 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:18:12.229 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=21, tag_tid=2, entry_tid=18577], [object_number=20, tag_tid=2, entry_tid=18576], [object_number=23, tag_tid=2, entry_tid=18575], [object_number=22, tag_tid=2, entry_tid=18574]]], state=connected] 2024-09-11T06:18:12.232 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:12.234 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=21, tag_tid=2, entry_tid=18577' 2024-09-11T06:18:12.235 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=21, tag_tid=3, entry_tid=1' -a 'object_number=21, tag_tid=3, entry_tid=1' = 'object_number=21, tag_tid=2, entry_tid=18577' 2024-09-11T06:18:12.235 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=21, tag_tid=2, entry_tid=18577' '!=' 'object_number=22, tag_tid=2, entry_tid=18458' 2024-09-11T06:18:12.235 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:18:12.235 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=21, tag_tid=2, entry_tid=18577' = 'object_number=22, tag_tid=2, entry_tid=18458' 2024-09-11T06:18:12.235 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=21, tag_tid=2, entry_tid=18577' 2024-09-11T06:18:12.235 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:18:12.235 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:18:12.235 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:12.235 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=3 2024-09-11T06:18:12.236 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=2, entry_tid=18577' 2024-09-11T06:18:12.236 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:18:12.236 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:12.237 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=2 2024-09-11T06:18:12.238 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:18:12.238 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:18:12.238 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:12.239 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:18:12.239 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=2, entry_tid=18577' 2024-09-11T06:18:12.239 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:18:12.240 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:12.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=18577 2024-09-11T06:18:12.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 3 = 2 -a 1 -le 18577 2024-09-11T06:18:12.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:18:12.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:18:12.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:18:12.443 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:18:12.443 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:18:12.443 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:18:12.443 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:18:12.443 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:18:12.443 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:18:12.443 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:18:12.443 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:18:12.443 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:18:12.443 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:18:12.443 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:18:12.443 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:18:12.444 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:18:12.444 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:18:12.444 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:18:12.444 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:18:12.444 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:18:12.447 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:18:12.447 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:18:12.447 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:18:12.447 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:18:12.447 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:18:12.447 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:18:12.447 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:18:12.447 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:18:12.447 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:18:12.447 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:18:12.447 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:18:12.447 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:18:12.448 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:18:12.448 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:18:12.448 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:18:12.448 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:18:12.448 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:18:12.448 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:18:12.448 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:18:12.448 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:18:12.448 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:18:12.448 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:18:12.448 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:18:12.450 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:18:12.450 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:18:12.450 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:18:12.450 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:18:12.450 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:18:12.450 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:18:12.450 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:18:12.450 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:18:12.451 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:18:12.473 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:18:12.473 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:18:12.473 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:18:12.473 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:18:12.603 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:18:12.603 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:18:12.603 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:12.603 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:12.603 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:12.604 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:18:12.604 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:12.604 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:12.604 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:12.604 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:18:12.604 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:18:12.605 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:18:12.605 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:18:12.606 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:12.606 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:18:12.606 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:12.652 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 5 2024-09-11T06:18:12.652 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:18:12.652 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:18:12.652 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:18:12.652 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=21, tag_tid=2, entry_tid=18681], [object_number=20, tag_tid=2, entry_tid=18680], [object_number=23, tag_tid=2, entry_tid=18679], [object_number=22, tag_tid=2, entry_tid=18678]]], state=connected] 2024-09-11T06:18:12.655 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:12.656 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:18:12.658 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:18:12.658 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:12.658 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:12.658 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:12.658 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:18:12.658 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:12.658 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:12.658 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:12.658 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:18:12.658 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:18:12.658 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:18:12.658 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:18:12.659 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:12.659 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:18:12.659 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:12.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 5 2024-09-11T06:18:12.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:18:12.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:18:12.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:18:12.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=21, tag_tid=2, entry_tid=18681], [object_number=20, tag_tid=2, entry_tid=18680], [object_number=23, tag_tid=2, entry_tid=18679], [object_number=22, tag_tid=2, entry_tid=18678]]], state=connected] 2024-09-11T06:18:12.707 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:12.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=21, tag_tid=2, entry_tid=18681' 2024-09-11T06:18:12.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=21, tag_tid=3, entry_tid=1' -a 'object_number=21, tag_tid=3, entry_tid=1' = 'object_number=21, tag_tid=2, entry_tid=18681' 2024-09-11T06:18:12.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=21, tag_tid=2, entry_tid=18681' '!=' 'object_number=21, tag_tid=2, entry_tid=18577' 2024-09-11T06:18:12.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:18:12.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=21, tag_tid=2, entry_tid=18681' = 'object_number=21, tag_tid=2, entry_tid=18577' 2024-09-11T06:18:12.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=21, tag_tid=2, entry_tid=18681' 2024-09-11T06:18:12.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:18:12.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:18:12.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:12.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=3 2024-09-11T06:18:12.711 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=2, entry_tid=18681' 2024-09-11T06:18:12.711 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:18:12.711 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:12.712 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=2 2024-09-11T06:18:12.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:18:12.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:18:12.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:12.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:18:12.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=2, entry_tid=18681' 2024-09-11T06:18:12.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:18:12.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:12.716 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=18681 2024-09-11T06:18:12.716 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 3 = 2 -a 1 -le 18681 2024-09-11T06:18:12.716 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:18:12.716 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:18:12.716 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:18:12.918 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:18:12.918 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:18:12.918 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:18:12.918 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:18:12.918 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:18:12.918 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:18:12.918 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:18:12.918 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:18:12.918 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:18:12.918 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:18:12.918 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:18:12.918 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:18:12.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:18:12.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:18:12.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:18:12.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:18:12.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:18:12.922 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:18:12.923 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:18:12.923 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:18:12.923 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:18:12.923 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:18:12.923 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:18:12.923 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:18:12.923 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:18:12.923 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:18:12.923 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:18:12.923 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:18:12.923 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:18:12.923 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:18:12.923 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:18:12.924 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:18:12.924 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:18:12.924 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:18:12.924 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:18:12.924 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:18:12.924 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:18:12.924 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:18:12.924 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:18:12.924 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:18:12.924 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:18:12.924 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:18:12.924 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:18:12.924 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:18:12.925 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:18:12.925 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:18:12.925 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:18:12.925 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:18:12.925 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:18:12.948 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:18:12.949 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:18:12.949 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:18:12.949 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:18:13.084 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:18:13.087 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:18:13.087 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:13.087 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:13.088 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:13.088 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:18:13.088 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:13.088 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:13.088 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:13.088 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:18:13.088 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:18:13.089 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:18:13.089 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:18:13.089 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:13.090 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:13.090 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:18:13.133 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 5 2024-09-11T06:18:13.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:18:13.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:18:13.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:18:13.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=21, tag_tid=2, entry_tid=18777], [object_number=20, tag_tid=2, entry_tid=18776], [object_number=23, tag_tid=2, entry_tid=18775], [object_number=22, tag_tid=2, entry_tid=18774]]], state=connected] 2024-09-11T06:18:13.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:13.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:18:13.139 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:18:13.139 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:13.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:13.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:13.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:18:13.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:13.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:13.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:13.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:18:13.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:18:13.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:18:13.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:18:13.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:13.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:18:13.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:13.185 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 5 2024-09-11T06:18:13.186 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:18:13.186 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:18:13.186 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:18:13.186 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=21, tag_tid=2, entry_tid=18777], [object_number=20, tag_tid=2, entry_tid=18776], [object_number=23, tag_tid=2, entry_tid=18775], [object_number=22, tag_tid=2, entry_tid=18774]]], state=connected] 2024-09-11T06:18:13.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:13.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=21, tag_tid=2, entry_tid=18777' 2024-09-11T06:18:13.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=21, tag_tid=3, entry_tid=1' -a 'object_number=21, tag_tid=3, entry_tid=1' = 'object_number=21, tag_tid=2, entry_tid=18777' 2024-09-11T06:18:13.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=21, tag_tid=2, entry_tid=18777' '!=' 'object_number=21, tag_tid=2, entry_tid=18681' 2024-09-11T06:18:13.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:18:13.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=21, tag_tid=2, entry_tid=18777' = 'object_number=21, tag_tid=2, entry_tid=18681' 2024-09-11T06:18:13.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=21, tag_tid=2, entry_tid=18777' 2024-09-11T06:18:13.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:18:13.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:18:13.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:13.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=3 2024-09-11T06:18:13.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=2, entry_tid=18777' 2024-09-11T06:18:13.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:18:13.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:13.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=2 2024-09-11T06:18:13.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:18:13.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:18:13.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:13.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:18:13.196 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=2, entry_tid=18777' 2024-09-11T06:18:13.196 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:18:13.196 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:13.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=18777 2024-09-11T06:18:13.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 3 = 2 -a 1 -le 18777 2024-09-11T06:18:13.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:18:13.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:18:13.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:18:13.399 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:18:13.399 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:18:13.399 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:18:13.399 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:18:13.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:18:13.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:18:13.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:18:13.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:18:13.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:18:13.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:18:13.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:18:13.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:18:13.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:18:13.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:18:13.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:18:13.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:18:13.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:18:13.403 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:18:13.403 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:18:13.403 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:18:13.403 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:18:13.403 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:18:13.403 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:18:13.403 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:18:13.404 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:18:13.404 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:18:13.404 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:18:13.404 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:18:13.404 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:18:13.404 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:18:13.404 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:18:13.404 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:18:13.404 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:18:13.406 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:18:13.406 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:18:13.406 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:18:13.406 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:18:13.406 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:18:13.406 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:18:13.407 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:18:13.407 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:18:13.407 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:18:13.407 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:18:13.407 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:18:13.407 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:18:13.407 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:18:13.407 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:18:13.407 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:18:13.407 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:18:13.431 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:18:13.432 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:18:13.432 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:18:13.432 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:18:13.564 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:18:13.565 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:18:13.565 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:13.565 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:13.565 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:13.565 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:18:13.565 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:13.566 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:13.566 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:13.566 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:18:13.566 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:18:13.566 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:18:13.566 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:18:13.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:13.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:18:13.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:13.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 5 2024-09-11T06:18:13.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:18:13.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:18:13.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:18:13.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=22, tag_tid=2, entry_tid=18890], [object_number=21, tag_tid=2, entry_tid=18889], [object_number=20, tag_tid=2, entry_tid=18888], [object_number=23, tag_tid=2, entry_tid=18887]]], state=connected] 2024-09-11T06:18:13.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:13.624 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:18:13.625 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:18:13.625 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:13.625 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:13.625 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:13.625 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:18:13.625 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:13.625 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:13.625 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:13.625 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:18:13.625 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:18:13.625 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:18:13.626 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:18:13.626 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:13.626 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:18:13.626 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:13.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 5 2024-09-11T06:18:13.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:18:13.674 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:18:13.674 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:18:13.674 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=22, tag_tid=2, entry_tid=18890], [object_number=21, tag_tid=2, entry_tid=18889], [object_number=20, tag_tid=2, entry_tid=18888], [object_number=23, tag_tid=2, entry_tid=18887]]], state=connected] 2024-09-11T06:18:13.677 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:13.679 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=22, tag_tid=2, entry_tid=18890' 2024-09-11T06:18:13.679 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=21, tag_tid=3, entry_tid=1' -a 'object_number=21, tag_tid=3, entry_tid=1' = 'object_number=22, tag_tid=2, entry_tid=18890' 2024-09-11T06:18:13.679 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=22, tag_tid=2, entry_tid=18890' '!=' 'object_number=21, tag_tid=2, entry_tid=18777' 2024-09-11T06:18:13.679 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:18:13.679 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=22, tag_tid=2, entry_tid=18890' = 'object_number=21, tag_tid=2, entry_tid=18777' 2024-09-11T06:18:13.679 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=22, tag_tid=2, entry_tid=18890' 2024-09-11T06:18:13.679 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:18:13.679 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:18:13.680 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:13.680 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=3 2024-09-11T06:18:13.681 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=22, tag_tid=2, entry_tid=18890' 2024-09-11T06:18:13.681 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:18:13.681 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:13.682 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=2 2024-09-11T06:18:13.682 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:18:13.682 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:18:13.682 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:13.683 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:18:13.684 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=22, tag_tid=2, entry_tid=18890' 2024-09-11T06:18:13.684 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:18:13.684 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:13.685 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=18890 2024-09-11T06:18:13.685 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 3 = 2 -a 1 -le 18890 2024-09-11T06:18:13.685 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:18:13.686 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:18:13.686 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:18:13.887 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:18:13.887 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:18:13.887 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:18:13.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:18:13.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:18:13.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:18:13.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:18:13.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:18:13.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:18:13.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:18:13.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:18:13.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:18:13.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:18:13.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:18:13.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:18:13.889 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:18:13.889 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:18:13.891 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:18:13.891 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:18:13.892 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:18:13.892 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:18:13.892 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:18:13.892 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:18:13.892 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:18:13.892 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:18:13.892 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:18:13.892 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:18:13.892 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:18:13.892 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:18:13.892 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:18:13.892 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:18:13.892 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:18:13.893 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:18:13.894 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:18:13.894 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:18:13.894 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:18:13.894 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:18:13.894 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:18:13.894 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:18:13.894 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:18:13.894 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:18:13.894 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:18:13.894 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:18:13.894 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:18:13.894 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:18:13.895 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:18:13.895 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:18:13.895 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:18:13.895 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:18:13.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:18:13.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:18:13.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:18:13.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:18:14.051 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:18:14.052 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:18:14.052 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:14.052 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:14.052 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:14.052 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:18:14.052 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:14.052 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:14.052 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:14.052 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:18:14.053 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:18:14.053 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:18:14.053 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:18:14.054 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:14.054 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:14.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:18:14.112 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 5 2024-09-11T06:18:14.112 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:18:14.112 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:18:14.112 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:18:14.112 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=23, tag_tid=2, entry_tid=19007], [object_number=22, tag_tid=2, entry_tid=19006], [object_number=21, tag_tid=2, entry_tid=19005], [object_number=20, tag_tid=2, entry_tid=19004]]], state=connected] 2024-09-11T06:18:14.115 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:14.117 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:18:14.118 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:18:14.118 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:14.118 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:14.118 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:14.118 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:18:14.118 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:14.118 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:14.118 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:14.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:18:14.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:18:14.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:18:14.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:18:14.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:14.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:18:14.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:14.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 5 2024-09-11T06:18:14.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:18:14.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:18:14.166 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:18:14.166 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=23, tag_tid=2, entry_tid=19007], [object_number=22, tag_tid=2, entry_tid=19006], [object_number=21, tag_tid=2, entry_tid=19005], [object_number=20, tag_tid=2, entry_tid=19004]]], state=connected] 2024-09-11T06:18:14.168 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:14.171 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=23, tag_tid=2, entry_tid=19007' 2024-09-11T06:18:14.171 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=21, tag_tid=3, entry_tid=1' -a 'object_number=21, tag_tid=3, entry_tid=1' = 'object_number=23, tag_tid=2, entry_tid=19007' 2024-09-11T06:18:14.171 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=23, tag_tid=2, entry_tid=19007' '!=' 'object_number=22, tag_tid=2, entry_tid=18890' 2024-09-11T06:18:14.172 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:18:14.172 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=23, tag_tid=2, entry_tid=19007' = 'object_number=22, tag_tid=2, entry_tid=18890' 2024-09-11T06:18:14.172 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=23, tag_tid=2, entry_tid=19007' 2024-09-11T06:18:14.172 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:18:14.172 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:18:14.173 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:14.175 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=3 2024-09-11T06:18:14.175 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=23, tag_tid=2, entry_tid=19007' 2024-09-11T06:18:14.175 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:18:14.175 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:14.175 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=2 2024-09-11T06:18:14.176 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:18:14.176 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:18:14.176 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:14.178 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:18:14.178 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=23, tag_tid=2, entry_tid=19007' 2024-09-11T06:18:14.178 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:18:14.179 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:14.179 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=19007 2024-09-11T06:18:14.179 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 3 = 2 -a 1 -le 19007 2024-09-11T06:18:14.179 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:18:14.179 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:18:14.180 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:18:14.379 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:18:14.379 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:18:14.379 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:18:14.380 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:18:14.380 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:18:14.380 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:18:14.380 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:18:14.380 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:18:14.380 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:18:14.380 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:18:14.380 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:18:14.380 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:18:14.380 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:18:14.380 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:18:14.380 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:18:14.381 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:18:14.381 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:18:14.383 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:18:14.383 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:18:14.383 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:18:14.383 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:18:14.383 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:18:14.383 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:18:14.384 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:18:14.384 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:18:14.384 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:18:14.384 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:18:14.384 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:18:14.384 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:18:14.384 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:18:14.384 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:18:14.384 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:18:14.384 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:18:14.386 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:18:14.386 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:18:14.386 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:18:14.386 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:18:14.386 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:18:14.386 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:18:14.386 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:18:14.386 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:18:14.386 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:18:14.386 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:18:14.386 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:18:14.387 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:18:14.387 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:18:14.387 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:18:14.387 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:18:14.387 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:18:14.409 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:18:14.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:18:14.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:18:14.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:18:14.538 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:18:14.539 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:18:14.540 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:14.540 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:14.540 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:14.540 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:18:14.540 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:14.540 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:14.540 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:14.540 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:18:14.540 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:18:14.541 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:18:14.541 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:18:14.543 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:14.543 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:18:14.543 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:14.589 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 5 2024-09-11T06:18:14.589 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:18:14.589 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:18:14.589 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:18:14.589 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=21, tag_tid=2, entry_tid=19121], [object_number=20, tag_tid=2, entry_tid=19120], [object_number=23, tag_tid=2, entry_tid=19119], [object_number=22, tag_tid=2, entry_tid=19118]]], state=connected] 2024-09-11T06:18:14.592 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:14.594 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:18:14.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:18:14.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:14.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:14.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:14.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:18:14.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:14.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:14.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:14.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:18:14.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:18:14.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:18:14.596 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:18:14.596 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:14.596 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:18:14.596 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:14.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 5 2024-09-11T06:18:14.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:18:14.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:18:14.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:18:14.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=21, tag_tid=2, entry_tid=19121], [object_number=20, tag_tid=2, entry_tid=19120], [object_number=23, tag_tid=2, entry_tid=19119], [object_number=22, tag_tid=2, entry_tid=19118]]], state=connected] 2024-09-11T06:18:14.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:14.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=21, tag_tid=2, entry_tid=19121' 2024-09-11T06:18:14.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=21, tag_tid=3, entry_tid=1' -a 'object_number=21, tag_tid=3, entry_tid=1' = 'object_number=21, tag_tid=2, entry_tid=19121' 2024-09-11T06:18:14.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=21, tag_tid=2, entry_tid=19121' '!=' 'object_number=23, tag_tid=2, entry_tid=19007' 2024-09-11T06:18:14.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:18:14.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=21, tag_tid=2, entry_tid=19121' = 'object_number=23, tag_tid=2, entry_tid=19007' 2024-09-11T06:18:14.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=21, tag_tid=2, entry_tid=19121' 2024-09-11T06:18:14.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:18:14.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:18:14.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:14.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=3 2024-09-11T06:18:14.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=2, entry_tid=19121' 2024-09-11T06:18:14.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:18:14.649 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:14.649 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=2 2024-09-11T06:18:14.650 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:18:14.650 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:18:14.650 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:14.651 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:18:14.652 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=2, entry_tid=19121' 2024-09-11T06:18:14.652 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:18:14.652 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:14.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=19121 2024-09-11T06:18:14.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 3 = 2 -a 1 -le 19121 2024-09-11T06:18:14.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:18:14.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:18:14.654 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:18:14.855 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:18:14.855 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:18:14.855 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:18:14.855 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:18:14.855 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:18:14.855 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:18:14.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:18:14.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:18:14.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:18:14.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:18:14.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:18:14.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:18:14.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:18:14.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:18:14.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:18:14.857 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:18:14.857 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:18:14.858 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:18:14.858 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:18:14.858 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:18:14.859 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:18:14.859 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:18:14.859 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:18:14.859 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:18:14.859 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:18:14.859 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:18:14.859 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:18:14.859 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:18:14.859 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:18:14.859 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:18:14.859 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:18:14.859 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:18:14.859 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:18:14.860 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:18:14.860 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:18:14.860 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:18:14.861 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:18:14.861 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:18:14.861 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:18:14.861 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:18:14.861 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:18:14.861 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:18:14.861 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:18:14.861 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:18:14.861 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:18:14.861 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:18:14.861 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:18:14.861 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:18:14.862 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:18:14.885 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:18:14.885 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:18:14.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:18:14.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:18:15.019 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:18:15.020 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:18:15.020 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:15.020 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:15.020 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:15.020 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:18:15.020 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:15.020 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:15.020 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:15.021 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:18:15.021 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:18:15.021 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:18:15.021 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:18:15.023 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:15.024 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:18:15.024 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:15.069 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 5 2024-09-11T06:18:15.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:18:15.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:18:15.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:18:15.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=23, tag_tid=2, entry_tid=19235], [object_number=22, tag_tid=2, entry_tid=19234], [object_number=21, tag_tid=2, entry_tid=19233], [object_number=20, tag_tid=2, entry_tid=19232]]], state=connected] 2024-09-11T06:18:15.073 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:15.074 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:18:15.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:18:15.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:15.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:15.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:15.076 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:18:15.076 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:15.076 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:15.076 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:15.076 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:18:15.076 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:18:15.076 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:18:15.076 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:18:15.077 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:15.077 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:18:15.077 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:15.122 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 5 2024-09-11T06:18:15.122 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:18:15.122 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:18:15.122 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:18:15.122 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=23, tag_tid=2, entry_tid=19235], [object_number=22, tag_tid=2, entry_tid=19234], [object_number=21, tag_tid=2, entry_tid=19233], [object_number=20, tag_tid=2, entry_tid=19232]]], state=connected] 2024-09-11T06:18:15.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:15.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=23, tag_tid=2, entry_tid=19235' 2024-09-11T06:18:15.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=21, tag_tid=3, entry_tid=1' -a 'object_number=21, tag_tid=3, entry_tid=1' = 'object_number=23, tag_tid=2, entry_tid=19235' 2024-09-11T06:18:15.128 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=23, tag_tid=2, entry_tid=19235' '!=' 'object_number=21, tag_tid=2, entry_tid=19121' 2024-09-11T06:18:15.128 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:18:15.128 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=23, tag_tid=2, entry_tid=19235' = 'object_number=21, tag_tid=2, entry_tid=19121' 2024-09-11T06:18:15.128 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=23, tag_tid=2, entry_tid=19235' 2024-09-11T06:18:15.128 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:18:15.128 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:18:15.128 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:15.128 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=3 2024-09-11T06:18:15.129 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=23, tag_tid=2, entry_tid=19235' 2024-09-11T06:18:15.129 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:18:15.129 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:15.130 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=2 2024-09-11T06:18:15.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:18:15.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:18:15.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:15.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:18:15.133 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=23, tag_tid=2, entry_tid=19235' 2024-09-11T06:18:15.133 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:18:15.133 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:15.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=19235 2024-09-11T06:18:15.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 3 = 2 -a 1 -le 19235 2024-09-11T06:18:15.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:18:15.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:18:15.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:18:15.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:18:15.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:18:15.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:18:15.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:18:15.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:18:15.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:18:15.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:18:15.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:18:15.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:18:15.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:18:15.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:18:15.337 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:18:15.337 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:18:15.337 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:18:15.337 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:18:15.337 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:18:15.337 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:18:15.339 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:18:15.340 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:18:15.340 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:18:15.340 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:18:15.340 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:18:15.340 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:18:15.340 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:18:15.340 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:18:15.340 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:18:15.340 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:18:15.340 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:18:15.340 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:18:15.340 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:18:15.340 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:18:15.341 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:18:15.341 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:18:15.341 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:18:15.341 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:18:15.341 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:18:15.341 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:18:15.341 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:18:15.341 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:18:15.341 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:18:15.341 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:18:15.341 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:18:15.341 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:18:15.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:18:15.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:18:15.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:18:15.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:18:15.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:18:15.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:18:15.365 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:18:15.366 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:18:15.366 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:18:15.366 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:18:15.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:18:15.501 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:18:15.501 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:15.501 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:15.501 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:15.501 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:18:15.501 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:15.501 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:15.501 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:15.502 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:18:15.502 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:18:15.502 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:18:15.502 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:18:15.503 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:15.504 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:18:15.504 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:15.553 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 5 2024-09-11T06:18:15.553 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:18:15.553 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:18:15.553 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:18:15.553 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=22, tag_tid=2, entry_tid=19346], [object_number=21, tag_tid=2, entry_tid=19345], [object_number=20, tag_tid=2, entry_tid=19344], [object_number=23, tag_tid=2, entry_tid=19343]]], state=connected] 2024-09-11T06:18:15.556 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:15.558 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:18:15.559 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:18:15.559 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:15.559 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:15.559 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:15.559 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:18:15.559 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:15.559 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:15.559 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:15.559 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:18:15.560 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:18:15.560 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:18:15.560 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:18:15.560 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:15.560 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:18:15.561 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:15.607 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 5 2024-09-11T06:18:15.607 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:18:15.607 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:18:15.607 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:18:15.607 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=22, tag_tid=2, entry_tid=19346], [object_number=21, tag_tid=2, entry_tid=19345], [object_number=20, tag_tid=2, entry_tid=19344], [object_number=23, tag_tid=2, entry_tid=19343]]], state=connected] 2024-09-11T06:18:15.610 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:15.612 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=22, tag_tid=2, entry_tid=19346' 2024-09-11T06:18:15.612 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=21, tag_tid=3, entry_tid=1' -a 'object_number=21, tag_tid=3, entry_tid=1' = 'object_number=22, tag_tid=2, entry_tid=19346' 2024-09-11T06:18:15.612 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=22, tag_tid=2, entry_tid=19346' '!=' 'object_number=23, tag_tid=2, entry_tid=19235' 2024-09-11T06:18:15.612 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:18:15.612 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=22, tag_tid=2, entry_tid=19346' = 'object_number=23, tag_tid=2, entry_tid=19235' 2024-09-11T06:18:15.612 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=22, tag_tid=2, entry_tid=19346' 2024-09-11T06:18:15.613 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:18:15.613 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:18:15.613 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:15.613 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=3 2024-09-11T06:18:15.614 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=22, tag_tid=2, entry_tid=19346' 2024-09-11T06:18:15.614 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:18:15.614 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:15.615 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=2 2024-09-11T06:18:15.616 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:18:15.616 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:18:15.616 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:15.617 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:18:15.617 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=22, tag_tid=2, entry_tid=19346' 2024-09-11T06:18:15.617 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:18:15.617 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:15.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=19346 2024-09-11T06:18:15.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 3 = 2 -a 1 -le 19346 2024-09-11T06:18:15.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:18:15.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:18:15.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:18:15.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:18:15.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:18:15.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:18:15.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:18:15.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:18:15.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:18:15.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:18:15.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:18:15.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:18:15.822 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:18:15.822 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:18:15.822 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:18:15.822 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:18:15.822 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:18:15.822 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:18:15.822 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:18:15.823 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:18:15.825 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:18:15.825 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:18:15.825 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:18:15.825 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:18:15.826 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:18:15.826 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:18:15.826 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:18:15.826 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:18:15.826 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:18:15.826 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:18:15.826 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:18:15.826 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:18:15.826 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:18:15.826 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:18:15.826 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:18:15.826 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:18:15.827 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:18:15.827 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:18:15.828 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:18:15.828 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:18:15.828 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:18:15.828 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:18:15.828 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:18:15.828 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:18:15.828 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:18:15.828 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:18:15.828 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:18:15.828 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:18:15.828 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:18:15.828 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:18:15.829 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:18:15.829 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:18:15.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:18:15.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:18:15.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:18:15.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:18:15.979 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:18:15.980 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:18:15.980 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:15.980 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:15.980 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:15.980 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:18:15.980 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:15.981 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:15.981 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:15.981 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:18:15.981 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:18:15.981 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:18:15.981 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:18:15.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:15.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:18:15.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:16.035 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 5 2024-09-11T06:18:16.035 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:18:16.036 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:18:16.036 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:18:16.036 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=21, tag_tid=2, entry_tid=19461], [object_number=20, tag_tid=2, entry_tid=19460], [object_number=23, tag_tid=2, entry_tid=19459], [object_number=22, tag_tid=2, entry_tid=19458]]], state=connected] 2024-09-11T06:18:16.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:16.040 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:18:16.041 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:18:16.041 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:16.041 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:16.041 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:16.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:18:16.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:16.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:16.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:16.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:18:16.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:18:16.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:18:16.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:18:16.043 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:16.043 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:18:16.043 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:16.088 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 5 2024-09-11T06:18:16.089 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:18:16.089 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:18:16.089 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:18:16.089 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=21, tag_tid=2, entry_tid=19461], [object_number=20, tag_tid=2, entry_tid=19460], [object_number=23, tag_tid=2, entry_tid=19459], [object_number=22, tag_tid=2, entry_tid=19458]]], state=connected] 2024-09-11T06:18:16.092 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:16.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=21, tag_tid=2, entry_tid=19461' 2024-09-11T06:18:16.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=21, tag_tid=3, entry_tid=1' -a 'object_number=21, tag_tid=3, entry_tid=1' = 'object_number=21, tag_tid=2, entry_tid=19461' 2024-09-11T06:18:16.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=21, tag_tid=2, entry_tid=19461' '!=' 'object_number=22, tag_tid=2, entry_tid=19346' 2024-09-11T06:18:16.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:18:16.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=21, tag_tid=2, entry_tid=19461' = 'object_number=22, tag_tid=2, entry_tid=19346' 2024-09-11T06:18:16.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=21, tag_tid=2, entry_tid=19461' 2024-09-11T06:18:16.095 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:18:16.095 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:18:16.095 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:16.095 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=3 2024-09-11T06:18:16.096 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=2, entry_tid=19461' 2024-09-11T06:18:16.096 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:18:16.096 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:16.097 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=2 2024-09-11T06:18:16.098 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:18:16.098 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:18:16.098 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:16.099 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:18:16.100 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=2, entry_tid=19461' 2024-09-11T06:18:16.100 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:18:16.100 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:16.101 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=19461 2024-09-11T06:18:16.101 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 3 = 2 -a 1 -le 19461 2024-09-11T06:18:16.101 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:18:16.101 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:18:16.101 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:18:16.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:18:16.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:18:16.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:18:16.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:18:16.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:18:16.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:18:16.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:18:16.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:18:16.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:18:16.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:18:16.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:18:16.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:18:16.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:18:16.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:18:16.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:18:16.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:18:16.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:18:16.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:18:16.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:18:16.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:18:16.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:18:16.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:18:16.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:18:16.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:18:16.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:18:16.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:18:16.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:18:16.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:18:16.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:18:16.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:18:16.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:18:16.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:18:16.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:18:16.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:18:16.310 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:18:16.310 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:18:16.310 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:18:16.310 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:18:16.310 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:18:16.310 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:18:16.310 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:18:16.310 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:18:16.310 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:18:16.310 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:18:16.310 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:18:16.310 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:18:16.310 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:18:16.310 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:18:16.311 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:18:16.333 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:18:16.334 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:18:16.334 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:18:16.334 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:18:16.459 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:18:16.461 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:18:16.461 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:16.461 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:16.461 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:16.461 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:18:16.461 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:16.461 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:16.461 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:16.461 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:18:16.462 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:18:16.462 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:18:16.462 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:18:16.463 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:16.463 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:18:16.464 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:16.514 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 5 2024-09-11T06:18:16.514 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:18:16.514 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:18:16.514 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:18:16.514 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=23, tag_tid=2, entry_tid=19567], [object_number=22, tag_tid=2, entry_tid=19566], [object_number=21, tag_tid=2, entry_tid=19565], [object_number=20, tag_tid=2, entry_tid=19564]]], state=connected] 2024-09-11T06:18:16.517 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:16.519 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:18:16.520 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:18:16.520 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:16.520 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:16.520 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:16.520 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:18:16.520 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:16.521 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:16.521 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:16.521 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:18:16.521 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:18:16.521 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:18:16.521 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:18:16.521 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:16.522 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:18:16.522 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:16.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 5 2024-09-11T06:18:16.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:18:16.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:18:16.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:18:16.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=23, tag_tid=2, entry_tid=19567], [object_number=22, tag_tid=2, entry_tid=19566], [object_number=21, tag_tid=2, entry_tid=19565], [object_number=20, tag_tid=2, entry_tid=19564]]], state=connected] 2024-09-11T06:18:16.570 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:16.572 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=23, tag_tid=2, entry_tid=19567' 2024-09-11T06:18:16.572 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=21, tag_tid=3, entry_tid=1' -a 'object_number=21, tag_tid=3, entry_tid=1' = 'object_number=23, tag_tid=2, entry_tid=19567' 2024-09-11T06:18:16.572 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=23, tag_tid=2, entry_tid=19567' '!=' 'object_number=21, tag_tid=2, entry_tid=19461' 2024-09-11T06:18:16.573 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:18:16.573 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=23, tag_tid=2, entry_tid=19567' = 'object_number=21, tag_tid=2, entry_tid=19461' 2024-09-11T06:18:16.573 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=23, tag_tid=2, entry_tid=19567' 2024-09-11T06:18:16.573 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:18:16.573 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:18:16.573 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:16.573 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=3 2024-09-11T06:18:16.574 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=23, tag_tid=2, entry_tid=19567' 2024-09-11T06:18:16.574 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:18:16.574 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:16.575 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=2 2024-09-11T06:18:16.575 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:18:16.576 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:18:16.576 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:16.576 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:18:16.577 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=23, tag_tid=2, entry_tid=19567' 2024-09-11T06:18:16.577 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:18:16.577 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:16.578 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=19567 2024-09-11T06:18:16.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 3 = 2 -a 1 -le 19567 2024-09-11T06:18:16.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:18:16.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:18:16.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:18:16.781 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:18:16.781 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:18:16.781 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:18:16.781 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:18:16.781 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:18:16.781 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:18:16.781 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:18:16.781 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:18:16.781 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:18:16.781 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:18:16.781 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:18:16.781 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:18:16.782 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:18:16.782 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:18:16.782 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:18:16.782 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:18:16.782 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:18:16.785 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:18:16.785 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:18:16.785 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:18:16.785 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:18:16.785 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:18:16.785 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:18:16.785 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:18:16.785 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:18:16.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:18:16.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:18:16.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:18:16.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:18:16.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:18:16.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:18:16.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:18:16.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:18:16.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:18:16.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:18:16.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:18:16.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:18:16.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:18:16.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:18:16.787 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:18:16.787 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:18:16.787 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:18:16.787 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:18:16.787 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:18:16.787 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:18:16.787 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:18:16.787 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:18:16.787 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:18:16.787 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:18:16.812 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:18:16.812 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:18:16.812 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:18:16.812 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:18:16.940 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:18:16.941 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:18:16.941 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:16.941 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:16.942 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:16.942 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:18:16.942 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:16.942 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:16.942 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:16.942 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:18:16.942 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:18:16.943 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:18:16.943 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:18:16.944 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:16.944 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:18:16.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:17.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 5 2024-09-11T06:18:17.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:18:17.199 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:18:17.199 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:18:17.199 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=22, tag_tid=2, entry_tid=19658], [object_number=21, tag_tid=2, entry_tid=19657], [object_number=20, tag_tid=2, entry_tid=19656], [object_number=23, tag_tid=2, entry_tid=19655]]], state=connected] 2024-09-11T06:18:17.203 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:17.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:18:17.205 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:18:17.205 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:17.205 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:17.205 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:17.205 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:18:17.206 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:17.206 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:17.206 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:17.206 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:18:17.206 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:18:17.206 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:18:17.206 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:18:17.207 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:17.207 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:18:17.207 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:17.255 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 5 2024-09-11T06:18:17.255 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:18:17.255 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:18:17.255 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:18:17.255 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=22, tag_tid=2, entry_tid=19658], [object_number=21, tag_tid=2, entry_tid=19657], [object_number=20, tag_tid=2, entry_tid=19656], [object_number=23, tag_tid=2, entry_tid=19655]]], state=connected] 2024-09-11T06:18:17.259 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:17.261 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=22, tag_tid=2, entry_tid=19658' 2024-09-11T06:18:17.261 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=21, tag_tid=3, entry_tid=1' -a 'object_number=21, tag_tid=3, entry_tid=1' = 'object_number=22, tag_tid=2, entry_tid=19658' 2024-09-11T06:18:17.261 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=22, tag_tid=2, entry_tid=19658' '!=' 'object_number=23, tag_tid=2, entry_tid=19567' 2024-09-11T06:18:17.262 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:18:17.262 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=22, tag_tid=2, entry_tid=19658' = 'object_number=23, tag_tid=2, entry_tid=19567' 2024-09-11T06:18:17.262 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=22, tag_tid=2, entry_tid=19658' 2024-09-11T06:18:17.262 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:18:17.262 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:18:17.262 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:17.262 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=3 2024-09-11T06:18:17.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=22, tag_tid=2, entry_tid=19658' 2024-09-11T06:18:17.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:18:17.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:17.264 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=2 2024-09-11T06:18:17.265 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:18:17.265 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:18:17.265 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:17.266 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:18:17.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=22, tag_tid=2, entry_tid=19658' 2024-09-11T06:18:17.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:18:17.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:17.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=19658 2024-09-11T06:18:17.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 3 = 2 -a 1 -le 19658 2024-09-11T06:18:17.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:18:17.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:18:17.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:18:17.470 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:18:17.470 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:18:17.470 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:18:17.470 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:18:17.470 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:18:17.470 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:18:17.470 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:18:17.470 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:18:17.471 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:18:17.471 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:18:17.471 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:18:17.471 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:18:17.471 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:18:17.471 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:18:17.471 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:18:17.472 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:18:17.472 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:18:17.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:18:17.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:18:17.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:18:17.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:18:17.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:18:17.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:18:17.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:18:17.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:18:17.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:18:17.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:18:17.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:18:17.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:18:17.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:18:17.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:18:17.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:18:17.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:18:17.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:18:17.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:18:17.476 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:18:17.476 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:18:17.476 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:18:17.476 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:18:17.476 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:18:17.476 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:18:17.476 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:18:17.476 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:18:17.476 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:18:17.476 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:18:17.476 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:18:17.476 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:18:17.476 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:18:17.477 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:18:17.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:18:17.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:18:17.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:18:17.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:18:17.626 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:18:17.627 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:18:17.627 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:17.627 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:17.627 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:17.627 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:18:17.628 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:17.628 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:17.628 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:17.628 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:18:17.628 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:18:17.628 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:18:17.628 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:18:17.631 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:17.631 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:18:17.632 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:17.683 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 5 2024-09-11T06:18:17.683 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:18:17.683 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:18:17.683 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:18:17.683 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=22, tag_tid=2, entry_tid=19770], [object_number=21, tag_tid=2, entry_tid=19769], [object_number=20, tag_tid=2, entry_tid=19768], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:18:17.686 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:17.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:18:17.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:18:17.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:17.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:17.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:17.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:18:17.689 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:17.689 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:17.689 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:17.689 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:18:17.689 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:18:17.689 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:18:17.689 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:18:17.690 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:17.690 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:18:17.690 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:17.736 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 5 2024-09-11T06:18:17.736 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:18:17.737 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:18:17.737 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:18:17.737 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=22, tag_tid=2, entry_tid=19770], [object_number=21, tag_tid=2, entry_tid=19769], [object_number=20, tag_tid=2, entry_tid=19768], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:18:17.740 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:17.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=22, tag_tid=2, entry_tid=19770' 2024-09-11T06:18:17.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=21, tag_tid=3, entry_tid=1' -a 'object_number=21, tag_tid=3, entry_tid=1' = 'object_number=22, tag_tid=2, entry_tid=19770' 2024-09-11T06:18:17.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=22, tag_tid=2, entry_tid=19770' '!=' 'object_number=22, tag_tid=2, entry_tid=19658' 2024-09-11T06:18:17.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:18:17.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=22, tag_tid=2, entry_tid=19770' = 'object_number=22, tag_tid=2, entry_tid=19658' 2024-09-11T06:18:17.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=22, tag_tid=2, entry_tid=19770' 2024-09-11T06:18:17.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:18:17.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:18:17.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:17.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=3 2024-09-11T06:18:17.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=22, tag_tid=2, entry_tid=19770' 2024-09-11T06:18:17.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:18:17.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:17.745 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=2 2024-09-11T06:18:17.746 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:18:17.746 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:18:17.746 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:17.746 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:18:17.747 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=22, tag_tid=2, entry_tid=19770' 2024-09-11T06:18:17.747 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:18:17.747 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:18:17.748 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=19770 2024-09-11T06:18:17.749 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 3 = 2 -a 1 -le 19770 2024-09-11T06:18:17.749 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:18:17.749 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:18:17.749 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:18:17.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:18:17.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:18:17.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:18:17.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:18:17.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:18:17.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:18:17.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:18:17.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:18:17.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:18:17.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:18:17.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:18:17.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:18:17.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:18:17.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:18:17.952 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:18:17.952 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:18:17.952 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:18:17.954 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:18:17.954 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:18:17.954 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:18:17.954 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:18:17.955 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:18:17.955 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:18:17.955 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:18:17.955 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:18:17.955 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:18:17.955 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:18:17.955 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:18:17.955 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:18:17.955 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:18:17.955 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:18:17.955 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:18:17.955 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:18:17.957 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:18:17.957 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:18:17.957 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:18:17.957 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:18:17.957 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:18:17.957 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:18:17.957 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:18:17.957 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:18:17.957 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:18:17.957 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:18:17.958 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:18:17.958 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:18:17.958 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:18:17.958 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:18:17.958 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:18:17.958 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:18:17.981 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:18:17.981 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:18:17.982 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:18:17.982 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:18:18.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:18:18.110 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:18:18.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:18.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:18.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:18.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:18:18.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:18.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:18.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:18.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:18:18.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:18:18.112 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:18:18.112 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:18:18.113 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:18.113 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:18:18.113 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:18.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 5 2024-09-11T06:18:18.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:18:18.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:18:18.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:18:18.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=22, tag_tid=2, entry_tid=19770], [object_number=21, tag_tid=2, entry_tid=19769], [object_number=20, tag_tid=2, entry_tid=19768], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:18:18.161 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:18.162 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:18:18.163 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:18:18.163 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:18.164 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:18.164 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:18.164 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:18:18.164 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:18.164 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:18.164 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:18.164 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:18:18.164 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:18:18.164 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:18:18.164 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:18:18.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:18.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:18:18.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:18.209 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 5 2024-09-11T06:18:18.209 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:18:18.209 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:18:18.209 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:18:18.209 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=22, tag_tid=2, entry_tid=19770], [object_number=21, tag_tid=2, entry_tid=19769], [object_number=20, tag_tid=2, entry_tid=19768], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:18:18.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:18.214 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=22, tag_tid=2, entry_tid=19770' 2024-09-11T06:18:18.214 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=21, tag_tid=3, entry_tid=1' -a 'object_number=21, tag_tid=3, entry_tid=1' = 'object_number=22, tag_tid=2, entry_tid=19770' 2024-09-11T06:18:18.214 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=22, tag_tid=2, entry_tid=19770' '!=' 'object_number=22, tag_tid=2, entry_tid=19770' 2024-09-11T06:18:18.214 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:18:18.214 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.4 2024-09-11T06:18:18.616 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:18:18.616 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:18:18.616 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:18:18.616 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:18:18.616 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:18:18.617 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:18:18.617 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:18:18.617 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:18:18.617 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:18:18.617 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:18:18.617 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:18:18.617 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:18:18.617 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:18:18.617 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:18:18.617 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:18:18.617 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:18:18.621 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:18:18.621 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:18:18.621 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:18:18.621 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:18:18.621 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:18:18.621 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:18:18.621 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:18:18.621 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:18:18.621 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:18:18.621 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:18:18.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:18:18.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:18:18.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:18:18.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:18:18.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:18:18.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:18:18.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:18:18.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:18:18.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:18:18.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:18:18.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:18:18.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:18:18.623 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:18:18.624 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:18:18.624 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:18:18.624 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:18:18.624 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:18:18.624 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:18:18.624 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:18:18.624 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:18:18.624 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:18:18.624 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:18:18.624 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:18:18.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:18:18.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:18:18.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:18:18.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:18:18.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:18:18.768 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:18:18.768 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:18.768 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:18.768 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:18.768 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:18:18.768 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:18.768 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:18.768 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:18.769 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:18:18.769 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:18:18.769 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:18:18.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:18:18.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:18.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:18:18.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:18.826 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 5 2024-09-11T06:18:18.826 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:18:18.826 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:18:18.826 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:18:18.826 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=22, tag_tid=2, entry_tid=19770], [object_number=21, tag_tid=2, entry_tid=19769], [object_number=20, tag_tid=2, entry_tid=19768], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:18:18.829 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:18.830 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:18:18.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:18:18.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:18.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:18.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:18.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:18:18.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:18.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:18.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:18.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:18:18.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:18:18.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:18:18.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:18:18.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:18.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:18:18.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:18.878 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 5 2024-09-11T06:18:18.878 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:18:18.878 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:18:18.878 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:18:18.878 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=22, tag_tid=2, entry_tid=19770], [object_number=21, tag_tid=2, entry_tid=19769], [object_number=20, tag_tid=2, entry_tid=19768], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:18:18.881 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:18.883 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=22, tag_tid=2, entry_tid=19770' 2024-09-11T06:18:18.883 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=21, tag_tid=3, entry_tid=1' -a 'object_number=21, tag_tid=3, entry_tid=1' = 'object_number=22, tag_tid=2, entry_tid=19770' 2024-09-11T06:18:18.883 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=22, tag_tid=2, entry_tid=19770' '!=' 'object_number=22, tag_tid=2, entry_tid=19770' 2024-09-11T06:18:18.883 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:18:18.883 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.8 2024-09-11T06:18:19.685 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:18:19.685 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:18:19.685 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:18:19.685 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:18:19.686 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:18:19.686 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:18:19.686 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:18:19.686 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:18:19.686 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:18:19.686 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:18:19.686 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:18:19.686 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:18:19.686 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:18:19.686 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:18:19.686 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:18:19.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:18:19.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:18:19.690 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:18:19.690 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:18:19.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:18:19.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:18:19.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:18:19.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:18:19.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:18:19.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:18:19.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:18:19.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:18:19.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:18:19.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:18:19.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:18:19.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:18:19.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:18:19.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:18:19.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:18:19.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:18:19.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:18:19.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:18:19.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:18:19.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:18:19.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:18:19.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:18:19.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:18:19.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:18:19.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:18:19.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:18:19.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:18:19.693 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:18:19.693 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:18:19.693 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:18:19.716 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:18:19.716 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:18:19.716 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:18:19.716 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:18:19.845 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:18:19.846 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:18:19.846 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:19.846 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:19.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:19.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:18:19.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:19.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:19.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:19.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:18:19.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:18:19.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:18:19.848 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:18:19.848 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:19.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:18:19.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:19.898 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 5 2024-09-11T06:18:19.898 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:18:19.898 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:18:19.898 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:18:19.898 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:18:19.901 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:19.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:18:19.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:18:19.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:19.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:19.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:19.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:18:19.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:18:19.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:18:19.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:18:19.905 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:18:19.905 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:18:19.905 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:18:19.905 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:18:19.905 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:19.905 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:18:19.906 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:19.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 5 2024-09-11T06:18:19.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 5 2024-09-11T06:18:19.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:18:19.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:18:19.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=21, tag_tid=3, entry_tid=1], [object_number=20, tag_tid=3, entry_tid=0], [object_number=22, tag_tid=2, entry_tid=19770], [object_number=23, tag_tid=2, entry_tid=19767]]], state=connected] 2024-09-11T06:18:19.954 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:18:19.956 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:18:19.956 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=21, tag_tid=3, entry_tid=1' -a 'object_number=21, tag_tid=3, entry_tid=1' = 'object_number=21, tag_tid=3, entry_tid=1' 2024-09-11T06:18:19.956 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:18:19.957 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_snap_present cluster1 mirror test snap1 2024-09-11T06:18:19.957 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:18:19.957 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:18:19.957 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:18:19.957 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap1 2024-09-11T06:18:19.957 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:18:19.957 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T06:18:19.957 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:18:20.957 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 info mirror/test@snap1 2024-09-11T06:18:21.023 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:rbd image 'test': 2024-09-11T06:18:21.024 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: size 512 MiB in 128 objects 2024-09-11T06:18:21.024 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: order 22 (4 MiB objects) 2024-09-11T06:18:21.024 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: snapshot_count: 1 2024-09-11T06:18:21.024 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: id: 10a618cfe337 2024-09-11T06:18:21.024 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: block_name_prefix: rbd_data.10a618cfe337 2024-09-11T06:18:21.024 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: format: 2 2024-09-11T06:18:21.024 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: features: layering, exclusive-lock, journaling 2024-09-11T06:18:21.024 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: op_features: 2024-09-11T06:18:21.024 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: flags: 2024-09-11T06:18:21.024 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: create_timestamp: Wed Sep 11 06:16:46 2024 2024-09-11T06:18:21.025 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: access_timestamp: Wed Sep 11 06:16:46 2024 2024-09-11T06:18:21.025 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: modify_timestamp: Wed Sep 11 06:17:47 2024 2024-09-11T06:18:21.025 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: protected: False 2024-09-11T06:18:21.025 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: journal: 10a618cfe337 2024-09-11T06:18:21.025 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring state: enabled 2024-09-11T06:18:21.025 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring mode: journal 2024-09-11T06:18:21.025 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring global id: a0f7d515-874e-4c67-a26b-4958c7ab3083 2024-09-11T06:18:21.025 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring primary: false 2024-09-11T06:18:21.029 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:18:21.029 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n '' ']' 2024-09-11T06:18:21.029 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ compare_image_snaps mirror test snap1 2024-09-11T06:18:21.029 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:18:21.029 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:18:21.029 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap1 2024-09-11T06:18:21.030 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local ret=0 2024-09-11T06:18:21.030 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local rmt_export=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export 2024-09-11T06:18:21.030 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loc_export=/tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:18:21.030 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:18:21.030 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1 /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export 2024-09-11T06:18:21.570 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T06:18:21.581 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1 /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:18:22.868 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T06:18:22.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmp /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:18:23.169 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:18:23.306 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:18:23.306 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ clean_snap_name=snap1-clean 2024-09-11T06:18:23.306 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_snap cluster2 mirror test snap1-clean 2024-09-11T06:18:23.306 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:18:23.306 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:18:23.306 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:18:23.306 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap1-clean 2024-09-11T06:18:23.306 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap1-clean --debug-rbd=20 --debug-journaler=20 2024-09-11T06:18:24.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 10` 2024-09-11T06:18:24.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ stress_write_image cluster2 mirror test 2024-09-11T06:18:24.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:18:24.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:18:24.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:18:24.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T06:18:24.128 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local duration=8 2024-09-11T06:18:24.128 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set +e 2024-09-11T06:18:24.129 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-11T06:18:25.749 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:100 IOs, 34731008 bytes 2024-09-11T06:18:26.640 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:200 IOs, 62774272 bytes 2024-09-11T06:18:29.411 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:300 IOs, 107915264 bytes 2024-09-11T06:18:30.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:400 IOs, 131301376 bytes 2024-09-11T06:18:30.680 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:500 IOs, 132197376 bytes 2024-09-11T06:18:32.139 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ error_code=124 2024-09-11T06:18:32.139 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set -e 2024-09-11T06:18:32.139 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T06:18:32.139 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:18:32.139 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-11T06:18:32.139 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:18:32.139 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:18:32.139 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:18:32.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state_pattern=up+replaying 2024-09-11T06:18:32.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local description_pattern= 2024-09-11T06:18:32.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local service_pattern= 2024-09-11T06:18:32.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:18:32.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:18:33.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-11T06:18:33.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:18:33.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:18:33.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:18:33.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state_pattern=up+replaying 2024-09-11T06:18:33.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local description_pattern= 2024-09-11T06:18:33.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local service_pattern= 2024-09-11T06:18:33.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local status 2024-09-11T06:18:33.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ CEPH_ARGS= 2024-09-11T06:18:33.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-11T06:18:33.230 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ status='test: 2024-09-11T06:18:33.230 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: global_id: a0f7d515-874e-4c67-a26b-4958c7ab3083 2024-09-11T06:18:33.230 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: state: up+replaying 2024-09-11T06:18:33.230 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: description: replaying, {"bytes_per_second":3708881.07,"entries_behind_primary":650,"entries_per_second":228.17,"non_primary_position":{"entry_tid":19121,"object_number":21,"tag_tid":2},"primary_position":{"entry_tid":1,"object_number":21,"tag_tid":3},"seconds_until_synced":2} 2024-09-11T06:18:33.231 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: service: mirror.0 on smithi112 2024-09-11T06:18:33.231 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: last_update: 2024-09-11 06:18:14 2024-09-11T06:18:33.231 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: peer_sites: 2024-09-11T06:18:33.231 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: name: cluster2 2024-09-11T06:18:33.231 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: state: up+stopped 2024-09-11T06:18:33.231 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: description: local image is primary 2024-09-11T06:18:33.231 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: last_update: 2024-09-11 06:18:30' 2024-09-11T06:18:33.231 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ grep '^ state: .*up+replaying' 2024-09-11T06:18:33.231 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: state: up+replaying 2024-09-11T06:18:33.232 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ grep '^ description: .*' 2024-09-11T06:18:33.232 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: description: replaying, {"bytes_per_second":3708881.07,"entries_behind_primary":650,"entries_per_second":228.17,"non_primary_position":{"entry_tid":19121,"object_number":21,"tag_tid":2},"primary_position":{"entry_tid":1,"object_number":21,"tag_tid":3},"seconds_until_synced":2} 2024-09-11T06:18:33.232 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n '' ']' 2024-09-11T06:18:33.233 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ echo up+replaying 2024-09-11T06:18:33.233 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ grep '^up+' 2024-09-11T06:18:33.233 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:up+replaying 2024-09-11T06:18:33.233 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ grep 'service: *mirror..* on ' 2024-09-11T06:18:33.234 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: service: mirror.0 on smithi112 2024-09-11T06:18:33.235 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local last_update 2024-09-11T06:18:33.235 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-11T06:18:33.237 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_update='2024-09-11 06:18:14' 2024-09-11T06:18:33.237 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-11 06:18:14' 2024-09-11T06:18:33.237 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:18:33.237 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:18:33.237 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:18:33.237 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state_pattern=up+replaying 2024-09-11T06:18:33.237 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'prev_last_update=2024-09-11 06:18:14' 2024-09-11T06:18:33.237 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local status 2024-09-11T06:18:33.237 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ CEPH_ARGS= 2024-09-11T06:18:33.238 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-11T06:18:33.318 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ status='OKOKOK142554262mirror.0smithi11219.1.1-298-gb53b6d3etrueOK42594265mirror.1smithi11219.1.1-298-gb53b6d3efalseOKtesta0f7d515-874e-4c67-a26b-4958c7ab3083up+replayingreplaying, {"bytes_per_second":3708881.07,"entries_behind_primary":650,"entries_per_second":228.17,"non_primary_position":{"entry_tid":19121,"object_number":21,"tag_tid":2},"primary_position":{"entry_tid":1,"object_number":21,"tag_tid":3},"seconds_until_synced":2}42554262mirror.0smithi1122024-09-11 06:18:14cluster20bfc6eda-f3c5-424a-8ff7-5c4d3d9fd488up+stoppedlocal image is primary2024-09-11 06:18:30' 2024-09-11T06:18:33.318 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local last_update state 2024-09-11T06:18:33.318 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-11T06:18:33.320 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_update='2024-09-11 06:18:14' 2024-09-11T06:18:33.321 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-11T06:18:33.324 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ state=up+replaying 2024-09-11T06:18:33.324 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ echo up+replaying 2024-09-11T06:18:33.324 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ grep up+replaying 2024-09-11T06:18:33.325 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:up+replaying 2024-09-11T06:18:33.326 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:18:33.326 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:18:33.326 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ snap_name=snap2 2024-09-11T06:18:33.326 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_snap cluster2 mirror test snap2 2024-09-11T06:18:33.326 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:18:33.326 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:18:33.326 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:18:33.326 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap2 2024-09-11T06:18:33.326 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap2 --debug-rbd=20 --debug-journaler=20 2024-09-11T06:19:04.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-11T06:19:04.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:19:04.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:19:04.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:19:04.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-11T06:19:04.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:19:04.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:19:04.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:19:04.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=started 2024-09-11T06:19:04.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:19:04.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:19:04.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:19:05.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-11T06:19:05.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:19:05.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:19:05.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:19:05.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=started 2024-09-11T06:19:05.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local status_result 2024-09-11T06:19:05.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=stopped 2024-09-11T06:19:05.221 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-11T06:19:05.221 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_instance=cluster1 2024-09-11T06:19:05.221 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:19:05.222 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -i state 2024-09-11T06:19:05.222 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:19:05.222 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance=cluster1 2024-09-11T06:19:05.222 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local loop_instance 2024-09-11T06:19:05.222 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:19:05.222 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sleep 0 2024-09-11T06:19:05.222 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:19:05.222 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ seq 0 1 2024-09-11T06:19:05.225 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:19:05.226 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-11T06:19:05.226 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1:0 2024-09-11T06:19:05.226 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:19:05.226 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:19:05.226 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:19:05.226 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:19:05.226 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=cluster 2024-09-11T06:19:05.226 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:19:05.226 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:19:05.226 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:19:05.226 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:19:05.226 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval cluster=cluster1 2024-09-11T06:19:05.226 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:19:05.227 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:19:05.227 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:19:05.227 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:19:05.227 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local local_cluster=cluster1:0 2024-09-11T06:19:05.227 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster=cluster1 2024-09-11T06:19:05.227 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance 2024-09-11T06:19:05.227 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:19:05.227 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local val=cluster1:0 2024-09-11T06:19:05.227 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T06:19:05.227 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance_var_name=instance 2024-09-11T06:19:05.227 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:19:05.227 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:19:05.227 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T06:19:05.227 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval local_cluster=cluster1 2024-09-11T06:19:05.228 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ local_cluster=cluster1 2024-09-11T06:19:05.228 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval instance=0 2024-09-11T06:19:05.228 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ instance=0 2024-09-11T06:19:05.228 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:19:05.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:19:05.250 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:19:05.250 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:19:05.250 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-11T06:19:05.369 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ return 0 2024-09-11T06:19:05.369 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T06:19:05.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ echo ' "state": "Replaying"' 2024-09-11T06:19:05.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ grep -i Replaying 2024-09-11T06:19:05.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: "state": "Replaying" 2024-09-11T06:19:05.373 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_state=started 2024-09-11T06:19:05.373 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test started = started 2024-09-11T06:19:05.373 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:19:05.373 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-11T06:19:05.373 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:19:05.374 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:19:05.374 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:19:05.374 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:19:05.374 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' journal = journal ']' 2024-09-11T06:19:05.374 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-09-11T06:19:05.374 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:19:05.375 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:19:05.375 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:19:05.375 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:19:05.375 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-11T06:19:05.375 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-11T06:19:05.375 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:19:05.376 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:19:05.376 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:19:05.574 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:19:05.574 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:19:05.574 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:19:05.574 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:19:05.574 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:19:05.574 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:19:05.574 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:19:05.574 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:19:05.574 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:19:05.575 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:19:05.575 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:19:05.575 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:19:05.575 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:19:05.575 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:19:05.575 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:19:05.575 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:19:05.576 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:19:05.578 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:19:05.578 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:19:05.578 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:19:05.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:19:05.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:19:05.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:19:05.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:19:05.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:19:05.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:19:05.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:19:05.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:19:05.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:19:05.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:19:05.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:19:05.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:19:05.580 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:19:05.580 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:19:05.580 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:19:05.580 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:19:05.580 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:19:05.580 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:19:05.580 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:19:05.580 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:19:05.580 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:19:05.580 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:19:05.580 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:19:05.580 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:19:05.580 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:19:05.581 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:19:05.581 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:19:05.581 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:19:05.581 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:19:05.604 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:19:05.605 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:19:05.605 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:19:05.605 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:19:05.732 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:19:05.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:19:05.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:05.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:05.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:05.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:19:05.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:05.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:05.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:05.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:19:05.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:19:05.734 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:19:05.734 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:19:05.735 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:05.736 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:19:05.736 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:05.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 7 2024-09-11T06:19:05.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 8 2024-09-11T06:19:05.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:19:05.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=6, entry_tid=1], [object_number=32, tag_tid=6, entry_tid=0], [object_number=34, tag_tid=5, entry_tid=10506], [object_number=35, tag_tid=5, entry_tid=10503]]], state=connected] 2024-09-11T06:19:05.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=30, tag_tid=5, entry_tid=7806], [object_number=29, tag_tid=5, entry_tid=7805], [object_number=28, tag_tid=5, entry_tid=7804], [object_number=31, tag_tid=5, entry_tid=7803]]], state=connected] 2024-09-11T06:19:05.789 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:05.791 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=33, tag_tid=6, entry_tid=1' 2024-09-11T06:19:05.792 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:19:05.792 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:05.792 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:05.792 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:05.792 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:19:05.792 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:05.792 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:05.792 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:05.792 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:19:05.792 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:19:05.792 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:19:05.792 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:19:05.793 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:05.793 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:19:05.793 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:05.839 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 7 2024-09-11T06:19:05.840 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 8 2024-09-11T06:19:05.840 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:19:05.840 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=6, entry_tid=1], [object_number=32, tag_tid=6, entry_tid=0], [object_number=34, tag_tid=5, entry_tid=10506], [object_number=35, tag_tid=5, entry_tid=10503]]], state=connected] 2024-09-11T06:19:05.840 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=30, tag_tid=5, entry_tid=7806], [object_number=29, tag_tid=5, entry_tid=7805], [object_number=28, tag_tid=5, entry_tid=7804], [object_number=31, tag_tid=5, entry_tid=7803]]], state=connected] 2024-09-11T06:19:05.843 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:05.845 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=30, tag_tid=5, entry_tid=7806' 2024-09-11T06:19:05.845 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=33, tag_tid=6, entry_tid=1' -a 'object_number=33, tag_tid=6, entry_tid=1' = 'object_number=30, tag_tid=5, entry_tid=7806' 2024-09-11T06:19:05.845 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=30, tag_tid=5, entry_tid=7806' '!=' '' 2024-09-11T06:19:05.845 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:19:05.845 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=30, tag_tid=5, entry_tid=7806' = '' 2024-09-11T06:19:05.845 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=30, tag_tid=5, entry_tid=7806' 2024-09-11T06:19:05.846 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=33, tag_tid=6, entry_tid=1' 2024-09-11T06:19:05.846 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:19:05.846 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:19:05.846 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=6 2024-09-11T06:19:05.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=30, tag_tid=5, entry_tid=7806' 2024-09-11T06:19:05.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:19:05.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:19:05.848 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=5 2024-09-11T06:19:05.848 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=33, tag_tid=6, entry_tid=1' 2024-09-11T06:19:05.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:19:05.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:19:05.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:19:05.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=30, tag_tid=5, entry_tid=7806' 2024-09-11T06:19:05.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:19:05.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:19:05.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=7806 2024-09-11T06:19:05.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 6 = 5 -a 1 -le 7806 2024-09-11T06:19:05.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:19:05.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:19:05.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:19:06.054 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:19:06.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:19:06.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:19:06.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:19:06.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:19:06.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:19:06.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:19:06.056 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:19:06.056 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:19:06.056 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:19:06.056 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:19:06.056 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:19:06.056 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:19:06.057 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:19:06.057 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:19:06.060 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:19:06.060 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:19:06.060 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:19:06.060 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:19:06.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:19:06.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:19:06.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:19:06.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:19:06.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:19:06.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:19:06.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:19:06.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:19:06.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:19:06.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:19:06.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:19:06.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:19:06.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:19:06.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:19:06.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:19:06.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:19:06.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:19:06.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:19:06.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:19:06.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:19:06.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:19:06.063 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:19:06.063 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:19:06.063 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:19:06.063 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:19:06.063 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:19:06.063 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:19:06.063 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:19:06.063 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:19:06.063 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:19:06.085 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:19:06.085 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:19:06.085 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:19:06.085 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:19:06.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:19:06.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:19:06.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:06.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:06.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:06.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:19:06.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:06.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:06.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:06.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:19:06.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:19:06.221 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:19:06.221 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:19:06.222 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:06.222 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:06.223 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:19:06.274 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 7 2024-09-11T06:19:06.275 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 8 2024-09-11T06:19:06.275 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:19:06.275 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=6, entry_tid=1], [object_number=32, tag_tid=6, entry_tid=0], [object_number=34, tag_tid=5, entry_tid=10506], [object_number=35, tag_tid=5, entry_tid=10503]]], state=connected] 2024-09-11T06:19:06.275 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=31, tag_tid=5, entry_tid=7919], [object_number=30, tag_tid=5, entry_tid=7918], [object_number=29, tag_tid=5, entry_tid=7917], [object_number=28, tag_tid=5, entry_tid=7916]]], state=connected] 2024-09-11T06:19:06.278 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:06.279 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=33, tag_tid=6, entry_tid=1' 2024-09-11T06:19:06.281 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:19:06.281 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:06.281 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:06.281 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:06.281 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:19:06.281 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:06.281 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:06.281 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:06.281 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:19:06.282 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:19:06.282 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:19:06.282 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:19:06.282 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:06.282 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:19:06.283 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:06.327 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 7 2024-09-11T06:19:06.328 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 8 2024-09-11T06:19:06.328 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:19:06.328 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=6, entry_tid=1], [object_number=32, tag_tid=6, entry_tid=0], [object_number=34, tag_tid=5, entry_tid=10506], [object_number=35, tag_tid=5, entry_tid=10503]]], state=connected] 2024-09-11T06:19:06.328 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=31, tag_tid=5, entry_tid=7919], [object_number=30, tag_tid=5, entry_tid=7918], [object_number=29, tag_tid=5, entry_tid=7917], [object_number=28, tag_tid=5, entry_tid=7916]]], state=connected] 2024-09-11T06:19:06.331 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:06.333 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=31, tag_tid=5, entry_tid=7919' 2024-09-11T06:19:06.333 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=33, tag_tid=6, entry_tid=1' -a 'object_number=33, tag_tid=6, entry_tid=1' = 'object_number=31, tag_tid=5, entry_tid=7919' 2024-09-11T06:19:06.333 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=31, tag_tid=5, entry_tid=7919' '!=' 'object_number=30, tag_tid=5, entry_tid=7806' 2024-09-11T06:19:06.333 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:19:06.333 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=31, tag_tid=5, entry_tid=7919' = 'object_number=30, tag_tid=5, entry_tid=7806' 2024-09-11T06:19:06.333 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=31, tag_tid=5, entry_tid=7919' 2024-09-11T06:19:06.334 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=33, tag_tid=6, entry_tid=1' 2024-09-11T06:19:06.334 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:19:06.334 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:19:06.334 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=6 2024-09-11T06:19:06.335 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=31, tag_tid=5, entry_tid=7919' 2024-09-11T06:19:06.335 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:19:06.335 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:19:06.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=5 2024-09-11T06:19:06.337 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=33, tag_tid=6, entry_tid=1' 2024-09-11T06:19:06.337 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:19:06.337 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:19:06.338 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:19:06.338 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=31, tag_tid=5, entry_tid=7919' 2024-09-11T06:19:06.338 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:19:06.339 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:19:06.340 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=7919 2024-09-11T06:19:06.340 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 6 = 5 -a 1 -le 7919 2024-09-11T06:19:06.340 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:19:06.340 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:19:06.340 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:19:06.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:19:06.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:19:06.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:19:06.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:19:06.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:19:06.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:19:06.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:19:06.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:19:06.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:19:06.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:19:06.543 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:19:06.543 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:19:06.543 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:19:06.543 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:19:06.543 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:19:06.543 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:19:06.544 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:19:06.546 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:19:06.547 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:19:06.547 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:19:06.547 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:19:06.547 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:19:06.547 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:19:06.547 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:19:06.547 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:19:06.547 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:19:06.547 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:19:06.548 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:19:06.548 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:19:06.548 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:19:06.548 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:19:06.548 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:19:06.548 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:19:06.548 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:19:06.548 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:19:06.548 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:19:06.548 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:19:06.548 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:19:06.548 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:19:06.548 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:19:06.549 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:19:06.549 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:19:06.549 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:19:06.549 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:19:06.549 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:19:06.549 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:19:06.549 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:19:06.549 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:19:06.549 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:19:06.572 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:19:06.573 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:19:06.573 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:19:06.573 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:19:06.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:19:06.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:19:06.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:06.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:06.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:06.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:19:06.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:06.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:06.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:06.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:19:06.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:19:06.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:19:06.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:19:06.707 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:06.707 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:19:06.707 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:06.753 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 7 2024-09-11T06:19:06.753 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 8 2024-09-11T06:19:06.753 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:19:06.754 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=6, entry_tid=1], [object_number=32, tag_tid=6, entry_tid=0], [object_number=34, tag_tid=5, entry_tid=10506], [object_number=35, tag_tid=5, entry_tid=10503]]], state=connected] 2024-09-11T06:19:06.754 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=30, tag_tid=5, entry_tid=8038], [object_number=29, tag_tid=5, entry_tid=8037], [object_number=28, tag_tid=5, entry_tid=8036], [object_number=31, tag_tid=5, entry_tid=8035]]], state=connected] 2024-09-11T06:19:06.756 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:06.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=33, tag_tid=6, entry_tid=1' 2024-09-11T06:19:06.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:19:06.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:06.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:06.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:06.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:19:06.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:06.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:06.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:06.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:19:06.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:19:06.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:19:06.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:19:06.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:06.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:19:06.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:06.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 7 2024-09-11T06:19:06.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 8 2024-09-11T06:19:06.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:19:06.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=6, entry_tid=1], [object_number=32, tag_tid=6, entry_tid=0], [object_number=34, tag_tid=5, entry_tid=10506], [object_number=35, tag_tid=5, entry_tid=10503]]], state=connected] 2024-09-11T06:19:06.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=30, tag_tid=5, entry_tid=8038], [object_number=29, tag_tid=5, entry_tid=8037], [object_number=28, tag_tid=5, entry_tid=8036], [object_number=31, tag_tid=5, entry_tid=8035]]], state=connected] 2024-09-11T06:19:06.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:06.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=30, tag_tid=5, entry_tid=8038' 2024-09-11T06:19:06.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=33, tag_tid=6, entry_tid=1' -a 'object_number=33, tag_tid=6, entry_tid=1' = 'object_number=30, tag_tid=5, entry_tid=8038' 2024-09-11T06:19:06.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=30, tag_tid=5, entry_tid=8038' '!=' 'object_number=31, tag_tid=5, entry_tid=7919' 2024-09-11T06:19:06.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:19:06.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=30, tag_tid=5, entry_tid=8038' = 'object_number=31, tag_tid=5, entry_tid=7919' 2024-09-11T06:19:06.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=30, tag_tid=5, entry_tid=8038' 2024-09-11T06:19:06.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=33, tag_tid=6, entry_tid=1' 2024-09-11T06:19:06.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:19:06.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:19:06.812 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=6 2024-09-11T06:19:06.812 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=30, tag_tid=5, entry_tid=8038' 2024-09-11T06:19:06.813 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:19:06.813 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:19:06.813 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=5 2024-09-11T06:19:06.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=33, tag_tid=6, entry_tid=1' 2024-09-11T06:19:06.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:19:06.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:19:06.815 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:19:06.816 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=30, tag_tid=5, entry_tid=8038' 2024-09-11T06:19:06.816 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:19:06.816 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:19:06.818 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=8038 2024-09-11T06:19:06.818 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 6 = 5 -a 1 -le 8038 2024-09-11T06:19:06.818 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:19:06.818 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:19:06.818 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:19:07.019 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:19:07.019 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:19:07.019 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:19:07.019 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:19:07.019 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:19:07.020 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:19:07.020 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:19:07.020 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:19:07.020 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:19:07.020 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:19:07.020 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:19:07.020 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:19:07.020 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:19:07.020 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:19:07.020 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:19:07.021 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:19:07.021 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:19:07.022 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:19:07.022 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:19:07.023 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:19:07.023 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:19:07.023 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:19:07.023 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:19:07.023 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:19:07.023 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:19:07.023 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:19:07.023 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:19:07.023 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:19:07.023 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:19:07.023 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:19:07.023 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:19:07.023 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:19:07.024 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:19:07.025 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:19:07.025 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:19:07.025 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:19:07.025 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:19:07.025 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:19:07.025 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:19:07.025 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:19:07.025 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:19:07.026 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:19:07.026 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:19:07.026 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:19:07.026 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:19:07.026 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:19:07.026 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:19:07.026 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:19:07.026 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:19:07.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:19:07.050 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:19:07.050 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:19:07.050 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:19:07.177 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:19:07.178 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:19:07.178 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:07.178 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:07.179 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:07.179 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:19:07.179 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:07.179 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:07.179 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:07.179 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:19:07.179 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:19:07.179 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:19:07.180 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:19:07.180 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:07.181 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:19:07.181 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:07.227 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 7 2024-09-11T06:19:07.227 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 8 2024-09-11T06:19:07.227 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:19:07.227 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=6, entry_tid=1], [object_number=32, tag_tid=6, entry_tid=0], [object_number=34, tag_tid=5, entry_tid=10506], [object_number=35, tag_tid=5, entry_tid=10503]]], state=connected] 2024-09-11T06:19:07.227 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=28, tag_tid=5, entry_tid=8148], [object_number=31, tag_tid=5, entry_tid=8147], [object_number=30, tag_tid=5, entry_tid=8146], [object_number=29, tag_tid=5, entry_tid=8145]]], state=connected] 2024-09-11T06:19:07.230 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:07.231 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=33, tag_tid=6, entry_tid=1' 2024-09-11T06:19:07.233 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:19:07.233 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:07.233 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:07.233 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:07.233 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:19:07.233 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:07.233 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:07.233 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:07.233 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:19:07.233 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:19:07.233 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:19:07.233 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:19:07.234 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:07.234 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:19:07.234 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:07.279 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 7 2024-09-11T06:19:07.279 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 8 2024-09-11T06:19:07.279 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:19:07.279 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=6, entry_tid=1], [object_number=32, tag_tid=6, entry_tid=0], [object_number=34, tag_tid=5, entry_tid=10506], [object_number=35, tag_tid=5, entry_tid=10503]]], state=connected] 2024-09-11T06:19:07.279 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=28, tag_tid=5, entry_tid=8148], [object_number=31, tag_tid=5, entry_tid=8147], [object_number=30, tag_tid=5, entry_tid=8146], [object_number=29, tag_tid=5, entry_tid=8145]]], state=connected] 2024-09-11T06:19:07.282 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:07.284 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=28, tag_tid=5, entry_tid=8148' 2024-09-11T06:19:07.284 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=33, tag_tid=6, entry_tid=1' -a 'object_number=33, tag_tid=6, entry_tid=1' = 'object_number=28, tag_tid=5, entry_tid=8148' 2024-09-11T06:19:07.284 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=28, tag_tid=5, entry_tid=8148' '!=' 'object_number=30, tag_tid=5, entry_tid=8038' 2024-09-11T06:19:07.284 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:19:07.284 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=28, tag_tid=5, entry_tid=8148' = 'object_number=30, tag_tid=5, entry_tid=8038' 2024-09-11T06:19:07.284 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=28, tag_tid=5, entry_tid=8148' 2024-09-11T06:19:07.285 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=33, tag_tid=6, entry_tid=1' 2024-09-11T06:19:07.285 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:19:07.285 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:19:07.285 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=6 2024-09-11T06:19:07.286 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=28, tag_tid=5, entry_tid=8148' 2024-09-11T06:19:07.286 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:19:07.286 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:19:07.287 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=5 2024-09-11T06:19:07.288 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=33, tag_tid=6, entry_tid=1' 2024-09-11T06:19:07.288 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:19:07.288 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:19:07.289 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:19:07.289 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=28, tag_tid=5, entry_tid=8148' 2024-09-11T06:19:07.290 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:19:07.290 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:19:07.291 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=8148 2024-09-11T06:19:07.291 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 6 = 5 -a 1 -le 8148 2024-09-11T06:19:07.291 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:19:07.291 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:19:07.291 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:19:07.493 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:19:07.493 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:19:07.493 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:19:07.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:19:07.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:19:07.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:19:07.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:19:07.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:19:07.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:19:07.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:19:07.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:19:07.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:19:07.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:19:07.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:19:07.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:19:07.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:19:07.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:19:07.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:19:07.498 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:19:07.498 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:19:07.498 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:19:07.498 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:19:07.498 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:19:07.498 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:19:07.498 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:19:07.498 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:19:07.498 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:19:07.498 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:19:07.498 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:19:07.498 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:19:07.498 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:19:07.498 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:19:07.499 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:19:07.499 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:19:07.499 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:19:07.499 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:19:07.499 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:19:07.499 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:19:07.499 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:19:07.499 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:19:07.499 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:19:07.499 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:19:07.499 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:19:07.499 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:19:07.499 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:19:07.499 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:19:07.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:19:07.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:19:07.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:19:07.524 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:19:07.525 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:19:07.525 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:19:07.525 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:19:07.656 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:19:07.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:19:07.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:07.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:07.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:07.658 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:19:07.658 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:07.658 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:07.658 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:07.658 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:19:07.658 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:19:07.658 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:19:07.659 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:19:07.660 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:07.661 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:19:07.661 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:07.707 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 7 2024-09-11T06:19:07.707 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 8 2024-09-11T06:19:07.707 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:19:07.707 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=6, entry_tid=1], [object_number=32, tag_tid=6, entry_tid=0], [object_number=34, tag_tid=5, entry_tid=10506], [object_number=35, tag_tid=5, entry_tid=10503]]], state=connected] 2024-09-11T06:19:07.707 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=28, tag_tid=5, entry_tid=8260], [object_number=31, tag_tid=5, entry_tid=8259], [object_number=30, tag_tid=5, entry_tid=8258], [object_number=29, tag_tid=5, entry_tid=8257]]], state=connected] 2024-09-11T06:19:07.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:07.712 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=33, tag_tid=6, entry_tid=1' 2024-09-11T06:19:07.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:19:07.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:07.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:07.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:07.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:19:07.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:07.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:07.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:07.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:19:07.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:19:07.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:19:07.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:19:07.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:07.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:19:07.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:07.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 7 2024-09-11T06:19:07.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 8 2024-09-11T06:19:07.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:19:07.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=6, entry_tid=1], [object_number=32, tag_tid=6, entry_tid=0], [object_number=34, tag_tid=5, entry_tid=10506], [object_number=35, tag_tid=5, entry_tid=10503]]], state=connected] 2024-09-11T06:19:07.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=28, tag_tid=5, entry_tid=8260], [object_number=31, tag_tid=5, entry_tid=8259], [object_number=30, tag_tid=5, entry_tid=8258], [object_number=29, tag_tid=5, entry_tid=8257]]], state=connected] 2024-09-11T06:19:07.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:07.765 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=28, tag_tid=5, entry_tid=8260' 2024-09-11T06:19:07.765 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=33, tag_tid=6, entry_tid=1' -a 'object_number=33, tag_tid=6, entry_tid=1' = 'object_number=28, tag_tid=5, entry_tid=8260' 2024-09-11T06:19:07.765 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=28, tag_tid=5, entry_tid=8260' '!=' 'object_number=28, tag_tid=5, entry_tid=8148' 2024-09-11T06:19:07.765 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:19:07.765 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=28, tag_tid=5, entry_tid=8260' = 'object_number=28, tag_tid=5, entry_tid=8148' 2024-09-11T06:19:07.765 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=28, tag_tid=5, entry_tid=8260' 2024-09-11T06:19:07.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=33, tag_tid=6, entry_tid=1' 2024-09-11T06:19:07.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:19:07.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:19:07.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=6 2024-09-11T06:19:07.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=28, tag_tid=5, entry_tid=8260' 2024-09-11T06:19:07.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:19:07.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:19:07.768 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=5 2024-09-11T06:19:07.769 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=33, tag_tid=6, entry_tid=1' 2024-09-11T06:19:07.769 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:19:07.769 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:19:07.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:19:07.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=28, tag_tid=5, entry_tid=8260' 2024-09-11T06:19:07.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:19:07.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:19:07.772 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=8260 2024-09-11T06:19:07.772 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 6 = 5 -a 1 -le 8260 2024-09-11T06:19:07.772 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:19:07.772 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:19:07.772 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:19:07.974 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:19:07.974 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:19:07.974 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:19:07.974 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:19:07.974 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:19:07.974 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:19:07.974 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:19:07.974 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:19:07.974 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:19:07.974 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:19:07.974 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:19:07.975 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:19:07.975 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:19:07.975 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:19:07.975 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:19:07.975 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:19:07.975 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:19:07.977 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:19:07.977 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:19:07.977 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:19:07.977 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:19:07.977 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:19:07.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:19:07.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:19:07.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:19:07.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:19:07.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:19:07.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:19:07.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:19:07.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:19:07.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:19:07.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:19:07.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:19:07.979 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:19:07.979 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:19:07.980 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:19:07.980 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:19:07.980 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:19:07.980 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:19:07.980 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:19:07.980 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:19:07.980 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:19:07.980 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:19:07.980 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:19:07.980 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:19:07.980 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:19:07.980 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:19:07.980 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:19:07.981 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:19:08.004 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:19:08.005 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:19:08.005 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:19:08.005 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:19:08.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:19:08.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:19:08.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:08.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:08.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:08.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:19:08.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:08.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:08.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:08.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:19:08.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:19:08.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:19:08.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:19:08.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:08.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:19:08.139 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:08.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 7 2024-09-11T06:19:08.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 8 2024-09-11T06:19:08.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:19:08.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=6, entry_tid=1], [object_number=32, tag_tid=6, entry_tid=0], [object_number=34, tag_tid=5, entry_tid=10506], [object_number=35, tag_tid=5, entry_tid=10503]]], state=connected] 2024-09-11T06:19:08.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=31, tag_tid=5, entry_tid=8351], [object_number=30, tag_tid=5, entry_tid=8350], [object_number=29, tag_tid=5, entry_tid=8349], [object_number=28, tag_tid=5, entry_tid=8348]]], state=connected] 2024-09-11T06:19:08.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:08.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=33, tag_tid=6, entry_tid=1' 2024-09-11T06:19:08.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:19:08.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:08.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:08.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:08.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:19:08.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:08.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:08.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:08.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:19:08.199 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:19:08.199 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:19:08.199 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:19:08.199 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:08.199 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:19:08.199 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:08.246 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 7 2024-09-11T06:19:08.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 8 2024-09-11T06:19:08.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:19:08.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=6, entry_tid=1], [object_number=32, tag_tid=6, entry_tid=0], [object_number=34, tag_tid=5, entry_tid=10506], [object_number=35, tag_tid=5, entry_tid=10503]]], state=connected] 2024-09-11T06:19:08.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=31, tag_tid=5, entry_tid=8351], [object_number=30, tag_tid=5, entry_tid=8350], [object_number=29, tag_tid=5, entry_tid=8349], [object_number=28, tag_tid=5, entry_tid=8348]]], state=connected] 2024-09-11T06:19:08.250 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:08.252 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=31, tag_tid=5, entry_tid=8351' 2024-09-11T06:19:08.252 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=33, tag_tid=6, entry_tid=1' -a 'object_number=33, tag_tid=6, entry_tid=1' = 'object_number=31, tag_tid=5, entry_tid=8351' 2024-09-11T06:19:08.252 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=31, tag_tid=5, entry_tid=8351' '!=' 'object_number=28, tag_tid=5, entry_tid=8260' 2024-09-11T06:19:08.252 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:19:08.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=31, tag_tid=5, entry_tid=8351' = 'object_number=28, tag_tid=5, entry_tid=8260' 2024-09-11T06:19:08.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=31, tag_tid=5, entry_tid=8351' 2024-09-11T06:19:08.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=33, tag_tid=6, entry_tid=1' 2024-09-11T06:19:08.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:19:08.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:19:08.254 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=6 2024-09-11T06:19:08.254 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=31, tag_tid=5, entry_tid=8351' 2024-09-11T06:19:08.254 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:19:08.255 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:19:08.255 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=5 2024-09-11T06:19:08.256 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=33, tag_tid=6, entry_tid=1' 2024-09-11T06:19:08.256 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:19:08.256 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:19:08.257 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:19:08.258 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=31, tag_tid=5, entry_tid=8351' 2024-09-11T06:19:08.258 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:19:08.258 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:19:08.259 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=8351 2024-09-11T06:19:08.259 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 6 = 5 -a 1 -le 8351 2024-09-11T06:19:08.259 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:19:08.259 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:19:08.260 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:19:08.461 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:19:08.461 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:19:08.462 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:19:08.462 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:19:08.462 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:19:08.462 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:19:08.462 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:19:08.462 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:19:08.462 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:19:08.462 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:19:08.462 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:19:08.462 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:19:08.462 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:19:08.462 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:19:08.463 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:19:08.463 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:19:08.463 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:19:08.466 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:19:08.467 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:19:08.467 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:19:08.467 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:19:08.467 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:19:08.467 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:19:08.467 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:19:08.467 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:19:08.467 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:19:08.467 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:19:08.467 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:19:08.467 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:19:08.467 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:19:08.467 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:19:08.468 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:19:08.468 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:19:08.468 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:19:08.468 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:19:08.468 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:19:08.468 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:19:08.468 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:19:08.468 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:19:08.468 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:19:08.468 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:19:08.468 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:19:08.468 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:19:08.468 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:19:08.469 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:19:08.469 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:19:08.469 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:19:08.469 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:19:08.469 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:19:08.491 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:19:08.492 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:19:08.492 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:19:08.492 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:19:08.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:19:08.706 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:19:08.706 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:08.706 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:08.706 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:08.706 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:19:08.706 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:08.706 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:08.706 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:08.706 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:19:08.707 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:19:08.707 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:19:08.707 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:19:08.707 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:08.708 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:19:08.708 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:08.753 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 8 2024-09-11T06:19:08.753 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 8 2024-09-11T06:19:08.753 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:19:08.753 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=6, entry_tid=1], [object_number=32, tag_tid=6, entry_tid=0], [object_number=34, tag_tid=5, entry_tid=10506], [object_number=35, tag_tid=5, entry_tid=10503]]], state=connected] 2024-09-11T06:19:08.753 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=35, tag_tid=5, entry_tid=8379], [object_number=34, tag_tid=5, entry_tid=8378], [object_number=33, tag_tid=5, entry_tid=8377], [object_number=32, tag_tid=5, entry_tid=8376]]], state=connected] 2024-09-11T06:19:08.756 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:08.758 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=33, tag_tid=6, entry_tid=1' 2024-09-11T06:19:08.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:19:08.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:08.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:08.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:08.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:19:08.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:08.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:08.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:08.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:19:08.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:19:08.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:19:08.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:19:08.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:08.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:19:08.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:08.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 8 2024-09-11T06:19:08.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 8 2024-09-11T06:19:08.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:19:08.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=6, entry_tid=1], [object_number=32, tag_tid=6, entry_tid=0], [object_number=34, tag_tid=5, entry_tid=10506], [object_number=35, tag_tid=5, entry_tid=10503]]], state=connected] 2024-09-11T06:19:08.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=35, tag_tid=5, entry_tid=8379], [object_number=34, tag_tid=5, entry_tid=8378], [object_number=33, tag_tid=5, entry_tid=8377], [object_number=32, tag_tid=5, entry_tid=8376]]], state=connected] 2024-09-11T06:19:08.808 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:08.810 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=35, tag_tid=5, entry_tid=8379' 2024-09-11T06:19:08.810 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=33, tag_tid=6, entry_tid=1' -a 'object_number=33, tag_tid=6, entry_tid=1' = 'object_number=35, tag_tid=5, entry_tid=8379' 2024-09-11T06:19:08.810 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=35, tag_tid=5, entry_tid=8379' '!=' 'object_number=31, tag_tid=5, entry_tid=8351' 2024-09-11T06:19:08.810 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:19:08.810 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=35, tag_tid=5, entry_tid=8379' = 'object_number=31, tag_tid=5, entry_tid=8351' 2024-09-11T06:19:08.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=35, tag_tid=5, entry_tid=8379' 2024-09-11T06:19:08.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=33, tag_tid=6, entry_tid=1' 2024-09-11T06:19:08.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:19:08.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:19:08.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=6 2024-09-11T06:19:08.812 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=35, tag_tid=5, entry_tid=8379' 2024-09-11T06:19:08.812 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:19:08.812 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:19:08.813 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=5 2024-09-11T06:19:08.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=33, tag_tid=6, entry_tid=1' 2024-09-11T06:19:08.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:19:08.815 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:19:08.815 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:19:08.815 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=35, tag_tid=5, entry_tid=8379' 2024-09-11T06:19:08.815 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:19:08.815 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:19:08.816 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=8379 2024-09-11T06:19:08.817 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 6 = 5 -a 1 -le 8379 2024-09-11T06:19:08.817 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:19:08.817 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:19:08.817 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:19:09.018 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:19:09.019 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:19:09.019 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:19:09.019 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:19:09.019 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:19:09.019 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:19:09.019 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:19:09.019 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:19:09.019 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:19:09.019 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:19:09.019 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:19:09.020 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:19:09.020 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:19:09.020 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:19:09.020 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:19:09.020 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:19:09.020 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:19:09.022 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:19:09.022 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:19:09.023 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:19:09.023 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:19:09.023 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:19:09.023 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:19:09.023 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:19:09.023 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:19:09.023 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:19:09.023 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:19:09.023 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:19:09.023 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:19:09.023 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:19:09.023 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:19:09.023 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:19:09.024 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:19:09.024 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:19:09.024 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:19:09.024 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:19:09.024 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:19:09.024 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:19:09.024 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:19:09.024 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:19:09.024 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:19:09.024 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:19:09.024 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:19:09.024 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:19:09.024 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:19:09.025 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:19:09.025 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:19:09.025 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:19:09.025 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:19:09.051 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:19:09.052 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:19:09.052 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:19:09.052 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:19:09.177 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:19:09.178 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:19:09.178 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:09.178 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:09.178 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:09.178 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:19:09.178 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:09.179 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:09.179 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:09.179 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:19:09.179 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:19:09.179 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:19:09.179 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:19:09.180 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:09.181 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:19:09.181 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:09.232 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 8 2024-09-11T06:19:09.232 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 8 2024-09-11T06:19:09.232 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:19:09.232 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=6, entry_tid=1], [object_number=32, tag_tid=6, entry_tid=0], [object_number=34, tag_tid=5, entry_tid=10506], [object_number=35, tag_tid=5, entry_tid=10503]]], state=connected] 2024-09-11T06:19:09.233 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=33, tag_tid=5, entry_tid=8481], [object_number=32, tag_tid=5, entry_tid=8480], [object_number=35, tag_tid=5, entry_tid=8479], [object_number=34, tag_tid=5, entry_tid=8478]]], state=connected] 2024-09-11T06:19:09.235 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:09.237 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=33, tag_tid=6, entry_tid=1' 2024-09-11T06:19:09.239 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:19:09.239 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:09.239 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:09.239 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:09.239 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:19:09.239 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:09.239 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:09.240 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:09.240 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:19:09.240 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:19:09.240 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:19:09.240 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:19:09.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:09.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:19:09.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:09.285 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 8 2024-09-11T06:19:09.285 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 8 2024-09-11T06:19:09.285 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:19:09.285 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=6, entry_tid=1], [object_number=32, tag_tid=6, entry_tid=0], [object_number=34, tag_tid=5, entry_tid=10506], [object_number=35, tag_tid=5, entry_tid=10503]]], state=connected] 2024-09-11T06:19:09.285 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=33, tag_tid=5, entry_tid=8481], [object_number=32, tag_tid=5, entry_tid=8480], [object_number=35, tag_tid=5, entry_tid=8479], [object_number=34, tag_tid=5, entry_tid=8478]]], state=connected] 2024-09-11T06:19:09.288 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:09.290 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=33, tag_tid=5, entry_tid=8481' 2024-09-11T06:19:09.290 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=33, tag_tid=6, entry_tid=1' -a 'object_number=33, tag_tid=6, entry_tid=1' = 'object_number=33, tag_tid=5, entry_tid=8481' 2024-09-11T06:19:09.290 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=33, tag_tid=5, entry_tid=8481' '!=' 'object_number=35, tag_tid=5, entry_tid=8379' 2024-09-11T06:19:09.290 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:19:09.290 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=33, tag_tid=5, entry_tid=8481' = 'object_number=35, tag_tid=5, entry_tid=8379' 2024-09-11T06:19:09.291 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=33, tag_tid=5, entry_tid=8481' 2024-09-11T06:19:09.291 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=33, tag_tid=6, entry_tid=1' 2024-09-11T06:19:09.291 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:19:09.291 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:19:09.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=6 2024-09-11T06:19:09.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=33, tag_tid=5, entry_tid=8481' 2024-09-11T06:19:09.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:19:09.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:19:09.293 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=5 2024-09-11T06:19:09.294 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=33, tag_tid=6, entry_tid=1' 2024-09-11T06:19:09.294 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:19:09.294 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:19:09.295 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:19:09.296 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=33, tag_tid=5, entry_tid=8481' 2024-09-11T06:19:09.296 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:19:09.296 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:19:09.297 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=8481 2024-09-11T06:19:09.297 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 6 = 5 -a 1 -le 8481 2024-09-11T06:19:09.297 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:19:09.297 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:19:09.297 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:19:09.499 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:19:09.499 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:19:09.499 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:19:09.499 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:19:09.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:19:09.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:19:09.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:19:09.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:19:09.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:19:09.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:19:09.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:19:09.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:19:09.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:19:09.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:19:09.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:19:09.501 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:19:09.501 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:19:09.504 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:19:09.504 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:19:09.504 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:19:09.504 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:19:09.504 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:19:09.504 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:19:09.504 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:19:09.504 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:19:09.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:19:09.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:19:09.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:19:09.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:19:09.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:19:09.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:19:09.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:19:09.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:19:09.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:19:09.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:19:09.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:19:09.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:19:09.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:19:09.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:19:09.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:19:09.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:19:09.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:19:09.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:19:09.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:19:09.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:19:09.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:19:09.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:19:09.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:19:09.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:19:09.530 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:19:09.531 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:19:09.531 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:19:09.531 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:19:09.661 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:19:09.663 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:19:09.663 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:09.663 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:09.663 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:09.663 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:19:09.663 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:09.663 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:09.663 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:09.663 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:19:09.664 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:19:09.664 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:19:09.664 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:19:09.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:09.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:19:09.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:09.712 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 8 2024-09-11T06:19:09.712 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 8 2024-09-11T06:19:09.712 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:19:09.712 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=6, entry_tid=1], [object_number=32, tag_tid=6, entry_tid=0], [object_number=34, tag_tid=5, entry_tid=10506], [object_number=35, tag_tid=5, entry_tid=10503]]], state=connected] 2024-09-11T06:19:09.712 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=32, tag_tid=5, entry_tid=8576], [object_number=35, tag_tid=5, entry_tid=8575], [object_number=34, tag_tid=5, entry_tid=8574], [object_number=33, tag_tid=5, entry_tid=8573]]], state=connected] 2024-09-11T06:19:09.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:09.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=33, tag_tid=6, entry_tid=1' 2024-09-11T06:19:09.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:19:09.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:09.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:09.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:09.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:19:09.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:09.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:09.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:09.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:19:09.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:19:09.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:19:09.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:19:09.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:09.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:19:09.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:09.764 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 8 2024-09-11T06:19:09.765 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 8 2024-09-11T06:19:09.765 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:19:09.765 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=6, entry_tid=1], [object_number=32, tag_tid=6, entry_tid=0], [object_number=34, tag_tid=5, entry_tid=10506], [object_number=35, tag_tid=5, entry_tid=10503]]], state=connected] 2024-09-11T06:19:09.765 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=32, tag_tid=5, entry_tid=8576], [object_number=35, tag_tid=5, entry_tid=8575], [object_number=34, tag_tid=5, entry_tid=8574], [object_number=33, tag_tid=5, entry_tid=8573]]], state=connected] 2024-09-11T06:19:09.768 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:09.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=32, tag_tid=5, entry_tid=8576' 2024-09-11T06:19:09.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=33, tag_tid=6, entry_tid=1' -a 'object_number=33, tag_tid=6, entry_tid=1' = 'object_number=32, tag_tid=5, entry_tid=8576' 2024-09-11T06:19:09.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=32, tag_tid=5, entry_tid=8576' '!=' 'object_number=33, tag_tid=5, entry_tid=8481' 2024-09-11T06:19:09.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:19:09.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=32, tag_tid=5, entry_tid=8576' = 'object_number=33, tag_tid=5, entry_tid=8481' 2024-09-11T06:19:09.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=32, tag_tid=5, entry_tid=8576' 2024-09-11T06:19:09.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=33, tag_tid=6, entry_tid=1' 2024-09-11T06:19:09.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:19:09.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:19:09.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=6 2024-09-11T06:19:09.772 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=32, tag_tid=5, entry_tid=8576' 2024-09-11T06:19:09.772 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:19:09.772 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:19:09.773 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=5 2024-09-11T06:19:09.774 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=33, tag_tid=6, entry_tid=1' 2024-09-11T06:19:09.774 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:19:09.774 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:19:09.774 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:19:09.775 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=32, tag_tid=5, entry_tid=8576' 2024-09-11T06:19:09.775 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:19:09.775 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:19:09.777 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=8576 2024-09-11T06:19:09.777 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 6 = 5 -a 1 -le 8576 2024-09-11T06:19:09.777 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:19:09.777 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:19:09.777 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:19:09.979 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:19:09.979 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:19:09.979 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:19:09.979 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:19:09.979 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:19:09.979 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:19:09.980 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:19:09.980 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:19:09.980 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:19:09.980 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:19:09.980 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:19:09.980 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:19:09.980 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:19:09.980 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:19:09.980 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:19:09.981 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:19:09.981 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:19:09.982 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:19:09.982 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:19:09.982 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:19:09.983 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:19:09.983 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:19:09.983 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:19:09.983 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:19:09.983 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:19:09.983 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:19:09.983 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:19:09.983 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:19:09.983 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:19:09.983 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:19:09.983 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:19:09.983 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:19:09.983 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:19:09.985 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:19:09.985 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:19:09.985 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:19:09.985 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:19:09.985 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:19:09.985 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:19:09.985 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:19:09.986 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:19:09.986 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:19:09.986 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:19:09.986 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:19:09.986 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:19:09.986 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:19:09.986 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:19:09.986 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:19:09.986 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:19:10.011 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:19:10.012 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:19:10.012 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:19:10.012 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:19:10.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:19:10.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:19:10.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:10.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:10.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:10.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:19:10.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:10.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:10.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:10.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:19:10.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:19:10.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:19:10.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:19:10.139 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:10.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:19:10.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:10.187 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 8 2024-09-11T06:19:10.187 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 8 2024-09-11T06:19:10.187 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:19:10.187 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=6, entry_tid=1], [object_number=32, tag_tid=6, entry_tid=0], [object_number=34, tag_tid=5, entry_tid=10506], [object_number=35, tag_tid=5, entry_tid=10503]]], state=connected] 2024-09-11T06:19:10.188 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=33, tag_tid=5, entry_tid=8673], [object_number=32, tag_tid=5, entry_tid=8672], [object_number=35, tag_tid=5, entry_tid=8671], [object_number=34, tag_tid=5, entry_tid=8670]]], state=connected] 2024-09-11T06:19:10.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:10.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=33, tag_tid=6, entry_tid=1' 2024-09-11T06:19:10.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:19:10.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:10.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:10.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:10.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:19:10.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:10.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:10.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:10.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:19:10.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:19:10.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:19:10.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:19:10.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:10.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:19:10.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:10.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 8 2024-09-11T06:19:10.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 8 2024-09-11T06:19:10.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:19:10.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=6, entry_tid=1], [object_number=32, tag_tid=6, entry_tid=0], [object_number=34, tag_tid=5, entry_tid=10506], [object_number=35, tag_tid=5, entry_tid=10503]]], state=connected] 2024-09-11T06:19:10.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=33, tag_tid=5, entry_tid=8673], [object_number=32, tag_tid=5, entry_tid=8672], [object_number=35, tag_tid=5, entry_tid=8671], [object_number=34, tag_tid=5, entry_tid=8670]]], state=connected] 2024-09-11T06:19:10.244 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:10.246 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=33, tag_tid=5, entry_tid=8673' 2024-09-11T06:19:10.246 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=33, tag_tid=6, entry_tid=1' -a 'object_number=33, tag_tid=6, entry_tid=1' = 'object_number=33, tag_tid=5, entry_tid=8673' 2024-09-11T06:19:10.246 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=33, tag_tid=5, entry_tid=8673' '!=' 'object_number=32, tag_tid=5, entry_tid=8576' 2024-09-11T06:19:10.246 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:19:10.246 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=33, tag_tid=5, entry_tid=8673' = 'object_number=32, tag_tid=5, entry_tid=8576' 2024-09-11T06:19:10.246 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=33, tag_tid=5, entry_tid=8673' 2024-09-11T06:19:10.246 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=33, tag_tid=6, entry_tid=1' 2024-09-11T06:19:10.246 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:19:10.246 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:19:10.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=6 2024-09-11T06:19:10.248 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=33, tag_tid=5, entry_tid=8673' 2024-09-11T06:19:10.248 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:19:10.248 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:19:10.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=5 2024-09-11T06:19:10.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=33, tag_tid=6, entry_tid=1' 2024-09-11T06:19:10.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:19:10.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:19:10.250 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:19:10.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=33, tag_tid=5, entry_tid=8673' 2024-09-11T06:19:10.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:19:10.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:19:10.252 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=8673 2024-09-11T06:19:10.252 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 6 = 5 -a 1 -le 8673 2024-09-11T06:19:10.252 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:19:10.252 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:19:10.252 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:19:10.454 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:19:10.454 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:19:10.454 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:19:10.454 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:19:10.454 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:19:10.454 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:19:10.455 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:19:10.455 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:19:10.455 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:19:10.455 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:19:10.455 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:19:10.455 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:19:10.455 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:19:10.455 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:19:10.455 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:19:10.456 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:19:10.456 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:19:10.459 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:19:10.459 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:19:10.459 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:19:10.459 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:19:10.459 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:19:10.459 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:19:10.459 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:19:10.459 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:19:10.459 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:19:10.460 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:19:10.460 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:19:10.460 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:19:10.460 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:19:10.460 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:19:10.460 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:19:10.460 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:19:10.460 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:19:10.460 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:19:10.460 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:19:10.460 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:19:10.460 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:19:10.461 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:19:10.461 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:19:10.461 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:19:10.461 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:19:10.461 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:19:10.461 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:19:10.461 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:19:10.461 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:19:10.461 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:19:10.461 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:19:10.461 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:19:10.484 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:19:10.485 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:19:10.485 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:19:10.485 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:19:10.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:19:10.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:19:10.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:10.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:10.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:10.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:19:10.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:10.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:10.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:10.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:19:10.623 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:19:10.623 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:19:10.623 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:19:10.624 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:10.625 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:19:10.625 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:10.670 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 8 2024-09-11T06:19:10.671 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 8 2024-09-11T06:19:10.671 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:19:10.671 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=6, entry_tid=1], [object_number=32, tag_tid=6, entry_tid=0], [object_number=34, tag_tid=5, entry_tid=10506], [object_number=35, tag_tid=5, entry_tid=10503]]], state=connected] 2024-09-11T06:19:10.671 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=33, tag_tid=5, entry_tid=8769], [object_number=32, tag_tid=5, entry_tid=8768], [object_number=35, tag_tid=5, entry_tid=8767], [object_number=34, tag_tid=5, entry_tid=8766]]], state=connected] 2024-09-11T06:19:10.674 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:10.675 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=33, tag_tid=6, entry_tid=1' 2024-09-11T06:19:10.677 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:19:10.677 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:10.677 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:10.677 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:10.677 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:19:10.677 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:10.677 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:10.677 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:10.677 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:19:10.677 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:19:10.677 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:19:10.677 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:19:10.678 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:10.678 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:19:10.678 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:10.941 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 8 2024-09-11T06:19:10.942 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 8 2024-09-11T06:19:10.942 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:19:10.942 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=6, entry_tid=1], [object_number=32, tag_tid=6, entry_tid=0], [object_number=34, tag_tid=5, entry_tid=10506], [object_number=35, tag_tid=5, entry_tid=10503]]], state=connected] 2024-09-11T06:19:10.942 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=33, tag_tid=5, entry_tid=8769], [object_number=32, tag_tid=5, entry_tid=8768], [object_number=35, tag_tid=5, entry_tid=8767], [object_number=34, tag_tid=5, entry_tid=8766]]], state=connected] 2024-09-11T06:19:10.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:10.947 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=33, tag_tid=5, entry_tid=8769' 2024-09-11T06:19:10.947 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=33, tag_tid=6, entry_tid=1' -a 'object_number=33, tag_tid=6, entry_tid=1' = 'object_number=33, tag_tid=5, entry_tid=8769' 2024-09-11T06:19:10.948 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=33, tag_tid=5, entry_tid=8769' '!=' 'object_number=33, tag_tid=5, entry_tid=8673' 2024-09-11T06:19:10.948 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:19:10.948 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=33, tag_tid=5, entry_tid=8769' = 'object_number=33, tag_tid=5, entry_tid=8673' 2024-09-11T06:19:10.948 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=33, tag_tid=5, entry_tid=8769' 2024-09-11T06:19:10.948 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=33, tag_tid=6, entry_tid=1' 2024-09-11T06:19:10.949 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:19:10.949 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:19:10.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=6 2024-09-11T06:19:10.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=33, tag_tid=5, entry_tid=8769' 2024-09-11T06:19:10.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:19:10.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:19:10.952 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=5 2024-09-11T06:19:10.952 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=33, tag_tid=6, entry_tid=1' 2024-09-11T06:19:10.952 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:19:10.953 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:19:10.953 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:19:10.954 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=33, tag_tid=5, entry_tid=8769' 2024-09-11T06:19:10.954 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:19:10.954 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:19:10.956 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=8769 2024-09-11T06:19:10.956 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 6 = 5 -a 1 -le 8769 2024-09-11T06:19:10.956 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:19:10.956 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:19:10.956 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:19:11.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:19:11.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:19:11.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:19:11.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:19:11.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:19:11.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:19:11.159 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:19:11.159 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:19:11.159 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:19:11.159 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:19:11.159 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:19:11.159 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:19:11.159 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:19:11.159 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:19:11.159 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:19:11.160 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:19:11.160 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:19:11.162 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:19:11.162 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:19:11.162 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:19:11.162 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:19:11.163 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:19:11.163 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:19:11.163 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:19:11.163 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:19:11.163 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:19:11.163 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:19:11.163 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:19:11.163 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:19:11.163 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:19:11.163 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:19:11.163 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:19:11.163 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:19:11.163 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:19:11.163 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:19:11.164 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:19:11.164 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:19:11.164 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:19:11.164 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:19:11.164 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:19:11.164 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:19:11.164 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:19:11.164 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:19:11.164 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:19:11.164 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:19:11.164 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:19:11.164 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:19:11.164 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:19:11.164 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:19:11.190 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:19:11.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:19:11.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:19:11.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:19:11.318 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:19:11.319 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:19:11.319 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:11.319 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:11.319 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:11.319 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:19:11.319 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:11.319 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:11.320 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:11.320 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:19:11.320 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:19:11.320 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:19:11.320 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:19:11.321 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:11.321 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:19:11.322 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:11.385 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 8 2024-09-11T06:19:11.386 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 8 2024-09-11T06:19:11.386 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:19:11.386 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=6, entry_tid=1], [object_number=32, tag_tid=6, entry_tid=0], [object_number=34, tag_tid=5, entry_tid=10506], [object_number=35, tag_tid=5, entry_tid=10503]]], state=connected] 2024-09-11T06:19:11.386 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=34, tag_tid=5, entry_tid=8902], [object_number=33, tag_tid=5, entry_tid=8901], [object_number=32, tag_tid=5, entry_tid=8900], [object_number=35, tag_tid=5, entry_tid=8899]]], state=connected] 2024-09-11T06:19:11.389 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:11.390 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=33, tag_tid=6, entry_tid=1' 2024-09-11T06:19:11.391 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:19:11.392 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:11.392 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:11.392 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:11.392 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:19:11.392 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:11.392 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:11.392 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:11.392 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:19:11.392 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:19:11.392 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:19:11.392 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:19:11.393 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:11.393 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:19:11.393 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:11.438 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 8 2024-09-11T06:19:11.438 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 8 2024-09-11T06:19:11.438 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:19:11.439 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=6, entry_tid=1], [object_number=32, tag_tid=6, entry_tid=0], [object_number=34, tag_tid=5, entry_tid=10506], [object_number=35, tag_tid=5, entry_tid=10503]]], state=connected] 2024-09-11T06:19:11.439 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=34, tag_tid=5, entry_tid=8902], [object_number=33, tag_tid=5, entry_tid=8901], [object_number=32, tag_tid=5, entry_tid=8900], [object_number=35, tag_tid=5, entry_tid=8899]]], state=connected] 2024-09-11T06:19:11.442 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:11.444 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=34, tag_tid=5, entry_tid=8902' 2024-09-11T06:19:11.444 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=33, tag_tid=6, entry_tid=1' -a 'object_number=33, tag_tid=6, entry_tid=1' = 'object_number=34, tag_tid=5, entry_tid=8902' 2024-09-11T06:19:11.444 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=34, tag_tid=5, entry_tid=8902' '!=' 'object_number=33, tag_tid=5, entry_tid=8769' 2024-09-11T06:19:11.444 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:19:11.445 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=34, tag_tid=5, entry_tid=8902' = 'object_number=33, tag_tid=5, entry_tid=8769' 2024-09-11T06:19:11.445 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=34, tag_tid=5, entry_tid=8902' 2024-09-11T06:19:11.445 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=33, tag_tid=6, entry_tid=1' 2024-09-11T06:19:11.445 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:19:11.445 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:19:11.446 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=6 2024-09-11T06:19:11.446 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=34, tag_tid=5, entry_tid=8902' 2024-09-11T06:19:11.447 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:19:11.447 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:19:11.447 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=5 2024-09-11T06:19:11.447 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=33, tag_tid=6, entry_tid=1' 2024-09-11T06:19:11.448 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:19:11.448 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:19:11.448 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:19:11.449 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=34, tag_tid=5, entry_tid=8902' 2024-09-11T06:19:11.449 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:19:11.449 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:19:11.450 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=8902 2024-09-11T06:19:11.450 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 6 = 5 -a 1 -le 8902 2024-09-11T06:19:11.450 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:19:11.451 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:19:11.451 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:19:11.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:19:11.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:19:11.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:19:11.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:19:11.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:19:11.654 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:19:11.654 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:19:11.654 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:19:11.654 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:19:11.654 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:19:11.654 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:19:11.654 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:19:11.655 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:19:11.655 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:19:11.655 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:19:11.659 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:19:11.659 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:19:11.660 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:19:11.660 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:19:11.660 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:19:11.660 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:19:11.660 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:19:11.661 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:19:11.661 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:19:11.661 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:19:11.661 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:19:11.661 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:19:11.661 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:19:11.661 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:19:11.662 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:19:11.662 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:19:11.662 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:19:11.662 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:19:11.662 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:19:11.662 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:19:11.662 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:19:11.663 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:19:11.663 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:19:11.663 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:19:11.663 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:19:11.663 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:19:11.663 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:19:11.664 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:19:11.664 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:19:11.664 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:19:11.664 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:19:11.664 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:19:11.664 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:19:11.664 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:19:11.683 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:19:11.683 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:19:11.683 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:19:11.683 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:19:11.810 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:19:11.813 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:19:11.813 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:11.813 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:11.813 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:11.813 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:19:11.813 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:11.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:11.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:11.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:19:11.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:19:11.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:19:11.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:19:11.815 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:11.815 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:19:11.816 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:11.859 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 8 2024-09-11T06:19:11.859 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 8 2024-09-11T06:19:11.859 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:19:11.860 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=6, entry_tid=1], [object_number=32, tag_tid=6, entry_tid=0], [object_number=34, tag_tid=5, entry_tid=10506], [object_number=35, tag_tid=5, entry_tid=10503]]], state=connected] 2024-09-11T06:19:11.860 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=32, tag_tid=5, entry_tid=9000], [object_number=35, tag_tid=5, entry_tid=8999], [object_number=34, tag_tid=5, entry_tid=8998], [object_number=33, tag_tid=5, entry_tid=8997]]], state=connected] 2024-09-11T06:19:11.863 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:11.864 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=33, tag_tid=6, entry_tid=1' 2024-09-11T06:19:11.865 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:19:11.865 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:11.865 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:11.865 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:11.865 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:19:11.866 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:11.866 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:11.866 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:11.866 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:19:11.866 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:19:11.866 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:19:11.866 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:19:11.867 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:11.867 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:19:11.867 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:11.912 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 8 2024-09-11T06:19:11.912 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 8 2024-09-11T06:19:11.912 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:19:11.912 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=6, entry_tid=1], [object_number=32, tag_tid=6, entry_tid=0], [object_number=34, tag_tid=5, entry_tid=10506], [object_number=35, tag_tid=5, entry_tid=10503]]], state=connected] 2024-09-11T06:19:11.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=32, tag_tid=5, entry_tid=9000], [object_number=35, tag_tid=5, entry_tid=8999], [object_number=34, tag_tid=5, entry_tid=8998], [object_number=33, tag_tid=5, entry_tid=8997]]], state=connected] 2024-09-11T06:19:11.915 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=32, tag_tid=5, entry_tid=9000' 2024-09-11T06:19:11.918 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=33, tag_tid=6, entry_tid=1' -a 'object_number=33, tag_tid=6, entry_tid=1' = 'object_number=32, tag_tid=5, entry_tid=9000' 2024-09-11T06:19:11.918 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=32, tag_tid=5, entry_tid=9000' '!=' 'object_number=34, tag_tid=5, entry_tid=8902' 2024-09-11T06:19:11.918 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:19:11.918 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=32, tag_tid=5, entry_tid=9000' = 'object_number=34, tag_tid=5, entry_tid=8902' 2024-09-11T06:19:11.918 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=32, tag_tid=5, entry_tid=9000' 2024-09-11T06:19:11.918 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=33, tag_tid=6, entry_tid=1' 2024-09-11T06:19:11.918 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:19:11.918 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:19:11.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=6 2024-09-11T06:19:11.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=32, tag_tid=5, entry_tid=9000' 2024-09-11T06:19:11.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:19:11.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:19:11.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=5 2024-09-11T06:19:11.921 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=33, tag_tid=6, entry_tid=1' 2024-09-11T06:19:11.921 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:19:11.921 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:19:11.922 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:19:11.922 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=32, tag_tid=5, entry_tid=9000' 2024-09-11T06:19:11.922 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:19:11.923 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:19:11.924 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=9000 2024-09-11T06:19:11.924 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 6 = 5 -a 1 -le 9000 2024-09-11T06:19:11.924 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:19:11.924 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:19:11.924 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:19:12.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:19:12.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:19:12.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:19:12.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:19:12.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:19:12.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:19:12.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:19:12.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:19:12.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:19:12.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:19:12.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:19:12.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:19:12.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:19:12.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:19:12.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:19:12.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:19:12.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:19:12.130 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:19:12.130 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:19:12.130 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:19:12.130 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:19:12.130 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:19:12.130 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:19:12.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:19:12.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:19:12.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:19:12.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:19:12.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:19:12.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:19:12.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:19:12.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:19:12.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:19:12.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:19:12.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:19:12.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:19:12.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:19:12.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:19:12.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:19:12.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:19:12.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:19:12.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:19:12.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:19:12.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:19:12.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:19:12.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:19:12.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:19:12.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:19:12.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:19:12.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:19:12.155 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:19:12.156 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:19:12.156 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:19:12.156 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:19:12.286 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:19:12.287 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:19:12.287 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:12.287 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:12.287 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:12.287 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:19:12.287 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:12.287 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:12.288 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:12.288 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:19:12.288 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:19:12.288 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:19:12.288 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:19:12.290 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:12.290 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:19:12.290 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:12.345 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 8 2024-09-11T06:19:12.346 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 8 2024-09-11T06:19:12.346 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:19:12.346 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=6, entry_tid=1], [object_number=32, tag_tid=6, entry_tid=0], [object_number=34, tag_tid=5, entry_tid=10506], [object_number=35, tag_tid=5, entry_tid=10503]]], state=connected] 2024-09-11T06:19:12.346 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=33, tag_tid=5, entry_tid=9093], [object_number=32, tag_tid=5, entry_tid=9092], [object_number=35, tag_tid=5, entry_tid=9091], [object_number=34, tag_tid=5, entry_tid=9090]]], state=connected] 2024-09-11T06:19:12.349 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:12.351 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=33, tag_tid=6, entry_tid=1' 2024-09-11T06:19:12.352 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:19:12.352 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:12.352 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:12.352 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:12.352 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:19:12.352 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:12.352 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:12.352 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:12.352 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:19:12.352 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:19:12.353 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:19:12.353 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:19:12.353 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:12.353 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:19:12.353 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:12.399 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 8 2024-09-11T06:19:12.399 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 8 2024-09-11T06:19:12.399 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:19:12.399 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=6, entry_tid=1], [object_number=32, tag_tid=6, entry_tid=0], [object_number=34, tag_tid=5, entry_tid=10506], [object_number=35, tag_tid=5, entry_tid=10503]]], state=connected] 2024-09-11T06:19:12.399 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=33, tag_tid=5, entry_tid=9093], [object_number=32, tag_tid=5, entry_tid=9092], [object_number=35, tag_tid=5, entry_tid=9091], [object_number=34, tag_tid=5, entry_tid=9090]]], state=connected] 2024-09-11T06:19:12.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:12.404 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=33, tag_tid=5, entry_tid=9093' 2024-09-11T06:19:12.404 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=33, tag_tid=6, entry_tid=1' -a 'object_number=33, tag_tid=6, entry_tid=1' = 'object_number=33, tag_tid=5, entry_tid=9093' 2024-09-11T06:19:12.404 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=33, tag_tid=5, entry_tid=9093' '!=' 'object_number=32, tag_tid=5, entry_tid=9000' 2024-09-11T06:19:12.404 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:19:12.404 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=33, tag_tid=5, entry_tid=9093' = 'object_number=32, tag_tid=5, entry_tid=9000' 2024-09-11T06:19:12.405 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=33, tag_tid=5, entry_tid=9093' 2024-09-11T06:19:12.405 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=33, tag_tid=6, entry_tid=1' 2024-09-11T06:19:12.405 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:19:12.405 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:19:12.406 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=6 2024-09-11T06:19:12.406 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=33, tag_tid=5, entry_tid=9093' 2024-09-11T06:19:12.406 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:19:12.407 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:19:12.407 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=5 2024-09-11T06:19:12.408 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=33, tag_tid=6, entry_tid=1' 2024-09-11T06:19:12.408 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:19:12.408 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:19:12.409 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:19:12.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=33, tag_tid=5, entry_tid=9093' 2024-09-11T06:19:12.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:19:12.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:19:12.411 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=9093 2024-09-11T06:19:12.412 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 6 = 5 -a 1 -le 9093 2024-09-11T06:19:12.412 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:19:12.412 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:19:12.412 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:19:12.613 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:19:12.613 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:19:12.613 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:19:12.613 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:19:12.613 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:19:12.614 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:19:12.614 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:19:12.614 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:19:12.614 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:19:12.614 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:19:12.614 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:19:12.614 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:19:12.614 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:19:12.614 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:19:12.614 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:19:12.615 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:19:12.615 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:19:12.618 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:19:12.618 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:19:12.618 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:19:12.618 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:19:12.618 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:19:12.618 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:19:12.618 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:19:12.618 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:19:12.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:19:12.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:19:12.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:19:12.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:19:12.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:19:12.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:19:12.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:19:12.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:19:12.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:19:12.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:19:12.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:19:12.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:19:12.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:19:12.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:19:12.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:19:12.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:19:12.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:19:12.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:19:12.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:19:12.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:19:12.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:19:12.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:19:12.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:19:12.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:19:12.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:19:12.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:19:12.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:19:12.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:19:12.772 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:19:12.774 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:19:12.774 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:12.774 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:12.774 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:12.774 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:19:12.774 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:12.774 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:12.774 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:12.774 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:19:12.775 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:19:12.775 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:19:12.775 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:19:12.777 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:12.777 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:19:12.777 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:12.822 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 8 2024-09-11T06:19:12.822 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 8 2024-09-11T06:19:12.822 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:19:12.822 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=6, entry_tid=1], [object_number=32, tag_tid=6, entry_tid=0], [object_number=34, tag_tid=5, entry_tid=10506], [object_number=35, tag_tid=5, entry_tid=10503]]], state=connected] 2024-09-11T06:19:12.822 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=32, tag_tid=5, entry_tid=9204], [object_number=35, tag_tid=5, entry_tid=9203], [object_number=34, tag_tid=5, entry_tid=9202], [object_number=33, tag_tid=5, entry_tid=9201]]], state=connected] 2024-09-11T06:19:12.825 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:12.826 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=33, tag_tid=6, entry_tid=1' 2024-09-11T06:19:12.827 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:19:12.828 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:12.828 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:12.828 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:12.828 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:19:12.828 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:12.828 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:12.828 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:12.828 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:19:12.828 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:19:12.828 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:19:12.829 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:19:12.829 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:12.829 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:19:12.830 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:12.874 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 8 2024-09-11T06:19:12.874 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 8 2024-09-11T06:19:12.874 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:19:12.874 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=6, entry_tid=1], [object_number=32, tag_tid=6, entry_tid=0], [object_number=34, tag_tid=5, entry_tid=10506], [object_number=35, tag_tid=5, entry_tid=10503]]], state=connected] 2024-09-11T06:19:12.874 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=32, tag_tid=5, entry_tid=9204], [object_number=35, tag_tid=5, entry_tid=9203], [object_number=34, tag_tid=5, entry_tid=9202], [object_number=33, tag_tid=5, entry_tid=9201]]], state=connected] 2024-09-11T06:19:12.877 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:12.879 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=32, tag_tid=5, entry_tid=9204' 2024-09-11T06:19:12.879 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=33, tag_tid=6, entry_tid=1' -a 'object_number=33, tag_tid=6, entry_tid=1' = 'object_number=32, tag_tid=5, entry_tid=9204' 2024-09-11T06:19:12.880 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=32, tag_tid=5, entry_tid=9204' '!=' 'object_number=33, tag_tid=5, entry_tid=9093' 2024-09-11T06:19:12.880 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:19:12.880 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=32, tag_tid=5, entry_tid=9204' = 'object_number=33, tag_tid=5, entry_tid=9093' 2024-09-11T06:19:12.880 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=32, tag_tid=5, entry_tid=9204' 2024-09-11T06:19:12.880 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=33, tag_tid=6, entry_tid=1' 2024-09-11T06:19:12.880 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:19:12.880 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:19:12.880 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=6 2024-09-11T06:19:12.881 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=32, tag_tid=5, entry_tid=9204' 2024-09-11T06:19:12.881 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:19:12.881 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:19:12.882 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=5 2024-09-11T06:19:12.883 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=33, tag_tid=6, entry_tid=1' 2024-09-11T06:19:12.883 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:19:12.883 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:19:12.884 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:19:12.885 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=32, tag_tid=5, entry_tid=9204' 2024-09-11T06:19:12.885 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:19:12.885 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:19:12.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=9204 2024-09-11T06:19:12.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 6 = 5 -a 1 -le 9204 2024-09-11T06:19:12.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:19:12.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:19:12.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:19:13.088 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:19:13.088 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:19:13.088 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:19:13.088 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:19:13.088 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:19:13.088 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:19:13.088 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:19:13.088 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:19:13.088 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:19:13.088 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:19:13.089 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:19:13.089 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:19:13.089 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:19:13.089 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:19:13.089 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:19:13.089 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:19:13.089 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:19:13.092 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:19:13.092 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:19:13.092 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:19:13.092 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:19:13.092 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:19:13.092 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:19:13.092 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:19:13.092 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:19:13.092 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:19:13.092 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:19:13.093 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:19:13.093 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:19:13.093 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:19:13.093 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:19:13.093 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:19:13.093 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:19:13.093 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:19:13.093 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:19:13.093 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:19:13.093 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:19:13.093 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:19:13.093 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:19:13.093 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:19:13.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:19:13.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:19:13.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:19:13.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:19:13.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:19:13.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:19:13.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:19:13.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:19:13.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:19:13.118 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:19:13.118 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:19:13.118 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:19:13.118 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:19:13.250 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:19:13.252 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:19:13.252 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:13.252 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:13.252 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:13.252 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:19:13.252 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:13.252 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:13.252 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:13.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:19:13.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:19:13.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:19:13.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:19:13.255 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:13.255 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:19:13.255 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:13.306 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 8 2024-09-11T06:19:13.306 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 8 2024-09-11T06:19:13.306 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:19:13.306 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=6, entry_tid=1], [object_number=32, tag_tid=6, entry_tid=0], [object_number=34, tag_tid=5, entry_tid=10506], [object_number=35, tag_tid=5, entry_tid=10503]]], state=connected] 2024-09-11T06:19:13.306 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=32, tag_tid=5, entry_tid=9292], [object_number=35, tag_tid=5, entry_tid=9291], [object_number=34, tag_tid=5, entry_tid=9290], [object_number=33, tag_tid=5, entry_tid=9289]]], state=connected] 2024-09-11T06:19:13.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:13.311 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=33, tag_tid=6, entry_tid=1' 2024-09-11T06:19:13.312 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:19:13.312 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:13.312 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:13.312 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:13.312 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:19:13.313 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:13.313 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:13.313 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:13.313 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:19:13.313 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:19:13.313 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:19:13.313 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:19:13.314 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:13.314 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:19:13.314 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:13.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 8 2024-09-11T06:19:13.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 8 2024-09-11T06:19:13.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:19:13.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=6, entry_tid=1], [object_number=32, tag_tid=6, entry_tid=0], [object_number=34, tag_tid=5, entry_tid=10506], [object_number=35, tag_tid=5, entry_tid=10503]]], state=connected] 2024-09-11T06:19:13.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=32, tag_tid=5, entry_tid=9292], [object_number=35, tag_tid=5, entry_tid=9291], [object_number=34, tag_tid=5, entry_tid=9290], [object_number=33, tag_tid=5, entry_tid=9289]]], state=connected] 2024-09-11T06:19:13.364 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:13.366 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=32, tag_tid=5, entry_tid=9292' 2024-09-11T06:19:13.366 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=33, tag_tid=6, entry_tid=1' -a 'object_number=33, tag_tid=6, entry_tid=1' = 'object_number=32, tag_tid=5, entry_tid=9292' 2024-09-11T06:19:13.366 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=32, tag_tid=5, entry_tid=9292' '!=' 'object_number=32, tag_tid=5, entry_tid=9204' 2024-09-11T06:19:13.366 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:19:13.366 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=32, tag_tid=5, entry_tid=9292' = 'object_number=32, tag_tid=5, entry_tid=9204' 2024-09-11T06:19:13.366 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=32, tag_tid=5, entry_tid=9292' 2024-09-11T06:19:13.367 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=33, tag_tid=6, entry_tid=1' 2024-09-11T06:19:13.367 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:19:13.367 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:19:13.368 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=6 2024-09-11T06:19:13.368 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=32, tag_tid=5, entry_tid=9292' 2024-09-11T06:19:13.368 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:19:13.368 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:19:13.369 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=5 2024-09-11T06:19:13.369 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=33, tag_tid=6, entry_tid=1' 2024-09-11T06:19:13.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:19:13.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:19:13.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:19:13.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=32, tag_tid=5, entry_tid=9292' 2024-09-11T06:19:13.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:19:13.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:19:13.372 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=9292 2024-09-11T06:19:13.372 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 6 = 5 -a 1 -le 9292 2024-09-11T06:19:13.373 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:19:13.373 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:19:13.373 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:19:13.574 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:19:13.574 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:19:13.574 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:19:13.574 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:19:13.574 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:19:13.575 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:19:13.575 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:19:13.575 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:19:13.575 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:19:13.575 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:19:13.575 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:19:13.575 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:19:13.575 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:19:13.575 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:19:13.575 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:19:13.576 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:19:13.576 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:19:13.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:19:13.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:19:13.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:19:13.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:19:13.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:19:13.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:19:13.580 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:19:13.580 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:19:13.580 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:19:13.580 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:19:13.580 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:19:13.580 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:19:13.580 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:19:13.580 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:19:13.581 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:19:13.581 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:19:13.582 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:19:13.582 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:19:13.582 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:19:13.582 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:19:13.582 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:19:13.582 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:19:13.582 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:19:13.582 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:19:13.582 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:19:13.582 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:19:13.583 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:19:13.583 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:19:13.583 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:19:13.583 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:19:13.583 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:19:13.583 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:19:13.605 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:19:13.605 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:19:13.605 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:19:13.605 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:19:13.734 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:19:13.735 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:19:13.735 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:13.735 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:13.735 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:13.735 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:19:13.735 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:13.735 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:13.735 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:13.735 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:19:13.736 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:19:13.736 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:19:13.736 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:19:13.737 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:13.737 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:19:13.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:13.788 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 8 2024-09-11T06:19:13.789 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 8 2024-09-11T06:19:13.789 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:19:13.790 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=6, entry_tid=1], [object_number=32, tag_tid=6, entry_tid=0], [object_number=34, tag_tid=5, entry_tid=10506], [object_number=35, tag_tid=5, entry_tid=10503]]], state=connected] 2024-09-11T06:19:13.790 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=32, tag_tid=5, entry_tid=9380], [object_number=35, tag_tid=5, entry_tid=9379], [object_number=34, tag_tid=5, entry_tid=9378], [object_number=33, tag_tid=5, entry_tid=9377]]], state=connected] 2024-09-11T06:19:13.791 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:13.793 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=33, tag_tid=6, entry_tid=1' 2024-09-11T06:19:13.794 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:19:13.794 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:13.794 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:13.794 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:13.794 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:19:13.795 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:13.795 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:13.795 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:13.795 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:19:13.795 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:19:13.795 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:19:13.795 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:19:13.796 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:13.796 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:19:13.796 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:13.841 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 8 2024-09-11T06:19:13.841 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 8 2024-09-11T06:19:13.841 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:19:13.842 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=6, entry_tid=1], [object_number=32, tag_tid=6, entry_tid=0], [object_number=34, tag_tid=5, entry_tid=10506], [object_number=35, tag_tid=5, entry_tid=10503]]], state=connected] 2024-09-11T06:19:13.842 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=32, tag_tid=5, entry_tid=9380], [object_number=35, tag_tid=5, entry_tid=9379], [object_number=34, tag_tid=5, entry_tid=9378], [object_number=33, tag_tid=5, entry_tid=9377]]], state=connected] 2024-09-11T06:19:13.845 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:13.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=32, tag_tid=5, entry_tid=9380' 2024-09-11T06:19:13.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=33, tag_tid=6, entry_tid=1' -a 'object_number=33, tag_tid=6, entry_tid=1' = 'object_number=32, tag_tid=5, entry_tid=9380' 2024-09-11T06:19:13.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=32, tag_tid=5, entry_tid=9380' '!=' 'object_number=32, tag_tid=5, entry_tid=9292' 2024-09-11T06:19:13.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:19:13.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=32, tag_tid=5, entry_tid=9380' = 'object_number=32, tag_tid=5, entry_tid=9292' 2024-09-11T06:19:13.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=32, tag_tid=5, entry_tid=9380' 2024-09-11T06:19:13.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=33, tag_tid=6, entry_tid=1' 2024-09-11T06:19:13.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:19:13.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:19:13.848 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=6 2024-09-11T06:19:13.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=32, tag_tid=5, entry_tid=9380' 2024-09-11T06:19:13.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:19:13.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:19:13.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=5 2024-09-11T06:19:13.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=33, tag_tid=6, entry_tid=1' 2024-09-11T06:19:13.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:19:13.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:19:13.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:19:13.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=32, tag_tid=5, entry_tid=9380' 2024-09-11T06:19:13.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:19:13.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:19:13.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=9380 2024-09-11T06:19:13.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 6 = 5 -a 1 -le 9380 2024-09-11T06:19:13.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:19:13.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:19:13.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:19:14.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:19:14.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:19:14.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:19:14.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:19:14.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:19:14.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:19:14.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:19:14.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:19:14.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:19:14.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:19:14.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:19:14.056 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:19:14.056 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:19:14.056 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:19:14.056 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:19:14.056 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:19:14.057 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:19:14.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:19:14.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:19:14.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:19:14.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:19:14.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:19:14.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:19:14.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:19:14.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:19:14.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:19:14.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:19:14.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:19:14.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:19:14.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:19:14.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:19:14.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:19:14.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:19:14.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:19:14.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:19:14.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:19:14.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:19:14.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:19:14.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:19:14.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:19:14.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:19:14.063 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:19:14.063 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:19:14.063 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:19:14.063 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:19:14.063 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:19:14.063 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:19:14.063 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:19:14.063 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:19:14.086 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:19:14.086 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:19:14.086 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:19:14.086 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:19:14.216 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:19:14.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:19:14.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:14.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:14.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:14.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:19:14.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:14.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:14.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:14.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:19:14.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:19:14.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:19:14.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:19:14.221 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:14.221 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:19:14.221 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:14.271 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 8 2024-09-11T06:19:14.271 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 8 2024-09-11T06:19:14.271 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:19:14.272 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=6, entry_tid=1], [object_number=32, tag_tid=6, entry_tid=0], [object_number=34, tag_tid=5, entry_tid=10506], [object_number=35, tag_tid=5, entry_tid=10503]]], state=connected] 2024-09-11T06:19:14.272 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=33, tag_tid=5, entry_tid=9465], [object_number=32, tag_tid=5, entry_tid=9464], [object_number=35, tag_tid=5, entry_tid=9463], [object_number=34, tag_tid=5, entry_tid=9462]]], state=connected] 2024-09-11T06:19:14.276 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:14.276 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=33, tag_tid=6, entry_tid=1' 2024-09-11T06:19:14.277 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:19:14.277 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:14.278 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:14.278 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:14.278 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:19:14.278 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:14.278 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:14.278 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:14.278 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:19:14.279 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:19:14.279 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:19:14.279 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:19:14.280 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:14.280 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:19:14.280 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:14.325 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 8 2024-09-11T06:19:14.325 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 8 2024-09-11T06:19:14.325 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:19:14.325 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=6, entry_tid=1], [object_number=32, tag_tid=6, entry_tid=0], [object_number=34, tag_tid=5, entry_tid=10506], [object_number=35, tag_tid=5, entry_tid=10503]]], state=connected] 2024-09-11T06:19:14.325 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=33, tag_tid=5, entry_tid=9465], [object_number=32, tag_tid=5, entry_tid=9464], [object_number=35, tag_tid=5, entry_tid=9463], [object_number=34, tag_tid=5, entry_tid=9462]]], state=connected] 2024-09-11T06:19:14.329 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:14.331 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=33, tag_tid=5, entry_tid=9465' 2024-09-11T06:19:14.331 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=33, tag_tid=6, entry_tid=1' -a 'object_number=33, tag_tid=6, entry_tid=1' = 'object_number=33, tag_tid=5, entry_tid=9465' 2024-09-11T06:19:14.331 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=33, tag_tid=5, entry_tid=9465' '!=' 'object_number=32, tag_tid=5, entry_tid=9380' 2024-09-11T06:19:14.331 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:19:14.331 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=33, tag_tid=5, entry_tid=9465' = 'object_number=32, tag_tid=5, entry_tid=9380' 2024-09-11T06:19:14.331 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=33, tag_tid=5, entry_tid=9465' 2024-09-11T06:19:14.332 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=33, tag_tid=6, entry_tid=1' 2024-09-11T06:19:14.332 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:19:14.332 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:19:14.332 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=6 2024-09-11T06:19:14.333 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=33, tag_tid=5, entry_tid=9465' 2024-09-11T06:19:14.333 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:19:14.333 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:19:14.334 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=5 2024-09-11T06:19:14.334 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=33, tag_tid=6, entry_tid=1' 2024-09-11T06:19:14.334 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:19:14.335 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:19:14.335 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:19:14.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=33, tag_tid=5, entry_tid=9465' 2024-09-11T06:19:14.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:19:14.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:19:14.338 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=9465 2024-09-11T06:19:14.338 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 6 = 5 -a 1 -le 9465 2024-09-11T06:19:14.338 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:19:14.338 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:19:14.338 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:19:14.540 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:19:14.540 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:19:14.540 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:19:14.540 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:19:14.540 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:19:14.540 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:19:14.540 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:19:14.540 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:19:14.540 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:19:14.540 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:19:14.541 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:19:14.541 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:19:14.541 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:19:14.541 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:19:14.541 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:19:14.541 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:19:14.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:19:14.545 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:19:14.545 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:19:14.545 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:19:14.545 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:19:14.545 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:19:14.546 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:19:14.546 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:19:14.546 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:19:14.546 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:19:14.546 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:19:14.546 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:19:14.546 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:19:14.546 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:19:14.546 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:19:14.546 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:19:14.546 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:19:14.546 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:19:14.546 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:19:14.547 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:19:14.547 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:19:14.547 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:19:14.547 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:19:14.547 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:19:14.547 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:19:14.547 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:19:14.547 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:19:14.547 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:19:14.547 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:19:14.547 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:19:14.547 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:19:14.547 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:19:14.547 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:19:14.572 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:19:14.572 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:19:14.572 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:19:14.572 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:19:14.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:19:14.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:19:14.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:14.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:14.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:14.706 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:19:14.706 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:14.706 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:14.706 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:14.706 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:19:14.706 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:19:14.706 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:19:14.707 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:19:14.708 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:14.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:19:14.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:14.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 8 2024-09-11T06:19:14.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 8 2024-09-11T06:19:14.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:19:14.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=6, entry_tid=1], [object_number=32, tag_tid=6, entry_tid=0], [object_number=34, tag_tid=5, entry_tid=10506], [object_number=35, tag_tid=5, entry_tid=10503]]], state=connected] 2024-09-11T06:19:14.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=33, tag_tid=5, entry_tid=9561], [object_number=32, tag_tid=5, entry_tid=9560], [object_number=35, tag_tid=5, entry_tid=9559], [object_number=34, tag_tid=5, entry_tid=9558]]], state=connected] 2024-09-11T06:19:14.758 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:14.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=33, tag_tid=6, entry_tid=1' 2024-09-11T06:19:14.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:19:14.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:14.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:14.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:14.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:19:14.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:14.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:14.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:14.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:19:14.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:19:14.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:19:14.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:19:14.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:14.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:19:14.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:14.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 8 2024-09-11T06:19:14.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 8 2024-09-11T06:19:14.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:19:14.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=6, entry_tid=1], [object_number=32, tag_tid=6, entry_tid=0], [object_number=34, tag_tid=5, entry_tid=10506], [object_number=35, tag_tid=5, entry_tid=10503]]], state=connected] 2024-09-11T06:19:14.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=33, tag_tid=5, entry_tid=9561], [object_number=32, tag_tid=5, entry_tid=9560], [object_number=35, tag_tid=5, entry_tid=9559], [object_number=34, tag_tid=5, entry_tid=9558]]], state=connected] 2024-09-11T06:19:14.812 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:14.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=33, tag_tid=5, entry_tid=9561' 2024-09-11T06:19:14.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=33, tag_tid=6, entry_tid=1' -a 'object_number=33, tag_tid=6, entry_tid=1' = 'object_number=33, tag_tid=5, entry_tid=9561' 2024-09-11T06:19:14.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=33, tag_tid=5, entry_tid=9561' '!=' 'object_number=33, tag_tid=5, entry_tid=9465' 2024-09-11T06:19:14.815 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:19:14.815 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=33, tag_tid=5, entry_tid=9561' = 'object_number=33, tag_tid=5, entry_tid=9465' 2024-09-11T06:19:14.815 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=33, tag_tid=5, entry_tid=9561' 2024-09-11T06:19:14.815 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=33, tag_tid=6, entry_tid=1' 2024-09-11T06:19:14.815 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:19:14.815 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:19:14.815 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=6 2024-09-11T06:19:14.816 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=33, tag_tid=5, entry_tid=9561' 2024-09-11T06:19:14.816 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:19:14.816 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:19:14.817 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=5 2024-09-11T06:19:14.818 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=33, tag_tid=6, entry_tid=1' 2024-09-11T06:19:14.818 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:19:14.818 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:19:14.819 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:19:14.819 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=33, tag_tid=5, entry_tid=9561' 2024-09-11T06:19:14.819 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:19:14.820 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:19:14.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=9561 2024-09-11T06:19:14.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 6 = 5 -a 1 -le 9561 2024-09-11T06:19:14.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:19:14.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:19:14.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:19:15.022 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:19:15.023 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:19:15.023 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:19:15.023 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:19:15.023 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:19:15.023 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:19:15.023 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:19:15.023 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:19:15.023 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:19:15.023 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:19:15.023 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:19:15.024 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:19:15.024 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:19:15.024 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:19:15.024 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:19:15.024 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:19:15.024 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:19:15.028 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:19:15.029 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:19:15.029 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:19:15.029 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:19:15.029 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:19:15.029 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:19:15.029 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:19:15.029 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:19:15.029 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:19:15.029 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:19:15.029 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:19:15.029 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:19:15.029 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:19:15.030 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:19:15.030 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:19:15.030 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:19:15.030 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:19:15.030 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:19:15.030 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:19:15.030 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:19:15.030 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:19:15.030 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:19:15.030 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:19:15.030 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:19:15.030 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:19:15.030 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:19:15.030 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:19:15.031 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:19:15.031 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:19:15.031 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:19:15.031 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:19:15.031 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:19:15.056 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:19:15.056 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:19:15.057 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:19:15.057 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:19:15.187 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:19:15.188 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:19:15.188 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:15.188 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:15.188 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:15.188 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:19:15.188 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:15.188 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:15.188 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:15.188 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:19:15.188 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:19:15.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:19:15.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:19:15.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:15.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:19:15.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:15.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 8 2024-09-11T06:19:15.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 8 2024-09-11T06:19:15.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:19:15.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=6, entry_tid=1], [object_number=32, tag_tid=6, entry_tid=0], [object_number=34, tag_tid=5, entry_tid=10506], [object_number=35, tag_tid=5, entry_tid=10503]]], state=connected] 2024-09-11T06:19:15.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=35, tag_tid=5, entry_tid=9659], [object_number=34, tag_tid=5, entry_tid=9658], [object_number=33, tag_tid=5, entry_tid=9657], [object_number=32, tag_tid=5, entry_tid=9656]]], state=connected] 2024-09-11T06:19:15.246 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:15.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=33, tag_tid=6, entry_tid=1' 2024-09-11T06:19:15.248 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:19:15.248 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:15.248 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:15.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:15.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:19:15.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:15.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:15.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:15.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:19:15.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:19:15.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:19:15.250 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:19:15.250 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:15.250 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:19:15.250 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:15.296 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 8 2024-09-11T06:19:15.296 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 8 2024-09-11T06:19:15.296 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:19:15.296 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=6, entry_tid=1], [object_number=32, tag_tid=6, entry_tid=0], [object_number=34, tag_tid=5, entry_tid=10506], [object_number=35, tag_tid=5, entry_tid=10503]]], state=connected] 2024-09-11T06:19:15.296 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=35, tag_tid=5, entry_tid=9659], [object_number=34, tag_tid=5, entry_tid=9658], [object_number=33, tag_tid=5, entry_tid=9657], [object_number=32, tag_tid=5, entry_tid=9656]]], state=connected] 2024-09-11T06:19:15.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:15.301 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=35, tag_tid=5, entry_tid=9659' 2024-09-11T06:19:15.301 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=33, tag_tid=6, entry_tid=1' -a 'object_number=33, tag_tid=6, entry_tid=1' = 'object_number=35, tag_tid=5, entry_tid=9659' 2024-09-11T06:19:15.301 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=35, tag_tid=5, entry_tid=9659' '!=' 'object_number=33, tag_tid=5, entry_tid=9561' 2024-09-11T06:19:15.301 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:19:15.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=35, tag_tid=5, entry_tid=9659' = 'object_number=33, tag_tid=5, entry_tid=9561' 2024-09-11T06:19:15.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=35, tag_tid=5, entry_tid=9659' 2024-09-11T06:19:15.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=33, tag_tid=6, entry_tid=1' 2024-09-11T06:19:15.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:19:15.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:19:15.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=6 2024-09-11T06:19:15.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=35, tag_tid=5, entry_tid=9659' 2024-09-11T06:19:15.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:19:15.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:19:15.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=5 2024-09-11T06:19:15.305 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=33, tag_tid=6, entry_tid=1' 2024-09-11T06:19:15.305 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:19:15.305 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:19:15.306 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:19:15.307 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=35, tag_tid=5, entry_tid=9659' 2024-09-11T06:19:15.307 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:19:15.307 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:19:15.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=9659 2024-09-11T06:19:15.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 6 = 5 -a 1 -le 9659 2024-09-11T06:19:15.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:19:15.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:19:15.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:19:15.510 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:19:15.510 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:19:15.510 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:19:15.510 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:19:15.510 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:19:15.511 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:19:15.511 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:19:15.511 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:19:15.511 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:19:15.511 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:19:15.511 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:19:15.511 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:19:15.511 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:19:15.511 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:19:15.511 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:19:15.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:19:15.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:19:15.515 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:19:15.515 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:19:15.515 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:19:15.515 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:19:15.515 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:19:15.515 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:19:15.515 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:19:15.515 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:19:15.515 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:19:15.515 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:19:15.515 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:19:15.516 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:19:15.516 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:19:15.516 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:19:15.516 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:19:15.516 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:19:15.517 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:19:15.517 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:19:15.517 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:19:15.517 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:19:15.517 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:19:15.517 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:19:15.517 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:19:15.517 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:19:15.517 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:19:15.517 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:19:15.518 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:19:15.518 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:19:15.518 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:19:15.518 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:19:15.518 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:19:15.518 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:19:15.541 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:19:15.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:19:15.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:19:15.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:19:15.670 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:19:15.672 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:19:15.672 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:15.672 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:15.672 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:15.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:19:15.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:15.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:15.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:15.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:19:15.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:19:15.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:19:15.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:19:15.674 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:15.674 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:19:15.674 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:15.724 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 8 2024-09-11T06:19:15.724 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 8 2024-09-11T06:19:15.724 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:19:15.724 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=6, entry_tid=1], [object_number=32, tag_tid=6, entry_tid=0], [object_number=34, tag_tid=5, entry_tid=10506], [object_number=35, tag_tid=5, entry_tid=10503]]], state=connected] 2024-09-11T06:19:15.724 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=34, tag_tid=5, entry_tid=9778], [object_number=33, tag_tid=5, entry_tid=9777], [object_number=32, tag_tid=5, entry_tid=9776], [object_number=35, tag_tid=5, entry_tid=9775]]], state=connected] 2024-09-11T06:19:15.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:15.729 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=33, tag_tid=6, entry_tid=1' 2024-09-11T06:19:15.730 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:19:15.730 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:15.730 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:15.730 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:15.730 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:19:15.731 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:15.731 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:15.731 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:15.731 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:19:15.731 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:19:15.731 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:19:15.731 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:19:15.732 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:15.732 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:19:15.732 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:15.778 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 8 2024-09-11T06:19:15.778 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 8 2024-09-11T06:19:15.778 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:19:15.778 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=6, entry_tid=1], [object_number=32, tag_tid=6, entry_tid=0], [object_number=34, tag_tid=5, entry_tid=10506], [object_number=35, tag_tid=5, entry_tid=10503]]], state=connected] 2024-09-11T06:19:15.778 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=34, tag_tid=5, entry_tid=9778], [object_number=33, tag_tid=5, entry_tid=9777], [object_number=32, tag_tid=5, entry_tid=9776], [object_number=35, tag_tid=5, entry_tid=9775]]], state=connected] 2024-09-11T06:19:15.781 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:15.783 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=34, tag_tid=5, entry_tid=9778' 2024-09-11T06:19:15.783 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=33, tag_tid=6, entry_tid=1' -a 'object_number=33, tag_tid=6, entry_tid=1' = 'object_number=34, tag_tid=5, entry_tid=9778' 2024-09-11T06:19:15.783 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=34, tag_tid=5, entry_tid=9778' '!=' 'object_number=35, tag_tid=5, entry_tid=9659' 2024-09-11T06:19:15.783 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:19:15.783 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=34, tag_tid=5, entry_tid=9778' = 'object_number=35, tag_tid=5, entry_tid=9659' 2024-09-11T06:19:15.783 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=34, tag_tid=5, entry_tid=9778' 2024-09-11T06:19:15.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=33, tag_tid=6, entry_tid=1' 2024-09-11T06:19:15.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:19:15.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:19:15.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=6 2024-09-11T06:19:15.785 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=34, tag_tid=5, entry_tid=9778' 2024-09-11T06:19:15.785 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:19:15.785 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:19:15.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=5 2024-09-11T06:19:15.787 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=33, tag_tid=6, entry_tid=1' 2024-09-11T06:19:15.787 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:19:15.787 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:19:15.788 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:19:15.788 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=34, tag_tid=5, entry_tid=9778' 2024-09-11T06:19:15.789 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:19:15.789 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:19:15.790 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=9778 2024-09-11T06:19:15.790 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 6 = 5 -a 1 -le 9778 2024-09-11T06:19:15.790 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:19:15.790 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:19:15.790 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:19:15.992 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:19:15.992 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:19:15.992 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:19:15.992 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:19:15.992 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:19:15.992 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:19:15.992 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:19:15.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:19:15.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:19:15.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:19:15.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:19:15.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:19:15.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:19:15.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:19:15.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:19:15.994 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:19:15.994 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:19:15.997 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:19:15.997 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:19:15.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:19:15.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:19:15.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:19:15.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:19:15.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:19:15.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:19:15.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:19:15.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:19:15.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:19:15.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:19:15.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:19:15.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:19:15.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:19:15.999 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:19:15.999 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:19:15.999 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:19:15.999 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:19:15.999 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:19:15.999 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:19:15.999 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:19:15.999 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:19:15.999 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:19:15.999 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:19:15.999 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:19:15.999 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:19:15.999 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:19:16.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:19:16.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:19:16.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:19:16.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:19:16.024 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:19:16.024 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:19:16.024 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:19:16.024 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:19:16.153 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:19:16.154 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:19:16.154 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:16.154 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:16.155 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:16.155 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:19:16.155 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:16.155 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:16.155 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:16.155 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:19:16.155 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:19:16.155 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:19:16.155 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:19:16.157 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:16.157 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:19:16.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:16.202 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 8 2024-09-11T06:19:16.203 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 8 2024-09-11T06:19:16.203 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:19:16.203 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=6, entry_tid=1], [object_number=32, tag_tid=6, entry_tid=0], [object_number=34, tag_tid=5, entry_tid=10506], [object_number=35, tag_tid=5, entry_tid=10503]]], state=connected] 2024-09-11T06:19:16.203 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=32, tag_tid=5, entry_tid=9892], [object_number=35, tag_tid=5, entry_tid=9891], [object_number=34, tag_tid=5, entry_tid=9890], [object_number=33, tag_tid=5, entry_tid=9889]]], state=connected] 2024-09-11T06:19:16.206 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:16.207 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=33, tag_tid=6, entry_tid=1' 2024-09-11T06:19:16.208 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:19:16.209 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:16.209 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:16.209 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:16.209 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:19:16.209 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:16.209 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:16.209 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:16.209 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:19:16.209 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:19:16.209 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:19:16.209 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:19:16.210 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:16.210 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:19:16.210 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:16.256 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 8 2024-09-11T06:19:16.256 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 8 2024-09-11T06:19:16.256 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:19:16.256 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=6, entry_tid=1], [object_number=32, tag_tid=6, entry_tid=0], [object_number=34, tag_tid=5, entry_tid=10506], [object_number=35, tag_tid=5, entry_tid=10503]]], state=connected] 2024-09-11T06:19:16.256 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=32, tag_tid=5, entry_tid=9892], [object_number=35, tag_tid=5, entry_tid=9891], [object_number=34, tag_tid=5, entry_tid=9890], [object_number=33, tag_tid=5, entry_tid=9889]]], state=connected] 2024-09-11T06:19:16.259 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:16.261 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=32, tag_tid=5, entry_tid=9892' 2024-09-11T06:19:16.261 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=33, tag_tid=6, entry_tid=1' -a 'object_number=33, tag_tid=6, entry_tid=1' = 'object_number=32, tag_tid=5, entry_tid=9892' 2024-09-11T06:19:16.261 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=32, tag_tid=5, entry_tid=9892' '!=' 'object_number=34, tag_tid=5, entry_tid=9778' 2024-09-11T06:19:16.261 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:19:16.261 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=32, tag_tid=5, entry_tid=9892' = 'object_number=34, tag_tid=5, entry_tid=9778' 2024-09-11T06:19:16.261 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=32, tag_tid=5, entry_tid=9892' 2024-09-11T06:19:16.261 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=33, tag_tid=6, entry_tid=1' 2024-09-11T06:19:16.262 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:19:16.262 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:19:16.262 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=6 2024-09-11T06:19:16.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=32, tag_tid=5, entry_tid=9892' 2024-09-11T06:19:16.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:19:16.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:19:16.264 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=5 2024-09-11T06:19:16.265 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=33, tag_tid=6, entry_tid=1' 2024-09-11T06:19:16.265 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:19:16.265 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:19:16.266 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:19:16.266 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=32, tag_tid=5, entry_tid=9892' 2024-09-11T06:19:16.266 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:19:16.266 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:19:16.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=9892 2024-09-11T06:19:16.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 6 = 5 -a 1 -le 9892 2024-09-11T06:19:16.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:19:16.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:19:16.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:19:16.469 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:19:16.469 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:19:16.470 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:19:16.470 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:19:16.470 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:19:16.470 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:19:16.470 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:19:16.470 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:19:16.470 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:19:16.470 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:19:16.470 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:19:16.470 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:19:16.470 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:19:16.470 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:19:16.470 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:19:16.471 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:19:16.471 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:19:16.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:19:16.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:19:16.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:19:16.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:19:16.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:19:16.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:19:16.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:19:16.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:19:16.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:19:16.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:19:16.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:19:16.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:19:16.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:19:16.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:19:16.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:19:16.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:19:16.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:19:16.476 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:19:16.476 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:19:16.476 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:19:16.476 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:19:16.476 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:19:16.476 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:19:16.476 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:19:16.476 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:19:16.476 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:19:16.476 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:19:16.476 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:19:16.476 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:19:16.476 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:19:16.477 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:19:16.477 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:19:16.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:19:16.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:19:16.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:19:16.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:19:16.629 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:19:16.631 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:19:16.631 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:16.631 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:16.631 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:16.631 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:19:16.631 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:16.632 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:16.632 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:16.632 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:19:16.632 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:19:16.632 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:19:16.632 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:19:16.632 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:16.633 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:19:16.633 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:16.678 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 8 2024-09-11T06:19:16.678 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 8 2024-09-11T06:19:16.678 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:19:16.679 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=6, entry_tid=1], [object_number=32, tag_tid=6, entry_tid=0], [object_number=34, tag_tid=5, entry_tid=10506], [object_number=35, tag_tid=5, entry_tid=10503]]], state=connected] 2024-09-11T06:19:16.679 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=35, tag_tid=5, entry_tid=10003], [object_number=34, tag_tid=5, entry_tid=10002], [object_number=33, tag_tid=5, entry_tid=10001], [object_number=32, tag_tid=5, entry_tid=10000]]], state=connected] 2024-09-11T06:19:16.682 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:16.683 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=33, tag_tid=6, entry_tid=1' 2024-09-11T06:19:16.684 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:19:16.684 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:16.684 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:16.684 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:16.685 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:19:16.685 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:16.685 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:16.685 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:16.685 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:19:16.685 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:19:16.685 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:19:16.685 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:19:16.686 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:16.686 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:19:16.686 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:16.730 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 8 2024-09-11T06:19:16.731 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 8 2024-09-11T06:19:16.731 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:19:16.731 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=6, entry_tid=1], [object_number=32, tag_tid=6, entry_tid=0], [object_number=34, tag_tid=5, entry_tid=10506], [object_number=35, tag_tid=5, entry_tid=10503]]], state=connected] 2024-09-11T06:19:16.731 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=35, tag_tid=5, entry_tid=10003], [object_number=34, tag_tid=5, entry_tid=10002], [object_number=33, tag_tid=5, entry_tid=10001], [object_number=32, tag_tid=5, entry_tid=10000]]], state=connected] 2024-09-11T06:19:16.734 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:16.737 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=35, tag_tid=5, entry_tid=10003' 2024-09-11T06:19:16.737 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=33, tag_tid=6, entry_tid=1' -a 'object_number=33, tag_tid=6, entry_tid=1' = 'object_number=35, tag_tid=5, entry_tid=10003' 2024-09-11T06:19:16.737 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=35, tag_tid=5, entry_tid=10003' '!=' 'object_number=32, tag_tid=5, entry_tid=9892' 2024-09-11T06:19:16.737 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:19:16.737 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=35, tag_tid=5, entry_tid=10003' = 'object_number=32, tag_tid=5, entry_tid=9892' 2024-09-11T06:19:16.737 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=35, tag_tid=5, entry_tid=10003' 2024-09-11T06:19:16.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=33, tag_tid=6, entry_tid=1' 2024-09-11T06:19:16.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:19:16.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:19:16.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=6 2024-09-11T06:19:16.739 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=35, tag_tid=5, entry_tid=10003' 2024-09-11T06:19:16.739 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:19:16.739 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:19:16.739 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=5 2024-09-11T06:19:16.740 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=33, tag_tid=6, entry_tid=1' 2024-09-11T06:19:16.740 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:19:16.740 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:19:16.740 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:19:16.741 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=35, tag_tid=5, entry_tid=10003' 2024-09-11T06:19:16.741 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:19:16.741 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:19:16.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=10003 2024-09-11T06:19:16.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 6 = 5 -a 1 -le 10003 2024-09-11T06:19:16.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:19:16.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:19:16.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:19:16.944 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:19:16.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:19:16.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:19:16.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:19:16.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:19:16.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:19:16.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:19:16.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:19:16.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:19:16.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:19:16.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:19:16.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:19:16.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:19:16.946 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:19:16.946 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:19:16.946 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:19:16.946 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:19:16.949 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:19:16.949 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:19:16.949 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:19:16.949 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:19:16.949 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:19:16.949 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:19:16.949 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:19:16.949 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:19:16.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:19:16.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:19:16.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:19:16.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:19:16.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:19:16.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:19:16.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:19:16.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:19:16.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:19:16.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:19:16.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:19:16.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:19:16.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:19:16.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:19:16.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:19:16.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:19:16.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:19:16.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:19:16.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:19:16.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:19:16.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:19:16.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:19:16.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:19:16.952 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:19:16.976 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:19:16.976 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:19:16.976 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:19:16.976 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:19:17.107 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:19:17.108 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:19:17.108 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:17.108 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:17.108 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:17.108 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:19:17.108 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:17.108 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:17.108 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:17.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:19:17.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:19:17.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:19:17.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:19:17.112 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:17.112 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:19:17.112 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:17.162 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 8 2024-09-11T06:19:17.163 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 8 2024-09-11T06:19:17.163 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:19:17.163 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=6, entry_tid=1], [object_number=32, tag_tid=6, entry_tid=0], [object_number=34, tag_tid=5, entry_tid=10506], [object_number=35, tag_tid=5, entry_tid=10503]]], state=connected] 2024-09-11T06:19:17.163 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=32, tag_tid=5, entry_tid=10120], [object_number=35, tag_tid=5, entry_tid=10119], [object_number=34, tag_tid=5, entry_tid=10118], [object_number=33, tag_tid=5, entry_tid=10117]]], state=connected] 2024-09-11T06:19:17.166 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:17.167 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=33, tag_tid=6, entry_tid=1' 2024-09-11T06:19:17.168 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:19:17.169 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:17.169 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:17.169 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:17.169 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:19:17.169 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:17.169 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:17.169 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:17.169 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:19:17.169 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:19:17.169 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:19:17.169 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:19:17.170 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:17.170 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:19:17.170 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:17.216 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 8 2024-09-11T06:19:17.216 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 8 2024-09-11T06:19:17.216 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:19:17.216 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=6, entry_tid=1], [object_number=32, tag_tid=6, entry_tid=0], [object_number=34, tag_tid=5, entry_tid=10506], [object_number=35, tag_tid=5, entry_tid=10503]]], state=connected] 2024-09-11T06:19:17.216 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=32, tag_tid=5, entry_tid=10120], [object_number=35, tag_tid=5, entry_tid=10119], [object_number=34, tag_tid=5, entry_tid=10118], [object_number=33, tag_tid=5, entry_tid=10117]]], state=connected] 2024-09-11T06:19:17.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:17.221 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=32, tag_tid=5, entry_tid=10120' 2024-09-11T06:19:17.222 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=33, tag_tid=6, entry_tid=1' -a 'object_number=33, tag_tid=6, entry_tid=1' = 'object_number=32, tag_tid=5, entry_tid=10120' 2024-09-11T06:19:17.222 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=32, tag_tid=5, entry_tid=10120' '!=' 'object_number=35, tag_tid=5, entry_tid=10003' 2024-09-11T06:19:17.222 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:19:17.222 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=32, tag_tid=5, entry_tid=10120' = 'object_number=35, tag_tid=5, entry_tid=10003' 2024-09-11T06:19:17.222 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=32, tag_tid=5, entry_tid=10120' 2024-09-11T06:19:17.222 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=33, tag_tid=6, entry_tid=1' 2024-09-11T06:19:17.222 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:19:17.222 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:19:17.223 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=6 2024-09-11T06:19:17.224 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=32, tag_tid=5, entry_tid=10120' 2024-09-11T06:19:17.224 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:19:17.225 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:19:17.225 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=5 2024-09-11T06:19:17.225 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=33, tag_tid=6, entry_tid=1' 2024-09-11T06:19:17.225 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:19:17.225 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:19:17.226 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:19:17.227 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=32, tag_tid=5, entry_tid=10120' 2024-09-11T06:19:17.227 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:19:17.227 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:19:17.228 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=10120 2024-09-11T06:19:17.228 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 6 = 5 -a 1 -le 10120 2024-09-11T06:19:17.228 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:19:17.228 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:19:17.228 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:19:17.430 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:19:17.430 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:19:17.430 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:19:17.430 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:19:17.430 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:19:17.430 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:19:17.430 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:19:17.430 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:19:17.430 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:19:17.431 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:19:17.431 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:19:17.431 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:19:17.431 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:19:17.431 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:19:17.431 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:19:17.431 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:19:17.431 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:19:17.434 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:19:17.435 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:19:17.435 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:19:17.435 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:19:17.435 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:19:17.435 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:19:17.435 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:19:17.435 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:19:17.435 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:19:17.435 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:19:17.435 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:19:17.435 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:19:17.435 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:19:17.435 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:19:17.436 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:19:17.436 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:19:17.436 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:19:17.436 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:19:17.436 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:19:17.436 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:19:17.436 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:19:17.436 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:19:17.436 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:19:17.436 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:19:17.436 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:19:17.436 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:19:17.437 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:19:17.437 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:19:17.437 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:19:17.437 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:19:17.437 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:19:17.437 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:19:17.460 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:19:17.461 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:19:17.461 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:19:17.461 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:19:17.593 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:19:17.594 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:19:17.594 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:17.594 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:17.594 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:17.594 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:19:17.594 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:17.594 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:17.594 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:17.594 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:19:17.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:19:17.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:19:17.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:19:17.597 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:17.597 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:19:17.597 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:17.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 8 2024-09-11T06:19:17.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 8 2024-09-11T06:19:17.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:19:17.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=6, entry_tid=1], [object_number=32, tag_tid=6, entry_tid=0], [object_number=34, tag_tid=5, entry_tid=10506], [object_number=35, tag_tid=5, entry_tid=10503]]], state=connected] 2024-09-11T06:19:17.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=32, tag_tid=5, entry_tid=10240], [object_number=35, tag_tid=5, entry_tid=10239], [object_number=34, tag_tid=5, entry_tid=10238], [object_number=33, tag_tid=5, entry_tid=10237]]], state=connected] 2024-09-11T06:19:17.656 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:17.658 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=33, tag_tid=6, entry_tid=1' 2024-09-11T06:19:17.659 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:19:17.659 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:17.659 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:17.659 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:17.659 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:19:17.659 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:17.659 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:17.659 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:17.659 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:19:17.659 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:19:17.660 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:19:17.660 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:19:17.660 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:17.660 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:19:17.661 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:17.707 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 8 2024-09-11T06:19:17.707 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 8 2024-09-11T06:19:17.707 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:19:17.707 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=6, entry_tid=1], [object_number=32, tag_tid=6, entry_tid=0], [object_number=34, tag_tid=5, entry_tid=10506], [object_number=35, tag_tid=5, entry_tid=10503]]], state=connected] 2024-09-11T06:19:17.707 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=32, tag_tid=5, entry_tid=10240], [object_number=35, tag_tid=5, entry_tid=10239], [object_number=34, tag_tid=5, entry_tid=10238], [object_number=33, tag_tid=5, entry_tid=10237]]], state=connected] 2024-09-11T06:19:17.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:17.712 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=32, tag_tid=5, entry_tid=10240' 2024-09-11T06:19:17.712 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=33, tag_tid=6, entry_tid=1' -a 'object_number=33, tag_tid=6, entry_tid=1' = 'object_number=32, tag_tid=5, entry_tid=10240' 2024-09-11T06:19:17.712 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=32, tag_tid=5, entry_tid=10240' '!=' 'object_number=32, tag_tid=5, entry_tid=10120' 2024-09-11T06:19:17.712 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:19:17.712 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=32, tag_tid=5, entry_tid=10240' = 'object_number=32, tag_tid=5, entry_tid=10120' 2024-09-11T06:19:17.712 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=32, tag_tid=5, entry_tid=10240' 2024-09-11T06:19:17.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=33, tag_tid=6, entry_tid=1' 2024-09-11T06:19:17.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:19:17.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:19:17.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=6 2024-09-11T06:19:17.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=32, tag_tid=5, entry_tid=10240' 2024-09-11T06:19:17.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:19:17.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:19:17.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=5 2024-09-11T06:19:17.716 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=33, tag_tid=6, entry_tid=1' 2024-09-11T06:19:17.716 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:19:17.716 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:19:17.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:19:17.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=32, tag_tid=5, entry_tid=10240' 2024-09-11T06:19:17.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:19:17.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:19:17.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=10240 2024-09-11T06:19:17.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 6 = 5 -a 1 -le 10240 2024-09-11T06:19:17.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:19:17.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:19:17.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:19:17.921 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:19:17.922 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:19:17.922 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:19:17.922 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:19:17.922 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:19:17.922 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:19:17.922 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:19:17.922 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:19:17.922 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:19:17.922 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:19:17.922 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:19:17.923 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:19:17.923 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:19:17.923 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:19:17.923 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:19:17.923 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:19:17.923 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:19:17.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:19:17.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:19:17.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:19:17.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:19:17.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:19:17.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:19:17.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:19:17.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:19:17.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:19:17.928 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:19:17.928 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:19:17.928 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:19:17.928 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:19:17.928 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:19:17.928 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:19:17.928 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:19:17.928 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:19:17.928 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:19:17.928 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:19:17.928 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:19:17.929 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:19:17.929 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:19:17.929 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:19:17.929 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:19:17.929 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:19:17.929 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:19:17.929 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:19:17.929 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:19:17.929 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:19:17.929 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:19:17.929 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:19:17.929 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:19:17.952 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:19:17.952 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:19:17.952 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:19:17.952 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:19:18.081 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:19:18.082 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:19:18.082 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:18.082 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:18.083 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:18.083 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:19:18.083 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:18.083 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:18.083 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:18.083 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:19:18.084 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:19:18.085 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:19:18.085 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:19:18.086 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:18.086 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:19:18.086 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:18.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 8 2024-09-11T06:19:18.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 8 2024-09-11T06:19:18.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:19:18.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=6, entry_tid=1], [object_number=32, tag_tid=6, entry_tid=0], [object_number=34, tag_tid=5, entry_tid=10506], [object_number=35, tag_tid=5, entry_tid=10503]]], state=connected] 2024-09-11T06:19:18.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=35, tag_tid=5, entry_tid=10355], [object_number=34, tag_tid=5, entry_tid=10354], [object_number=33, tag_tid=5, entry_tid=10353], [object_number=32, tag_tid=5, entry_tid=10352]]], state=connected] 2024-09-11T06:19:18.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:18.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=33, tag_tid=6, entry_tid=1' 2024-09-11T06:19:18.143 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:19:18.143 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:18.143 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:18.143 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:18.143 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:19:18.143 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:18.143 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:18.143 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:18.143 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:19:18.143 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:19:18.143 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:19:18.143 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:19:18.144 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:18.144 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:19:18.144 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:18.190 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 8 2024-09-11T06:19:18.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 8 2024-09-11T06:19:18.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:19:18.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=6, entry_tid=1], [object_number=32, tag_tid=6, entry_tid=0], [object_number=34, tag_tid=5, entry_tid=10506], [object_number=35, tag_tid=5, entry_tid=10503]]], state=connected] 2024-09-11T06:19:18.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=35, tag_tid=5, entry_tid=10355], [object_number=34, tag_tid=5, entry_tid=10354], [object_number=33, tag_tid=5, entry_tid=10353], [object_number=32, tag_tid=5, entry_tid=10352]]], state=connected] 2024-09-11T06:19:18.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:18.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=35, tag_tid=5, entry_tid=10355' 2024-09-11T06:19:18.196 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=33, tag_tid=6, entry_tid=1' -a 'object_number=33, tag_tid=6, entry_tid=1' = 'object_number=35, tag_tid=5, entry_tid=10355' 2024-09-11T06:19:18.196 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=35, tag_tid=5, entry_tid=10355' '!=' 'object_number=32, tag_tid=5, entry_tid=10240' 2024-09-11T06:19:18.196 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:19:18.196 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=35, tag_tid=5, entry_tid=10355' = 'object_number=32, tag_tid=5, entry_tid=10240' 2024-09-11T06:19:18.196 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=35, tag_tid=5, entry_tid=10355' 2024-09-11T06:19:18.196 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=33, tag_tid=6, entry_tid=1' 2024-09-11T06:19:18.197 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:19:18.197 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:19:18.197 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=6 2024-09-11T06:19:18.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=35, tag_tid=5, entry_tid=10355' 2024-09-11T06:19:18.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:19:18.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:19:18.199 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=5 2024-09-11T06:19:18.199 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=33, tag_tid=6, entry_tid=1' 2024-09-11T06:19:18.200 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:19:18.200 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:19:18.200 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:19:18.201 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=35, tag_tid=5, entry_tid=10355' 2024-09-11T06:19:18.201 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:19:18.202 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:19:18.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=10355 2024-09-11T06:19:18.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 6 = 5 -a 1 -le 10355 2024-09-11T06:19:18.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:19:18.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:19:18.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:19:18.404 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:19:18.404 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:19:18.405 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:19:18.405 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:19:18.405 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:19:18.405 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:19:18.405 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:19:18.405 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:19:18.405 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:19:18.405 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:19:18.405 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:19:18.405 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:19:18.405 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:19:18.405 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:19:18.405 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:19:18.406 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:19:18.406 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:19:18.409 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:19:18.409 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:19:18.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:19:18.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:19:18.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:19:18.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:19:18.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:19:18.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:19:18.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:19:18.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:19:18.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:19:18.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:19:18.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:19:18.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:19:18.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:19:18.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:19:18.411 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:19:18.411 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:19:18.411 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:19:18.411 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:19:18.411 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:19:18.411 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:19:18.411 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:19:18.411 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:19:18.411 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:19:18.411 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:19:18.411 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:19:18.411 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:19:18.411 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:19:18.411 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:19:18.412 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:19:18.412 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:19:18.435 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:19:18.435 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:19:18.436 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:19:18.436 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:19:18.563 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:19:18.564 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:19:18.564 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:18.564 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:18.564 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:18.564 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:19:18.564 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:18.565 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:18.565 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:18.565 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:19:18.565 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:19:18.565 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:19:18.566 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:19:18.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:18.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:19:18.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:18.614 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 8 2024-09-11T06:19:18.614 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 8 2024-09-11T06:19:18.614 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:19:18.614 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=6, entry_tid=1], [object_number=32, tag_tid=6, entry_tid=0], [object_number=34, tag_tid=5, entry_tid=10506], [object_number=35, tag_tid=5, entry_tid=10503]]], state=connected] 2024-09-11T06:19:18.614 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=34, tag_tid=5, entry_tid=10470], [object_number=33, tag_tid=5, entry_tid=10469], [object_number=32, tag_tid=5, entry_tid=10468], [object_number=35, tag_tid=5, entry_tid=10467]]], state=connected] 2024-09-11T06:19:18.617 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:18.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=33, tag_tid=6, entry_tid=1' 2024-09-11T06:19:18.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:19:18.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:18.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:18.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:18.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:19:18.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:18.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:18.621 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:18.621 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:19:18.621 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:19:18.621 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:19:18.621 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:19:18.621 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:18.621 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:19:18.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:18.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 8 2024-09-11T06:19:18.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 8 2024-09-11T06:19:18.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:19:18.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=6, entry_tid=1], [object_number=32, tag_tid=6, entry_tid=0], [object_number=34, tag_tid=5, entry_tid=10506], [object_number=35, tag_tid=5, entry_tid=10503]]], state=connected] 2024-09-11T06:19:18.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=34, tag_tid=5, entry_tid=10470], [object_number=33, tag_tid=5, entry_tid=10469], [object_number=32, tag_tid=5, entry_tid=10468], [object_number=35, tag_tid=5, entry_tid=10467]]], state=connected] 2024-09-11T06:19:18.669 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:18.671 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=34, tag_tid=5, entry_tid=10470' 2024-09-11T06:19:18.672 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=33, tag_tid=6, entry_tid=1' -a 'object_number=33, tag_tid=6, entry_tid=1' = 'object_number=34, tag_tid=5, entry_tid=10470' 2024-09-11T06:19:18.672 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=34, tag_tid=5, entry_tid=10470' '!=' 'object_number=35, tag_tid=5, entry_tid=10355' 2024-09-11T06:19:18.672 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:19:18.672 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=34, tag_tid=5, entry_tid=10470' = 'object_number=35, tag_tid=5, entry_tid=10355' 2024-09-11T06:19:18.672 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=34, tag_tid=5, entry_tid=10470' 2024-09-11T06:19:18.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=33, tag_tid=6, entry_tid=1' 2024-09-11T06:19:18.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:19:18.674 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:19:18.674 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=6 2024-09-11T06:19:18.674 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=34, tag_tid=5, entry_tid=10470' 2024-09-11T06:19:18.674 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:19:18.674 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:19:18.674 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=5 2024-09-11T06:19:18.675 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=33, tag_tid=6, entry_tid=1' 2024-09-11T06:19:18.675 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:19:18.675 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:19:18.676 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:19:18.677 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=34, tag_tid=5, entry_tid=10470' 2024-09-11T06:19:18.677 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:19:18.677 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:19:18.678 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=10470 2024-09-11T06:19:18.678 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 6 = 5 -a 1 -le 10470 2024-09-11T06:19:18.678 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:19:18.678 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:19:18.678 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:19:18.880 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:19:18.880 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:19:18.880 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:19:18.881 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:19:18.881 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:19:18.881 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:19:18.881 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:19:18.881 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:19:18.881 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:19:18.881 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:19:18.881 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:19:18.881 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:19:18.881 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:19:18.881 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:19:18.882 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:19:18.882 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:19:18.882 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:19:18.885 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:19:18.885 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:19:18.885 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:19:18.885 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:19:18.885 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:19:18.885 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:19:18.885 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:19:18.885 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:19:18.885 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:19:18.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:19:18.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:19:18.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:19:18.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:19:18.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:19:18.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:19:18.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:19:18.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:19:18.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:19:18.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:19:18.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:19:18.887 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:19:18.887 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:19:18.887 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:19:18.887 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:19:18.887 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:19:18.887 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:19:18.887 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:19:18.887 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:19:18.887 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:19:18.887 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:19:18.887 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:19:18.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:19:18.911 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:19:18.911 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:19:18.911 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:19:18.911 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:19:19.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:19:19.043 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:19:19.043 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:19.043 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:19.043 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:19.043 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:19:19.043 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:19.043 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:19.043 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:19.043 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:19:19.044 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:19:19.044 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:19:19.044 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:19:19.047 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:19.047 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:19:19.047 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:19.092 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 8 2024-09-11T06:19:19.092 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 8 2024-09-11T06:19:19.093 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:19:19.093 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=6, entry_tid=1], [object_number=32, tag_tid=6, entry_tid=0], [object_number=34, tag_tid=5, entry_tid=10506], [object_number=35, tag_tid=5, entry_tid=10503]]], state=connected] 2024-09-11T06:19:19.093 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=34, tag_tid=5, entry_tid=10506], [object_number=33, tag_tid=5, entry_tid=10505], [object_number=32, tag_tid=5, entry_tid=10504], [object_number=35, tag_tid=5, entry_tid=10503]]], state=connected] 2024-09-11T06:19:19.096 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:19.097 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=33, tag_tid=6, entry_tid=1' 2024-09-11T06:19:19.100 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:19:19.100 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:19.100 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:19.101 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:19.101 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:19:19.101 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:19.101 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:19.101 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:19.101 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:19:19.102 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:19:19.102 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:19:19.102 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:19:19.102 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:19.102 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:19:19.102 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:19.145 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 8 2024-09-11T06:19:19.146 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 8 2024-09-11T06:19:19.146 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:19:19.146 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=6, entry_tid=1], [object_number=32, tag_tid=6, entry_tid=0], [object_number=34, tag_tid=5, entry_tid=10506], [object_number=35, tag_tid=5, entry_tid=10503]]], state=connected] 2024-09-11T06:19:19.146 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=34, tag_tid=5, entry_tid=10506], [object_number=33, tag_tid=5, entry_tid=10505], [object_number=32, tag_tid=5, entry_tid=10504], [object_number=35, tag_tid=5, entry_tid=10503]]], state=connected] 2024-09-11T06:19:19.149 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:19.151 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=34, tag_tid=5, entry_tid=10506' 2024-09-11T06:19:19.151 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=33, tag_tid=6, entry_tid=1' -a 'object_number=33, tag_tid=6, entry_tid=1' = 'object_number=34, tag_tid=5, entry_tid=10506' 2024-09-11T06:19:19.151 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=34, tag_tid=5, entry_tid=10506' '!=' 'object_number=34, tag_tid=5, entry_tid=10470' 2024-09-11T06:19:19.151 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:19:19.151 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=34, tag_tid=5, entry_tid=10506' = 'object_number=34, tag_tid=5, entry_tid=10470' 2024-09-11T06:19:19.151 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=34, tag_tid=5, entry_tid=10506' 2024-09-11T06:19:19.151 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=33, tag_tid=6, entry_tid=1' 2024-09-11T06:19:19.151 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:19:19.152 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:19:19.152 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=6 2024-09-11T06:19:19.153 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=34, tag_tid=5, entry_tid=10506' 2024-09-11T06:19:19.153 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:19:19.153 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:19:19.154 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=5 2024-09-11T06:19:19.154 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=33, tag_tid=6, entry_tid=1' 2024-09-11T06:19:19.154 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:19:19.155 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:19:19.155 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:19:19.157 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=34, tag_tid=5, entry_tid=10506' 2024-09-11T06:19:19.157 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:19:19.157 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:19:19.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=10506 2024-09-11T06:19:19.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 6 = 5 -a 1 -le 10506 2024-09-11T06:19:19.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:19:19.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:19:19.159 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:19:19.359 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:19:19.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:19:19.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:19:19.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:19:19.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:19:19.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:19:19.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:19:19.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:19:19.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:19:19.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:19:19.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:19:19.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:19:19.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:19:19.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:19:19.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:19:19.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:19:19.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:19:19.364 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:19:19.364 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:19:19.365 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:19:19.365 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:19:19.365 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:19:19.365 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:19:19.365 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:19:19.365 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:19:19.365 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:19:19.365 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:19:19.365 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:19:19.365 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:19:19.366 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:19:19.366 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:19:19.366 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:19:19.366 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:19:19.366 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:19:19.366 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:19:19.366 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:19:19.366 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:19:19.366 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:19:19.366 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:19:19.366 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:19:19.366 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:19:19.366 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:19:19.366 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:19:19.367 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:19:19.367 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:19:19.367 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:19:19.367 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:19:19.367 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:19:19.367 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:19:19.393 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:19:19.394 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:19:19.394 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:19:19.394 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:19:19.511 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:19:19.511 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:19:19.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:19.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:19.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:19.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:19:19.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:19.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:19.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:19.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:19:19.513 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:19:19.513 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:19:19.513 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:19:19.514 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:19.515 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:19.515 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:19:19.573 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 8 2024-09-11T06:19:19.573 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 8 2024-09-11T06:19:19.573 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:19:19.574 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=6, entry_tid=1], [object_number=32, tag_tid=6, entry_tid=0], [object_number=34, tag_tid=5, entry_tid=10506], [object_number=35, tag_tid=5, entry_tid=10503]]], state=connected] 2024-09-11T06:19:19.574 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=34, tag_tid=5, entry_tid=10506], [object_number=33, tag_tid=5, entry_tid=10505], [object_number=32, tag_tid=5, entry_tid=10504], [object_number=35, tag_tid=5, entry_tid=10503]]], state=connected] 2024-09-11T06:19:19.577 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:19.578 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=33, tag_tid=6, entry_tid=1' 2024-09-11T06:19:19.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:19:19.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:19.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:19.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:19.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:19:19.580 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:19.580 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:19.580 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:19.580 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:19:19.580 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:19:19.580 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:19:19.580 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:19:19.580 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:19.581 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:19:19.581 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:19.627 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 8 2024-09-11T06:19:19.627 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 8 2024-09-11T06:19:19.627 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:19:19.627 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=6, entry_tid=1], [object_number=32, tag_tid=6, entry_tid=0], [object_number=34, tag_tid=5, entry_tid=10506], [object_number=35, tag_tid=5, entry_tid=10503]]], state=connected] 2024-09-11T06:19:19.627 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=34, tag_tid=5, entry_tid=10506], [object_number=33, tag_tid=5, entry_tid=10505], [object_number=32, tag_tid=5, entry_tid=10504], [object_number=35, tag_tid=5, entry_tid=10503]]], state=connected] 2024-09-11T06:19:19.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:19.632 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=34, tag_tid=5, entry_tid=10506' 2024-09-11T06:19:19.632 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=33, tag_tid=6, entry_tid=1' -a 'object_number=33, tag_tid=6, entry_tid=1' = 'object_number=34, tag_tid=5, entry_tid=10506' 2024-09-11T06:19:19.632 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=34, tag_tid=5, entry_tid=10506' '!=' 'object_number=34, tag_tid=5, entry_tid=10506' 2024-09-11T06:19:19.633 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:19:19.633 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.4 2024-09-11T06:19:20.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:19:20.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:19:20.035 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:19:20.035 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:19:20.035 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:19:20.035 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:19:20.035 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:19:20.035 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:19:20.035 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:19:20.035 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:19:20.035 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:19:20.035 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:19:20.035 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:19:20.035 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:19:20.036 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:19:20.036 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:19:20.036 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:19:20.037 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:19:20.038 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:19:20.038 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:19:20.038 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:19:20.038 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:19:20.038 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:19:20.038 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:19:20.038 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:19:20.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:19:20.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:19:20.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:19:20.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:19:20.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:19:20.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:19:20.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:19:20.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:19:20.041 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:19:20.041 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:19:20.041 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:19:20.041 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:19:20.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:19:20.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:19:20.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:19:20.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:19:20.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:19:20.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:19:20.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:19:20.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:19:20.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:19:20.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:19:20.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:19:20.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:19:20.065 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:19:20.065 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:19:20.065 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:19:20.065 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:19:20.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:19:20.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:19:20.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:20.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:20.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:20.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:19:20.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:20.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:20.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:20.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:19:20.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:19:20.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:19:20.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:19:20.196 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:20.197 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:19:20.197 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:20.248 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 8 2024-09-11T06:19:20.248 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 8 2024-09-11T06:19:20.248 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:19:20.248 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=6, entry_tid=1], [object_number=32, tag_tid=6, entry_tid=0], [object_number=34, tag_tid=5, entry_tid=10506], [object_number=35, tag_tid=5, entry_tid=10503]]], state=connected] 2024-09-11T06:19:20.248 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=34, tag_tid=5, entry_tid=10506], [object_number=33, tag_tid=5, entry_tid=10505], [object_number=32, tag_tid=5, entry_tid=10504], [object_number=35, tag_tid=5, entry_tid=10503]]], state=connected] 2024-09-11T06:19:20.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:20.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=33, tag_tid=6, entry_tid=1' 2024-09-11T06:19:20.254 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:19:20.254 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:20.254 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:20.254 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:20.255 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:19:20.255 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:20.255 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:20.255 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:20.255 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:19:20.255 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:19:20.255 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:19:20.255 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:19:20.256 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:20.256 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:19:20.256 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:20.301 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 8 2024-09-11T06:19:20.301 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 8 2024-09-11T06:19:20.301 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:19:20.301 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=6, entry_tid=1], [object_number=32, tag_tid=6, entry_tid=0], [object_number=34, tag_tid=5, entry_tid=10506], [object_number=35, tag_tid=5, entry_tid=10503]]], state=connected] 2024-09-11T06:19:20.301 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=34, tag_tid=5, entry_tid=10506], [object_number=33, tag_tid=5, entry_tid=10505], [object_number=32, tag_tid=5, entry_tid=10504], [object_number=35, tag_tid=5, entry_tid=10503]]], state=connected] 2024-09-11T06:19:20.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:20.306 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=34, tag_tid=5, entry_tid=10506' 2024-09-11T06:19:20.306 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=33, tag_tid=6, entry_tid=1' -a 'object_number=33, tag_tid=6, entry_tid=1' = 'object_number=34, tag_tid=5, entry_tid=10506' 2024-09-11T06:19:20.306 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=34, tag_tid=5, entry_tid=10506' '!=' 'object_number=34, tag_tid=5, entry_tid=10506' 2024-09-11T06:19:20.306 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:19:20.306 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.8 2024-09-11T06:19:21.108 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:19:21.108 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:19:21.108 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:19:21.108 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:19:21.108 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:19:21.108 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:19:21.108 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:19:21.108 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:19:21.108 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:19:21.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:19:21.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:19:21.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:19:21.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:19:21.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:19:21.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:19:21.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:19:21.110 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:19:21.112 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:19:21.112 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:19:21.112 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:19:21.112 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:19:21.113 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:19:21.113 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:19:21.113 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:19:21.113 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:19:21.113 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:19:21.113 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:19:21.113 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:19:21.113 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:19:21.113 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:19:21.113 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:19:21.113 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:19:21.113 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:19:21.114 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:19:21.114 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:19:21.115 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:19:21.115 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:19:21.115 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:19:21.115 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:19:21.115 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:19:21.115 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:19:21.115 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:19:21.115 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:19:21.115 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:19:21.115 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:19:21.115 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:19:21.115 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:19:21.115 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:19:21.115 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:19:21.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:19:21.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:19:21.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:19:21.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:19:21.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:19:21.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:19:21.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:21.269 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:21.269 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:21.269 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:19:21.269 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:21.269 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:21.269 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:21.269 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:19:21.269 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:19:21.269 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:19:21.270 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:19:21.271 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:21.272 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:21.272 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:19:21.324 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 8 2024-09-11T06:19:21.324 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 8 2024-09-11T06:19:21.325 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:19:21.325 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=6, entry_tid=1], [object_number=32, tag_tid=6, entry_tid=0], [object_number=34, tag_tid=5, entry_tid=10506], [object_number=35, tag_tid=5, entry_tid=10503]]], state=connected] 2024-09-11T06:19:21.325 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=33, tag_tid=6, entry_tid=1], [object_number=32, tag_tid=6, entry_tid=0], [object_number=34, tag_tid=5, entry_tid=10506], [object_number=35, tag_tid=5, entry_tid=10503]]], state=connected] 2024-09-11T06:19:21.328 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:21.329 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=33, tag_tid=6, entry_tid=1' 2024-09-11T06:19:21.330 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:19:21.330 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:21.330 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:21.330 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:21.331 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:19:21.331 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:19:21.331 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:19:21.331 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:19:21.331 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:19:21.331 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:19:21.331 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:19:21.331 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:19:21.331 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:21.332 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:19:21.332 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:21.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 8 2024-09-11T06:19:21.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 8 2024-09-11T06:19:21.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:19:21.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=33, tag_tid=6, entry_tid=1], [object_number=32, tag_tid=6, entry_tid=0], [object_number=34, tag_tid=5, entry_tid=10506], [object_number=35, tag_tid=5, entry_tid=10503]]], state=connected] 2024-09-11T06:19:21.378 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=33, tag_tid=6, entry_tid=1], [object_number=32, tag_tid=6, entry_tid=0], [object_number=34, tag_tid=5, entry_tid=10506], [object_number=35, tag_tid=5, entry_tid=10503]]], state=connected] 2024-09-11T06:19:21.380 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:19:21.382 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=33, tag_tid=6, entry_tid=1' 2024-09-11T06:19:21.383 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=33, tag_tid=6, entry_tid=1' -a 'object_number=33, tag_tid=6, entry_tid=1' = 'object_number=33, tag_tid=6, entry_tid=1' 2024-09-11T06:19:21.383 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:19:21.383 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_snap_present cluster1 mirror test snap2 2024-09-11T06:19:21.383 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:19:21.383 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:19:21.383 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:19:21.383 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap2 2024-09-11T06:19:21.383 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:19:21.383 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T06:19:21.383 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:19:22.383 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 info mirror/test@snap2 2024-09-11T06:19:22.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:rbd image 'test': 2024-09-11T06:19:22.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: size 512 MiB in 128 objects 2024-09-11T06:19:22.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: order 22 (4 MiB objects) 2024-09-11T06:19:22.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: snapshot_count: 3 2024-09-11T06:19:22.454 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: id: 10a618cfe337 2024-09-11T06:19:22.454 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: block_name_prefix: rbd_data.10a618cfe337 2024-09-11T06:19:22.454 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: format: 2 2024-09-11T06:19:22.454 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: features: layering, exclusive-lock, journaling 2024-09-11T06:19:22.454 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: op_features: 2024-09-11T06:19:22.454 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: flags: 2024-09-11T06:19:22.454 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: create_timestamp: Wed Sep 11 06:16:46 2024 2024-09-11T06:19:22.454 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: access_timestamp: Wed Sep 11 06:18:21 2024 2024-09-11T06:19:22.454 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: modify_timestamp: Wed Sep 11 06:18:48 2024 2024-09-11T06:19:22.454 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: protected: False 2024-09-11T06:19:22.454 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: journal: 10a618cfe337 2024-09-11T06:19:22.454 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring state: enabled 2024-09-11T06:19:22.454 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring mode: journal 2024-09-11T06:19:22.454 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring global id: a0f7d515-874e-4c67-a26b-4958c7ab3083 2024-09-11T06:19:22.455 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring primary: false 2024-09-11T06:19:22.458 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:19:22.458 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n snap1-clean ']' 2024-09-11T06:19:22.459 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ compare_image_snaps mirror test snap1-clean 2024-09-11T06:19:22.459 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:19:22.459 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:19:22.459 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap1-clean 2024-09-11T06:19:22.459 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local ret=0 2024-09-11T06:19:22.459 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local rmt_export=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export 2024-09-11T06:19:22.459 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loc_export=/tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:19:22.459 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:19:22.459 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1-clean /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export 2024-09-11T06:19:23.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T06:19:23.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1-clean /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:19:24.331 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T06:19:24.352 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmp /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:19:24.635 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:19:24.773 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:19:24.773 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ compare_image_snaps mirror test snap2 2024-09-11T06:19:24.773 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:19:24.773 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:19:24.773 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap2 2024-09-11T06:19:24.773 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local ret=0 2024-09-11T06:19:24.773 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local rmt_export=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export 2024-09-11T06:19:24.773 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loc_export=/tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:19:24.773 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:19:24.774 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2 /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export 2024-09-11T06:19:25.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T06:19:25.358 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2 /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:19:26.658 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T06:19:26.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmp /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:19:26.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:19:27.082 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:19:27.082 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ clean_snap_name=snap2-clean 2024-09-11T06:19:27.082 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_snap cluster2 mirror test snap2-clean 2024-09-11T06:19:27.082 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:19:27.082 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:19:27.083 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:19:27.083 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap2-clean 2024-09-11T06:19:27.083 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap2-clean --debug-rbd=20 --debug-journaler=20 2024-09-11T06:19:28.254 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 10` 2024-09-11T06:19:28.255 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ stress_write_image cluster2 mirror test 2024-09-11T06:19:28.255 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:19:28.255 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:19:28.255 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:19:28.255 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T06:19:28.257 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local duration=7 2024-09-11T06:19:28.257 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set +e 2024-09-11T06:19:28.257 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-11T06:19:30.613 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:100 IOs, 58692608 bytes 2024-09-11T06:19:31.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:200 IOs, 86481920 bytes 2024-09-11T06:19:33.893 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:300 IOs, 140903424 bytes 2024-09-11T06:19:35.269 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ error_code=124 2024-09-11T06:19:35.269 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set -e 2024-09-11T06:19:35.269 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T06:19:35.269 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:19:35.269 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-11T06:19:35.269 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:19:35.269 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:19:35.269 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:19:35.269 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state_pattern=up+replaying 2024-09-11T06:19:35.269 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local description_pattern= 2024-09-11T06:19:35.269 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local service_pattern= 2024-09-11T06:19:35.270 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:19:35.270 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:19:36.270 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-11T06:19:36.270 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:19:36.270 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:19:36.270 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:19:36.270 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state_pattern=up+replaying 2024-09-11T06:19:36.270 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local description_pattern= 2024-09-11T06:19:36.270 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local service_pattern= 2024-09-11T06:19:36.271 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local status 2024-09-11T06:19:36.271 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ CEPH_ARGS= 2024-09-11T06:19:36.271 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-11T06:19:36.362 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ status='test: 2024-09-11T06:19:36.362 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: global_id: a0f7d515-874e-4c67-a26b-4958c7ab3083 2024-09-11T06:19:36.362 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: state: up+replaying 2024-09-11T06:19:36.362 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: description: replaying, {"bytes_per_second":3221148.53,"entries_behind_primary":1042,"entries_per_second":203.8,"non_primary_position":{"entry_tid":9465,"object_number":33,"tag_tid":5},"primary_position":{"entry_tid":1,"object_number":33,"tag_tid":6},"seconds_until_synced":5} 2024-09-11T06:19:36.362 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: service: mirror.0 on smithi112 2024-09-11T06:19:36.362 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: last_update: 2024-09-11 06:19:14 2024-09-11T06:19:36.362 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: peer_sites: 2024-09-11T06:19:36.363 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: name: cluster2 2024-09-11T06:19:36.363 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: state: up+stopped 2024-09-11T06:19:36.363 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: description: local image is primary 2024-09-11T06:19:36.363 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: last_update: 2024-09-11 06:19:34' 2024-09-11T06:19:36.363 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ grep '^ state: .*up+replaying' 2024-09-11T06:19:36.363 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: state: up+replaying 2024-09-11T06:19:36.363 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ grep '^ description: .*' 2024-09-11T06:19:36.364 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: description: replaying, {"bytes_per_second":3221148.53,"entries_behind_primary":1042,"entries_per_second":203.8,"non_primary_position":{"entry_tid":9465,"object_number":33,"tag_tid":5},"primary_position":{"entry_tid":1,"object_number":33,"tag_tid":6},"seconds_until_synced":5} 2024-09-11T06:19:36.364 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n '' ']' 2024-09-11T06:19:36.365 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ echo up+replaying 2024-09-11T06:19:36.365 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ grep '^up+' 2024-09-11T06:19:36.366 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:up+replaying 2024-09-11T06:19:36.366 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ grep 'service: *mirror..* on ' 2024-09-11T06:19:36.367 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: service: mirror.0 on smithi112 2024-09-11T06:19:36.367 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local last_update 2024-09-11T06:19:36.367 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-11T06:19:36.369 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_update='2024-09-11 06:19:14' 2024-09-11T06:19:36.369 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-11 06:19:14' 2024-09-11T06:19:36.369 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:19:36.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:19:36.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:19:36.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state_pattern=up+replaying 2024-09-11T06:19:36.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'prev_last_update=2024-09-11 06:19:14' 2024-09-11T06:19:36.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local status 2024-09-11T06:19:36.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ CEPH_ARGS= 2024-09-11T06:19:36.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-11T06:19:36.449 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ status='OKOKOK142554262mirror.0smithi11219.1.1-298-gb53b6d3etrueOK42594265mirror.1smithi11219.1.1-298-gb53b6d3efalseOKtesta0f7d515-874e-4c67-a26b-4958c7ab3083up+replayingreplaying, {"bytes_per_second":3221148.53,"entries_behind_primary":1042,"entries_per_second":203.8,"non_primary_position":{"entry_tid":9465,"object_number":33,"tag_tid":5},"primary_position":{"entry_tid":1,"object_number":33,"tag_tid":6},"seconds_until_synced":5}42554262mirror.0smithi1122024-09-11 06:19:14cluster20bfc6eda-f3c5-424a-8ff7-5c4d3d9fd488up+stoppedlocal image is primary2024-09-11 06:19:34' 2024-09-11T06:19:36.449 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local last_update state 2024-09-11T06:19:36.449 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-11T06:19:36.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_update='2024-09-11 06:19:14' 2024-09-11T06:19:36.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-11T06:19:36.456 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ state=up+replaying 2024-09-11T06:19:36.456 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ echo up+replaying 2024-09-11T06:19:36.457 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ grep up+replaying 2024-09-11T06:19:36.458 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:up+replaying 2024-09-11T06:19:36.458 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:19:36.459 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:19:36.459 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ snap_name=snap3 2024-09-11T06:19:36.459 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_snap cluster2 mirror test snap3 2024-09-11T06:19:36.459 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:19:36.459 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:19:36.459 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:19:36.459 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap3 2024-09-11T06:19:36.459 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap3 --debug-rbd=20 --debug-journaler=20 2024-09-11T06:20:06.348 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-11T06:20:06.348 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:20:06.348 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:20:06.348 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:20:06.348 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-11T06:20:06.348 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:20:06.349 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:20:06.349 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:20:06.349 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=started 2024-09-11T06:20:06.349 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:20:06.349 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:20:06.349 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:20:07.349 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-11T06:20:07.349 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:20:07.349 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:20:07.349 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:20:07.349 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=started 2024-09-11T06:20:07.350 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local status_result 2024-09-11T06:20:07.350 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=stopped 2024-09-11T06:20:07.350 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-11T06:20:07.351 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_instance=cluster1 2024-09-11T06:20:07.351 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:20:07.351 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -i state 2024-09-11T06:20:07.351 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:20:07.351 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance=cluster1 2024-09-11T06:20:07.351 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local loop_instance 2024-09-11T06:20:07.351 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:20:07.351 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sleep 0 2024-09-11T06:20:07.352 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:20:07.352 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ seq 0 1 2024-09-11T06:20:07.354 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:20:07.354 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-11T06:20:07.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1:0 2024-09-11T06:20:07.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:20:07.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:20:07.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:20:07.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:20:07.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=cluster 2024-09-11T06:20:07.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:20:07.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:20:07.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:20:07.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:20:07.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval cluster=cluster1 2024-09-11T06:20:07.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:20:07.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:20:07.356 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:20:07.356 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:20:07.356 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local local_cluster=cluster1:0 2024-09-11T06:20:07.357 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster=cluster1 2024-09-11T06:20:07.357 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance 2024-09-11T06:20:07.357 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:20:07.357 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local val=cluster1:0 2024-09-11T06:20:07.357 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T06:20:07.357 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance_var_name=instance 2024-09-11T06:20:07.357 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:20:07.357 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:20:07.358 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T06:20:07.358 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval local_cluster=cluster1 2024-09-11T06:20:07.358 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ local_cluster=cluster1 2024-09-11T06:20:07.358 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval instance=0 2024-09-11T06:20:07.358 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ instance=0 2024-09-11T06:20:07.358 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:20:07.381 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:07.381 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:07.381 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:07.381 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-11T06:20:07.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ return 0 2024-09-11T06:20:07.498 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T06:20:07.498 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ echo ' "state": "Replaying"' 2024-09-11T06:20:07.498 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ grep -i Replaying 2024-09-11T06:20:07.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: "state": "Replaying" 2024-09-11T06:20:07.501 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_state=started 2024-09-11T06:20:07.502 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test started = started 2024-09-11T06:20:07.502 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:20:07.502 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-11T06:20:07.502 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:20:07.502 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:20:07.502 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:20:07.502 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:20:07.502 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' journal = journal ']' 2024-09-11T06:20:07.502 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-09-11T06:20:07.502 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:20:07.502 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:20:07.502 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:20:07.502 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:20:07.503 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-11T06:20:07.503 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-11T06:20:07.503 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:20:07.503 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:20:07.503 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:20:07.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:20:07.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:20:07.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:20:07.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:20:07.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:20:07.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:20:07.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:20:07.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:20:07.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:20:07.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:20:07.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:20:07.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:20:07.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:20:07.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:20:07.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:20:07.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:20:07.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:20:07.707 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:20:07.707 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:20:07.708 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:20:07.708 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:20:07.708 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:20:07.708 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:20:07.708 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:20:07.708 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:20:07.708 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:20:07.708 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:20:07.708 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:20:07.708 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:20:07.708 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:20:07.708 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:20:07.708 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:20:07.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:20:07.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:20:07.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:20:07.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:20:07.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:20:07.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:20:07.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:20:07.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:20:07.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:20:07.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:20:07.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:20:07.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:20:07.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:20:07.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:20:07.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:20:07.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:20:07.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:20:07.736 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:07.737 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:07.737 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:07.737 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:20:07.863 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:20:07.863 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:20:07.864 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:07.864 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:07.864 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:07.864 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:20:07.864 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:07.864 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:07.864 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:07.864 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:20:07.865 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:20:07.865 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:20:07.865 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:20:07.867 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:07.867 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:20:07.868 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:07.917 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 10 2024-09-11T06:20:07.918 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 11 2024-09-11T06:20:07.918 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:20:07.918 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=10638], [object_number=47, tag_tid=8, entry_tid=10635]]], state=connected] 2024-09-11T06:20:07.918 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=40, tag_tid=8, entry_tid=7164], [object_number=43, tag_tid=8, entry_tid=7163], [object_number=42, tag_tid=8, entry_tid=7162], [object_number=41, tag_tid=8, entry_tid=7161]]], state=connected] 2024-09-11T06:20:07.921 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:07.922 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-09-11T06:20:07.923 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:20:07.923 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:07.923 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:07.923 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:07.923 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:20:07.924 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:07.924 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:07.924 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:07.924 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:20:07.924 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:20:07.924 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:20:07.924 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:20:07.925 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:07.925 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:20:07.925 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:07.970 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 10 2024-09-11T06:20:07.970 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 11 2024-09-11T06:20:07.970 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:20:07.970 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=10638], [object_number=47, tag_tid=8, entry_tid=10635]]], state=connected] 2024-09-11T06:20:07.970 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=40, tag_tid=8, entry_tid=7164], [object_number=43, tag_tid=8, entry_tid=7163], [object_number=42, tag_tid=8, entry_tid=7162], [object_number=41, tag_tid=8, entry_tid=7161]]], state=connected] 2024-09-11T06:20:07.973 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:07.975 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=40, tag_tid=8, entry_tid=7164' 2024-09-11T06:20:07.975 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=40, tag_tid=8, entry_tid=7164' 2024-09-11T06:20:07.975 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=40, tag_tid=8, entry_tid=7164' '!=' '' 2024-09-11T06:20:07.975 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:20:07.976 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=40, tag_tid=8, entry_tid=7164' = '' 2024-09-11T06:20:07.976 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=40, tag_tid=8, entry_tid=7164' 2024-09-11T06:20:07.976 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-09-11T06:20:07.976 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:20:07.976 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:07.976 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=9 2024-09-11T06:20:07.977 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=40, tag_tid=8, entry_tid=7164' 2024-09-11T06:20:07.977 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:20:07.977 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:07.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=8 2024-09-11T06:20:07.979 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-09-11T06:20:07.979 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:20:07.979 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:07.980 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:20:07.980 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=40, tag_tid=8, entry_tid=7164' 2024-09-11T06:20:07.980 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:20:07.981 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:07.982 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=7164 2024-09-11T06:20:07.982 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 9 = 8 -a 1 -le 7164 2024-09-11T06:20:07.982 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:20:07.982 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:20:07.982 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:20:08.184 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:20:08.184 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:20:08.184 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:20:08.184 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:20:08.184 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:20:08.184 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:20:08.184 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:20:08.184 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:20:08.184 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:20:08.184 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:20:08.184 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:20:08.185 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:20:08.185 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:20:08.185 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:20:08.185 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:20:08.185 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:20:08.185 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:20:08.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:20:08.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:20:08.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:20:08.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:20:08.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:20:08.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:20:08.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:20:08.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:20:08.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:20:08.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:20:08.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:20:08.190 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:20:08.190 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:20:08.190 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:20:08.190 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:20:08.190 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:20:08.190 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:20:08.190 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:20:08.190 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:20:08.190 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:20:08.190 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:20:08.190 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:20:08.190 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:20:08.190 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:20:08.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:20:08.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:20:08.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:20:08.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:20:08.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:20:08.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:20:08.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:20:08.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:20:08.213 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:08.214 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:08.214 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:08.214 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:20:08.343 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:20:08.344 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:20:08.345 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:08.345 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:08.345 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:08.345 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:20:08.345 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:08.345 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:08.345 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:08.345 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:20:08.346 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:20:08.346 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:20:08.346 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:20:08.348 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:08.348 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:08.348 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:20:08.399 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 10 2024-09-11T06:20:08.399 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 11 2024-09-11T06:20:08.399 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:20:08.399 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=10638], [object_number=47, tag_tid=8, entry_tid=10635]]], state=connected] 2024-09-11T06:20:08.399 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=43, tag_tid=8, entry_tid=7255], [object_number=42, tag_tid=8, entry_tid=7254], [object_number=41, tag_tid=8, entry_tid=7253], [object_number=40, tag_tid=8, entry_tid=7252]]], state=connected] 2024-09-11T06:20:08.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:08.403 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-09-11T06:20:08.405 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:20:08.405 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:08.405 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:08.405 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:08.405 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:20:08.405 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:08.405 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:08.405 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:08.405 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:20:08.405 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:20:08.405 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:20:08.406 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:20:08.406 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:08.406 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:20:08.406 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:08.452 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 10 2024-09-11T06:20:08.452 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 11 2024-09-11T06:20:08.452 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:20:08.452 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=10638], [object_number=47, tag_tid=8, entry_tid=10635]]], state=connected] 2024-09-11T06:20:08.452 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=43, tag_tid=8, entry_tid=7255], [object_number=42, tag_tid=8, entry_tid=7254], [object_number=41, tag_tid=8, entry_tid=7253], [object_number=40, tag_tid=8, entry_tid=7252]]], state=connected] 2024-09-11T06:20:08.455 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:08.457 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=43, tag_tid=8, entry_tid=7255' 2024-09-11T06:20:08.458 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=43, tag_tid=8, entry_tid=7255' 2024-09-11T06:20:08.458 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=43, tag_tid=8, entry_tid=7255' '!=' 'object_number=40, tag_tid=8, entry_tid=7164' 2024-09-11T06:20:08.458 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:20:08.458 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=43, tag_tid=8, entry_tid=7255' = 'object_number=40, tag_tid=8, entry_tid=7164' 2024-09-11T06:20:08.458 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=43, tag_tid=8, entry_tid=7255' 2024-09-11T06:20:08.458 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-09-11T06:20:08.458 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:20:08.458 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:08.458 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=9 2024-09-11T06:20:08.459 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=43, tag_tid=8, entry_tid=7255' 2024-09-11T06:20:08.459 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:20:08.459 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:08.460 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=8 2024-09-11T06:20:08.461 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-09-11T06:20:08.461 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:20:08.461 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:08.462 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:20:08.463 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=43, tag_tid=8, entry_tid=7255' 2024-09-11T06:20:08.463 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:20:08.463 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:08.464 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=7255 2024-09-11T06:20:08.464 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 9 = 8 -a 1 -le 7255 2024-09-11T06:20:08.464 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:20:08.464 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:20:08.464 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:20:08.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:20:08.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:20:08.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:20:08.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:20:08.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:20:08.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:20:08.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:20:08.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:20:08.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:20:08.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:20:08.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:20:08.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:20:08.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:20:08.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:20:08.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:20:08.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:20:08.668 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:20:08.670 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:20:08.670 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:20:08.670 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:20:08.670 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:20:08.670 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:20:08.671 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:20:08.671 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:20:08.671 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:20:08.671 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:20:08.671 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:20:08.671 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:20:08.671 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:20:08.671 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:20:08.671 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:20:08.671 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:20:08.671 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:20:08.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:20:08.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:20:08.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:20:08.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:20:08.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:20:08.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:20:08.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:20:08.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:20:08.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:20:08.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:20:08.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:20:08.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:20:08.674 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:20:08.674 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:20:08.674 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:20:08.674 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:20:08.697 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:08.697 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:08.697 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:08.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:20:08.826 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:20:08.828 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:20:08.828 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:08.828 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:08.828 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:08.828 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:20:08.828 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:08.828 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:08.828 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:08.828 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:20:08.829 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:20:08.829 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:20:08.829 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:20:08.830 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:08.830 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:20:08.831 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:08.885 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 10 2024-09-11T06:20:08.885 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 11 2024-09-11T06:20:08.885 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:20:08.885 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=10638], [object_number=47, tag_tid=8, entry_tid=10635]]], state=connected] 2024-09-11T06:20:08.885 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=41, tag_tid=8, entry_tid=7353], [object_number=40, tag_tid=8, entry_tid=7352], [object_number=43, tag_tid=8, entry_tid=7351], [object_number=42, tag_tid=8, entry_tid=7350]]], state=connected] 2024-09-11T06:20:08.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:08.890 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-09-11T06:20:08.891 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:20:08.891 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:08.891 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:08.891 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:08.891 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:20:08.892 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:08.892 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:08.892 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:08.892 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:20:08.892 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:20:08.892 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:20:08.892 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:20:08.893 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:08.893 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:20:08.893 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:08.938 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 10 2024-09-11T06:20:08.939 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 11 2024-09-11T06:20:08.939 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:20:08.939 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=10638], [object_number=47, tag_tid=8, entry_tid=10635]]], state=connected] 2024-09-11T06:20:08.939 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=41, tag_tid=8, entry_tid=7353], [object_number=40, tag_tid=8, entry_tid=7352], [object_number=43, tag_tid=8, entry_tid=7351], [object_number=42, tag_tid=8, entry_tid=7350]]], state=connected] 2024-09-11T06:20:08.942 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:08.944 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=41, tag_tid=8, entry_tid=7353' 2024-09-11T06:20:08.944 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=41, tag_tid=8, entry_tid=7353' 2024-09-11T06:20:08.944 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=41, tag_tid=8, entry_tid=7353' '!=' 'object_number=43, tag_tid=8, entry_tid=7255' 2024-09-11T06:20:08.944 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:20:08.944 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=41, tag_tid=8, entry_tid=7353' = 'object_number=43, tag_tid=8, entry_tid=7255' 2024-09-11T06:20:08.944 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=41, tag_tid=8, entry_tid=7353' 2024-09-11T06:20:08.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-09-11T06:20:08.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:20:08.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:08.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=9 2024-09-11T06:20:08.946 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=41, tag_tid=8, entry_tid=7353' 2024-09-11T06:20:08.946 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:20:08.946 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:08.947 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=8 2024-09-11T06:20:08.948 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-09-11T06:20:08.948 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:20:08.948 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:08.949 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:20:08.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=41, tag_tid=8, entry_tid=7353' 2024-09-11T06:20:08.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:20:08.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:08.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=7353 2024-09-11T06:20:08.952 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 9 = 8 -a 1 -le 7353 2024-09-11T06:20:08.952 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:20:08.952 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:20:08.952 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:20:09.153 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:20:09.153 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:20:09.153 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:20:09.154 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:20:09.154 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:20:09.154 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:20:09.154 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:20:09.154 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:20:09.154 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:20:09.154 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:20:09.154 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:20:09.154 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:20:09.154 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:20:09.154 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:20:09.154 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:20:09.155 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:20:09.155 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:20:09.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:20:09.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:20:09.159 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:20:09.159 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:20:09.159 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:20:09.159 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:20:09.159 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:20:09.159 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:20:09.159 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:20:09.159 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:20:09.159 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:20:09.159 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:20:09.159 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:20:09.159 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:20:09.159 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:20:09.160 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:20:09.160 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:20:09.160 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:20:09.160 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:20:09.160 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:20:09.160 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:20:09.160 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:20:09.160 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:20:09.160 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:20:09.160 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:20:09.160 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:20:09.160 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:20:09.160 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:20:09.160 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:20:09.161 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:20:09.161 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:20:09.161 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:20:09.184 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:09.184 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:09.185 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:09.185 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:20:09.314 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:20:09.315 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:20:09.315 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:09.315 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:09.315 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:09.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:20:09.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:09.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:09.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:09.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:20:09.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:20:09.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:20:09.317 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:20:09.318 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:09.318 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:20:09.318 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:09.369 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 10 2024-09-11T06:20:09.372 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 11 2024-09-11T06:20:09.372 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:20:09.372 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=10638], [object_number=47, tag_tid=8, entry_tid=10635]]], state=connected] 2024-09-11T06:20:09.372 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=42, tag_tid=8, entry_tid=7450], [object_number=41, tag_tid=8, entry_tid=7449], [object_number=40, tag_tid=8, entry_tid=7448], [object_number=43, tag_tid=8, entry_tid=7447]]], state=connected] 2024-09-11T06:20:09.373 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:09.373 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-09-11T06:20:09.374 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:20:09.374 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:09.374 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:09.375 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:09.375 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:20:09.375 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:09.375 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:09.375 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:09.375 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:20:09.375 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:20:09.375 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:20:09.375 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:20:09.376 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:09.376 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:20:09.376 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:09.421 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 10 2024-09-11T06:20:09.421 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 11 2024-09-11T06:20:09.422 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:20:09.422 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=10638], [object_number=47, tag_tid=8, entry_tid=10635]]], state=connected] 2024-09-11T06:20:09.422 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=42, tag_tid=8, entry_tid=7450], [object_number=41, tag_tid=8, entry_tid=7449], [object_number=40, tag_tid=8, entry_tid=7448], [object_number=43, tag_tid=8, entry_tid=7447]]], state=connected] 2024-09-11T06:20:09.425 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:09.428 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=42, tag_tid=8, entry_tid=7450' 2024-09-11T06:20:09.428 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=42, tag_tid=8, entry_tid=7450' 2024-09-11T06:20:09.428 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=42, tag_tid=8, entry_tid=7450' '!=' 'object_number=41, tag_tid=8, entry_tid=7353' 2024-09-11T06:20:09.428 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:20:09.428 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=42, tag_tid=8, entry_tid=7450' = 'object_number=41, tag_tid=8, entry_tid=7353' 2024-09-11T06:20:09.428 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=42, tag_tid=8, entry_tid=7450' 2024-09-11T06:20:09.428 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-09-11T06:20:09.428 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:20:09.428 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:09.429 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=9 2024-09-11T06:20:09.430 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=42, tag_tid=8, entry_tid=7450' 2024-09-11T06:20:09.430 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:20:09.430 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:09.431 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=8 2024-09-11T06:20:09.431 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-09-11T06:20:09.431 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:20:09.431 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:09.432 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:20:09.433 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=42, tag_tid=8, entry_tid=7450' 2024-09-11T06:20:09.433 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:20:09.433 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:09.435 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=7450 2024-09-11T06:20:09.435 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 9 = 8 -a 1 -le 7450 2024-09-11T06:20:09.435 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:20:09.435 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:20:09.435 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:20:09.636 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:20:09.637 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:20:09.637 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:20:09.637 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:20:09.637 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:20:09.637 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:20:09.637 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:20:09.637 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:20:09.637 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:20:09.637 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:20:09.637 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:20:09.637 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:20:09.637 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:20:09.637 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:20:09.638 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:20:09.638 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:20:09.638 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:20:09.640 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:20:09.640 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:20:09.640 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:20:09.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:20:09.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:20:09.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:20:09.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:20:09.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:20:09.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:20:09.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:20:09.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:20:09.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:20:09.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:20:09.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:20:09.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:20:09.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:20:09.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:20:09.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:20:09.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:20:09.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:20:09.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:20:09.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:20:09.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:20:09.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:20:09.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:20:09.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:20:09.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:20:09.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:20:09.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:20:09.645 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:20:09.645 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:20:09.645 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:20:09.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:09.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:09.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:09.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:20:09.794 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:20:09.795 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:20:09.795 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:09.795 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:09.796 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:09.796 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:20:09.796 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:09.796 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:09.796 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:09.796 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:20:09.796 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:20:09.797 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:20:09.797 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:20:09.798 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:09.799 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:20:09.799 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:09.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 10 2024-09-11T06:20:09.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 11 2024-09-11T06:20:09.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:20:09.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=10638], [object_number=47, tag_tid=8, entry_tid=10635]]], state=connected] 2024-09-11T06:20:09.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=42, tag_tid=8, entry_tid=7538], [object_number=41, tag_tid=8, entry_tid=7537], [object_number=40, tag_tid=8, entry_tid=7536], [object_number=43, tag_tid=8, entry_tid=7535]]], state=connected] 2024-09-11T06:20:09.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:09.855 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-09-11T06:20:09.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:20:09.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:09.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:09.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:09.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:20:09.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:09.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:09.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:09.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:20:09.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:20:09.857 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:20:09.857 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:20:09.857 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:09.857 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:20:09.857 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:09.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 10 2024-09-11T06:20:09.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 11 2024-09-11T06:20:09.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:20:09.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=10638], [object_number=47, tag_tid=8, entry_tid=10635]]], state=connected] 2024-09-11T06:20:09.905 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=42, tag_tid=8, entry_tid=7538], [object_number=41, tag_tid=8, entry_tid=7537], [object_number=40, tag_tid=8, entry_tid=7536], [object_number=43, tag_tid=8, entry_tid=7535]]], state=connected] 2024-09-11T06:20:09.953 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:09.955 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=42, tag_tid=8, entry_tid=7538' 2024-09-11T06:20:09.955 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=42, tag_tid=8, entry_tid=7538' 2024-09-11T06:20:09.955 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=42, tag_tid=8, entry_tid=7538' '!=' 'object_number=42, tag_tid=8, entry_tid=7450' 2024-09-11T06:20:09.955 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:20:09.955 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=42, tag_tid=8, entry_tid=7538' = 'object_number=42, tag_tid=8, entry_tid=7450' 2024-09-11T06:20:09.955 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=42, tag_tid=8, entry_tid=7538' 2024-09-11T06:20:09.956 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-09-11T06:20:09.956 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:09.956 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:20:09.958 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=9 2024-09-11T06:20:09.959 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=42, tag_tid=8, entry_tid=7538' 2024-09-11T06:20:09.959 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:20:09.959 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:09.960 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=8 2024-09-11T06:20:09.961 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-09-11T06:20:09.961 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:20:09.961 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:09.962 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:20:09.963 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=42, tag_tid=8, entry_tid=7538' 2024-09-11T06:20:09.963 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:20:09.963 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:09.964 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=7538 2024-09-11T06:20:09.964 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 9 = 8 -a 1 -le 7538 2024-09-11T06:20:09.965 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:20:09.965 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:20:09.965 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:20:10.167 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:20:10.167 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:20:10.167 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:20:10.167 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:20:10.167 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:20:10.167 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:20:10.167 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:20:10.167 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:20:10.167 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:20:10.167 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:20:10.167 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:20:10.167 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:20:10.168 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:20:10.168 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:20:10.168 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:20:10.168 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:20:10.168 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:20:10.171 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:20:10.171 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:20:10.171 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:20:10.171 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:20:10.171 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:20:10.171 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:20:10.171 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:20:10.171 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:20:10.171 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:20:10.171 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:20:10.172 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:20:10.172 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:20:10.172 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:20:10.172 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:20:10.172 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:20:10.172 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:20:10.173 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:20:10.173 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:20:10.173 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:20:10.173 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:20:10.173 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:20:10.173 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:20:10.173 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:20:10.173 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:20:10.174 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:20:10.174 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:20:10.174 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:20:10.174 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:20:10.174 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:20:10.174 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:20:10.174 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:20:10.174 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:20:10.197 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:10.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:10.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:10.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:20:10.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:20:10.337 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:20:10.337 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:10.337 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:10.337 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:10.337 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:20:10.337 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:10.337 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:10.337 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:10.337 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:20:10.338 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:20:10.338 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:20:10.338 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:20:10.340 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:10.340 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:20:10.341 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:10.388 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 10 2024-09-11T06:20:10.389 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 11 2024-09-11T06:20:10.389 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:20:10.389 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=10638], [object_number=47, tag_tid=8, entry_tid=10635]]], state=connected] 2024-09-11T06:20:10.389 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=42, tag_tid=8, entry_tid=7650], [object_number=41, tag_tid=8, entry_tid=7649], [object_number=40, tag_tid=8, entry_tid=7648], [object_number=43, tag_tid=8, entry_tid=7647]]], state=connected] 2024-09-11T06:20:10.445 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:10.447 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-09-11T06:20:10.448 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:20:10.448 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:10.448 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:10.448 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:10.448 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:20:10.448 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:10.448 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:10.448 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:10.448 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:20:10.448 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:20:10.448 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:20:10.449 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:20:10.449 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:10.450 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:20:10.450 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:10.498 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 10 2024-09-11T06:20:10.498 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 11 2024-09-11T06:20:10.498 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:20:10.498 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=10638], [object_number=47, tag_tid=8, entry_tid=10635]]], state=connected] 2024-09-11T06:20:10.498 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=42, tag_tid=8, entry_tid=7650], [object_number=41, tag_tid=8, entry_tid=7649], [object_number=40, tag_tid=8, entry_tid=7648], [object_number=43, tag_tid=8, entry_tid=7647]]], state=connected] 2024-09-11T06:20:10.501 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:10.503 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=42, tag_tid=8, entry_tid=7650' 2024-09-11T06:20:10.504 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=42, tag_tid=8, entry_tid=7650' 2024-09-11T06:20:10.504 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=42, tag_tid=8, entry_tid=7650' '!=' 'object_number=42, tag_tid=8, entry_tid=7538' 2024-09-11T06:20:10.504 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:20:10.504 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=42, tag_tid=8, entry_tid=7650' = 'object_number=42, tag_tid=8, entry_tid=7538' 2024-09-11T06:20:10.504 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=42, tag_tid=8, entry_tid=7650' 2024-09-11T06:20:10.504 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-09-11T06:20:10.504 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:20:10.504 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:10.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=9 2024-09-11T06:20:10.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=42, tag_tid=8, entry_tid=7650' 2024-09-11T06:20:10.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:20:10.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:10.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=8 2024-09-11T06:20:10.507 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-09-11T06:20:10.507 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:20:10.507 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:10.508 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:20:10.509 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=42, tag_tid=8, entry_tid=7650' 2024-09-11T06:20:10.509 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:20:10.509 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:10.510 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=7650 2024-09-11T06:20:10.510 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 9 = 8 -a 1 -le 7650 2024-09-11T06:20:10.510 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:20:10.510 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:20:10.511 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:20:10.712 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:20:10.712 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:20:10.712 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:20:10.712 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:20:10.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:20:10.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:20:10.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:20:10.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:20:10.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:20:10.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:20:10.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:20:10.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:20:10.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:20:10.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:20:10.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:20:10.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:20:10.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:20:10.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:20:10.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:20:10.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:20:10.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:20:10.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:20:10.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:20:10.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:20:10.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:20:10.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:20:10.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:20:10.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:20:10.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:20:10.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:20:10.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:20:10.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:20:10.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:20:10.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:20:10.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:20:10.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:20:10.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:20:10.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:20:10.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:20:10.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:20:10.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:20:10.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:20:10.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:20:10.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:20:10.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:20:10.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:20:10.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:20:10.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:20:10.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:20:10.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:10.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:10.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:10.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:20:10.884 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:20:10.885 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:20:10.885 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:10.885 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:10.885 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:10.885 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:20:10.885 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:10.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:10.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:10.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:20:10.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:20:10.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:20:10.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:20:10.887 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:10.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:20:10.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:10.942 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 10 2024-09-11T06:20:10.942 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 11 2024-09-11T06:20:10.943 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:20:10.943 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=10638], [object_number=47, tag_tid=8, entry_tid=10635]]], state=connected] 2024-09-11T06:20:10.943 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=40, tag_tid=8, entry_tid=7752], [object_number=43, tag_tid=8, entry_tid=7751], [object_number=42, tag_tid=8, entry_tid=7750], [object_number=41, tag_tid=8, entry_tid=7749]]], state=connected] 2024-09-11T06:20:10.946 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:10.947 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-09-11T06:20:10.948 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:20:10.949 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:10.949 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:10.949 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:10.949 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:20:10.949 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:10.949 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:10.949 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:10.949 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:20:10.949 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:20:10.949 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:20:10.949 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:20:10.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:10.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:20:10.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:10.996 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 10 2024-09-11T06:20:10.996 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 11 2024-09-11T06:20:10.996 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:20:10.997 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=10638], [object_number=47, tag_tid=8, entry_tid=10635]]], state=connected] 2024-09-11T06:20:10.997 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=40, tag_tid=8, entry_tid=7752], [object_number=43, tag_tid=8, entry_tid=7751], [object_number=42, tag_tid=8, entry_tid=7750], [object_number=41, tag_tid=8, entry_tid=7749]]], state=connected] 2024-09-11T06:20:11.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:11.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=40, tag_tid=8, entry_tid=7752' 2024-09-11T06:20:11.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=40, tag_tid=8, entry_tid=7752' 2024-09-11T06:20:11.002 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=40, tag_tid=8, entry_tid=7752' '!=' 'object_number=42, tag_tid=8, entry_tid=7650' 2024-09-11T06:20:11.002 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:20:11.002 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=40, tag_tid=8, entry_tid=7752' = 'object_number=42, tag_tid=8, entry_tid=7650' 2024-09-11T06:20:11.002 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=40, tag_tid=8, entry_tid=7752' 2024-09-11T06:20:11.002 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-09-11T06:20:11.002 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:20:11.003 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:11.003 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=9 2024-09-11T06:20:11.003 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=40, tag_tid=8, entry_tid=7752' 2024-09-11T06:20:11.004 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:20:11.004 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:11.005 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=8 2024-09-11T06:20:11.009 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-09-11T06:20:11.009 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:20:11.010 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:11.010 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:20:11.010 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=40, tag_tid=8, entry_tid=7752' 2024-09-11T06:20:11.010 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:20:11.010 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:11.010 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=7752 2024-09-11T06:20:11.010 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 9 = 8 -a 1 -le 7752 2024-09-11T06:20:11.010 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:20:11.010 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:20:11.010 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:20:11.216 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:20:11.216 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:20:11.216 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:20:11.216 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:20:11.216 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:20:11.216 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:20:11.216 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:20:11.216 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:20:11.216 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:20:11.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:20:11.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:20:11.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:20:11.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:20:11.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:20:11.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:20:11.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:20:11.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:20:11.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:20:11.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:20:11.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:20:11.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:20:11.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:20:11.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:20:11.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:20:11.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:20:11.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:20:11.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:20:11.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:20:11.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:20:11.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:20:11.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:20:11.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:20:11.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:20:11.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:20:11.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:20:11.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:20:11.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:20:11.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:20:11.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:20:11.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:20:11.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:20:11.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:20:11.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:20:11.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:20:11.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:20:11.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:20:11.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:20:11.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:20:11.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:20:11.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:11.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:11.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:11.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:20:11.409 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:20:11.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:20:11.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:11.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:11.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:11.411 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:20:11.411 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:11.411 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:11.411 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:11.411 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:20:11.412 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:20:11.412 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:20:11.412 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:20:11.412 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:11.412 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:20:11.413 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:11.464 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 10 2024-09-11T06:20:11.465 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 11 2024-09-11T06:20:11.465 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:20:11.465 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=10638], [object_number=47, tag_tid=8, entry_tid=10635]]], state=connected] 2024-09-11T06:20:11.465 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=43, tag_tid=8, entry_tid=7811], [object_number=42, tag_tid=8, entry_tid=7810], [object_number=45, tag_tid=8, entry_tid=7809], [object_number=40, tag_tid=8, entry_tid=7808]]], state=connected] 2024-09-11T06:20:11.468 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:11.469 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-09-11T06:20:11.470 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:20:11.470 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:11.470 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:11.471 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:11.471 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:20:11.471 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:11.471 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:11.471 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:11.471 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:20:11.471 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:20:11.471 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:20:11.471 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:20:11.472 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:11.472 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:20:11.472 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:11.522 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 10 2024-09-11T06:20:11.522 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 11 2024-09-11T06:20:11.522 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:20:11.522 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=10638], [object_number=47, tag_tid=8, entry_tid=10635]]], state=connected] 2024-09-11T06:20:11.522 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=43, tag_tid=8, entry_tid=7811], [object_number=42, tag_tid=8, entry_tid=7810], [object_number=45, tag_tid=8, entry_tid=7809], [object_number=40, tag_tid=8, entry_tid=7808]]], state=connected] 2024-09-11T06:20:11.525 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:11.527 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=43, tag_tid=8, entry_tid=7811' 2024-09-11T06:20:11.527 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=43, tag_tid=8, entry_tid=7811' 2024-09-11T06:20:11.527 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=43, tag_tid=8, entry_tid=7811' '!=' 'object_number=40, tag_tid=8, entry_tid=7752' 2024-09-11T06:20:11.527 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:20:11.527 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=43, tag_tid=8, entry_tid=7811' = 'object_number=40, tag_tid=8, entry_tid=7752' 2024-09-11T06:20:11.527 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=43, tag_tid=8, entry_tid=7811' 2024-09-11T06:20:11.528 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-09-11T06:20:11.528 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:20:11.528 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:11.528 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=9 2024-09-11T06:20:11.529 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=43, tag_tid=8, entry_tid=7811' 2024-09-11T06:20:11.529 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:20:11.529 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:11.530 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=8 2024-09-11T06:20:11.531 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-09-11T06:20:11.531 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:20:11.531 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:11.532 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:20:11.532 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=43, tag_tid=8, entry_tid=7811' 2024-09-11T06:20:11.533 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:20:11.533 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:11.534 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=7811 2024-09-11T06:20:11.534 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 9 = 8 -a 1 -le 7811 2024-09-11T06:20:11.534 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:20:11.535 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:20:11.535 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:20:11.736 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:20:11.736 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:20:11.736 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:20:11.736 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:20:11.736 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:20:11.736 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:20:11.737 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:20:11.737 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:20:11.737 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:20:11.737 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:20:11.737 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:20:11.737 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:20:11.737 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:20:11.737 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:20:11.737 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:20:11.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:20:11.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:20:11.741 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:20:11.741 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:20:11.741 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:20:11.741 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:20:11.741 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:20:11.741 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:20:11.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:20:11.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:20:11.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:20:11.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:20:11.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:20:11.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:20:11.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:20:11.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:20:11.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:20:11.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:20:11.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:20:11.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:20:11.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:20:11.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:20:11.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:20:11.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:20:11.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:20:11.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:20:11.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:20:11.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:20:11.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:20:11.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:20:11.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:20:11.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:20:11.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:20:11.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:20:11.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:11.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:11.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:11.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:20:11.902 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:20:11.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:20:11.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:11.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:11.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:11.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:20:11.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:11.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:11.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:11.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:20:11.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:20:11.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:20:11.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:20:11.905 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:11.906 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:11.907 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:20:11.970 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 11 2024-09-11T06:20:11.970 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 11 2024-09-11T06:20:11.970 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:20:11.970 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=10638], [object_number=47, tag_tid=8, entry_tid=10635]]], state=connected] 2024-09-11T06:20:11.970 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=44, tag_tid=8, entry_tid=7872], [object_number=47, tag_tid=8, entry_tid=7871], [object_number=46, tag_tid=8, entry_tid=7870], [object_number=45, tag_tid=8, entry_tid=7869]]], state=connected] 2024-09-11T06:20:11.973 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:11.975 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-09-11T06:20:11.976 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:20:11.976 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:11.976 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:11.976 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:11.976 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:20:11.976 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:11.976 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:11.976 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:11.976 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:20:11.976 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:20:11.977 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:20:11.977 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:20:11.977 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:11.977 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:20:11.977 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:12.023 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 11 2024-09-11T06:20:12.023 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 11 2024-09-11T06:20:12.023 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:20:12.023 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=10638], [object_number=47, tag_tid=8, entry_tid=10635]]], state=connected] 2024-09-11T06:20:12.023 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=44, tag_tid=8, entry_tid=7872], [object_number=47, tag_tid=8, entry_tid=7871], [object_number=46, tag_tid=8, entry_tid=7870], [object_number=45, tag_tid=8, entry_tid=7869]]], state=connected] 2024-09-11T06:20:12.026 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:12.028 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=44, tag_tid=8, entry_tid=7872' 2024-09-11T06:20:12.028 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=44, tag_tid=8, entry_tid=7872' 2024-09-11T06:20:12.028 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=44, tag_tid=8, entry_tid=7872' '!=' 'object_number=43, tag_tid=8, entry_tid=7811' 2024-09-11T06:20:12.029 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:20:12.029 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=44, tag_tid=8, entry_tid=7872' = 'object_number=43, tag_tid=8, entry_tid=7811' 2024-09-11T06:20:12.029 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=44, tag_tid=8, entry_tid=7872' 2024-09-11T06:20:12.029 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-09-11T06:20:12.029 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:20:12.030 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:12.030 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=9 2024-09-11T06:20:12.030 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=44, tag_tid=8, entry_tid=7872' 2024-09-11T06:20:12.031 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:20:12.031 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:12.031 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=8 2024-09-11T06:20:12.032 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-09-11T06:20:12.032 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:20:12.032 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:12.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:20:12.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=44, tag_tid=8, entry_tid=7872' 2024-09-11T06:20:12.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:20:12.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:12.035 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=7872 2024-09-11T06:20:12.035 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 9 = 8 -a 1 -le 7872 2024-09-11T06:20:12.035 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:20:12.035 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:20:12.035 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:20:12.237 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:20:12.237 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:20:12.237 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:20:12.237 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:20:12.238 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:20:12.238 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:20:12.238 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:20:12.238 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:20:12.238 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:20:12.238 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:20:12.238 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:20:12.238 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:20:12.238 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:20:12.238 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:20:12.238 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:20:12.239 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:20:12.239 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:20:12.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:20:12.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:20:12.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:20:12.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:20:12.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:20:12.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:20:12.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:20:12.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:20:12.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:20:12.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:20:12.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:20:12.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:20:12.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:20:12.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:20:12.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:20:12.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:20:12.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:20:12.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:20:12.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:20:12.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:20:12.244 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:20:12.244 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:20:12.244 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:20:12.244 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:20:12.244 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:20:12.244 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:20:12.244 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:20:12.244 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:20:12.244 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:20:12.244 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:20:12.244 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:20:12.245 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:20:12.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:12.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:12.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:12.269 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:20:12.399 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:20:12.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:20:12.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:12.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:12.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:12.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:20:12.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:12.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:12.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:12.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:20:12.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:20:12.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:20:12.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:20:12.403 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:12.403 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:20:12.404 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:12.452 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 11 2024-09-11T06:20:12.452 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 11 2024-09-11T06:20:12.452 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:20:12.452 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=10638], [object_number=47, tag_tid=8, entry_tid=10635]]], state=connected] 2024-09-11T06:20:12.452 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=45, tag_tid=8, entry_tid=7985], [object_number=44, tag_tid=8, entry_tid=7984], [object_number=47, tag_tid=8, entry_tid=7983], [object_number=46, tag_tid=8, entry_tid=7982]]], state=connected] 2024-09-11T06:20:12.456 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:12.457 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-09-11T06:20:12.458 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:20:12.458 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:12.458 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:12.458 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:12.458 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:20:12.458 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:12.458 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:12.459 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:12.459 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:20:12.459 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:20:12.459 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:20:12.459 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:20:12.459 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:12.459 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:20:12.460 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:12.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 11 2024-09-11T06:20:12.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 11 2024-09-11T06:20:12.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:20:12.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=10638], [object_number=47, tag_tid=8, entry_tid=10635]]], state=connected] 2024-09-11T06:20:12.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=45, tag_tid=8, entry_tid=7985], [object_number=44, tag_tid=8, entry_tid=7984], [object_number=47, tag_tid=8, entry_tid=7983], [object_number=46, tag_tid=8, entry_tid=7982]]], state=connected] 2024-09-11T06:20:12.508 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:12.511 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=45, tag_tid=8, entry_tid=7985' 2024-09-11T06:20:12.511 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=45, tag_tid=8, entry_tid=7985' 2024-09-11T06:20:12.511 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=45, tag_tid=8, entry_tid=7985' '!=' 'object_number=44, tag_tid=8, entry_tid=7872' 2024-09-11T06:20:12.511 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:20:12.511 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=45, tag_tid=8, entry_tid=7985' = 'object_number=44, tag_tid=8, entry_tid=7872' 2024-09-11T06:20:12.511 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=45, tag_tid=8, entry_tid=7985' 2024-09-11T06:20:12.511 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-09-11T06:20:12.511 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:20:12.511 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:12.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=9 2024-09-11T06:20:12.513 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=45, tag_tid=8, entry_tid=7985' 2024-09-11T06:20:12.513 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:20:12.513 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:12.513 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=8 2024-09-11T06:20:12.514 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-09-11T06:20:12.514 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:20:12.514 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:12.515 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:20:12.516 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=45, tag_tid=8, entry_tid=7985' 2024-09-11T06:20:12.516 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:20:12.516 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:12.517 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=7985 2024-09-11T06:20:12.517 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 9 = 8 -a 1 -le 7985 2024-09-11T06:20:12.518 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:20:12.518 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:20:12.518 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:20:12.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:20:12.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:20:12.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:20:12.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:20:12.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:20:12.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:20:12.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:20:12.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:20:12.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:20:12.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:20:12.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:20:12.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:20:12.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:20:12.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:20:12.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:20:12.721 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:20:12.721 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:20:12.724 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:20:12.724 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:20:12.724 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:20:12.724 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:20:12.724 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:20:12.724 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:20:12.724 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:20:12.724 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:20:12.724 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:20:12.725 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:20:12.725 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:20:12.725 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:20:12.725 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:20:12.725 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:20:12.725 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:20:12.725 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:20:12.725 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:20:12.725 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:20:12.725 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:20:12.725 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:20:12.725 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:20:12.725 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:20:12.725 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:20:12.726 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:20:12.726 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:20:12.726 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:20:12.726 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:20:12.726 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:20:12.726 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:20:12.726 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:20:12.726 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:20:12.726 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:20:12.749 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:12.749 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:12.749 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:12.749 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:20:12.879 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:20:12.880 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:20:12.880 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:12.880 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:12.880 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:12.880 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:20:12.880 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:12.880 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:12.880 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:12.880 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:20:12.881 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:20:12.881 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:20:12.881 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:20:12.884 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:12.884 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:20:12.884 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:12.934 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 11 2024-09-11T06:20:12.934 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 11 2024-09-11T06:20:12.935 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:20:12.935 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=10638], [object_number=47, tag_tid=8, entry_tid=10635]]], state=connected] 2024-09-11T06:20:12.935 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=44, tag_tid=8, entry_tid=8088], [object_number=47, tag_tid=8, entry_tid=8087], [object_number=46, tag_tid=8, entry_tid=8086], [object_number=45, tag_tid=8, entry_tid=8085]]], state=connected] 2024-09-11T06:20:12.938 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:12.939 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-09-11T06:20:12.940 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:20:12.941 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:12.941 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:12.941 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:12.941 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:20:12.941 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:12.941 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:12.941 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:12.941 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:20:12.941 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:20:12.941 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:20:12.941 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:20:12.942 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:12.942 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:20:12.942 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:12.987 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 11 2024-09-11T06:20:12.987 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 11 2024-09-11T06:20:12.987 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:20:12.987 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=10638], [object_number=47, tag_tid=8, entry_tid=10635]]], state=connected] 2024-09-11T06:20:12.987 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=44, tag_tid=8, entry_tid=8088], [object_number=47, tag_tid=8, entry_tid=8087], [object_number=46, tag_tid=8, entry_tid=8086], [object_number=45, tag_tid=8, entry_tid=8085]]], state=connected] 2024-09-11T06:20:12.991 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:12.992 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=44, tag_tid=8, entry_tid=8088' 2024-09-11T06:20:12.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=44, tag_tid=8, entry_tid=8088' 2024-09-11T06:20:12.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=44, tag_tid=8, entry_tid=8088' '!=' 'object_number=45, tag_tid=8, entry_tid=7985' 2024-09-11T06:20:12.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:20:12.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=44, tag_tid=8, entry_tid=8088' = 'object_number=45, tag_tid=8, entry_tid=7985' 2024-09-11T06:20:12.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=44, tag_tid=8, entry_tid=8088' 2024-09-11T06:20:12.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-09-11T06:20:12.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:20:12.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:12.994 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=9 2024-09-11T06:20:12.994 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=44, tag_tid=8, entry_tid=8088' 2024-09-11T06:20:12.994 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:20:12.994 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:12.995 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=8 2024-09-11T06:20:12.996 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-09-11T06:20:12.996 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:20:12.996 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:12.997 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:20:12.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=44, tag_tid=8, entry_tid=8088' 2024-09-11T06:20:12.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:20:12.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:12.999 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=8088 2024-09-11T06:20:12.999 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 9 = 8 -a 1 -le 8088 2024-09-11T06:20:12.999 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:20:12.999 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:20:12.999 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:20:13.201 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:20:13.201 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:20:13.201 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:20:13.201 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:20:13.201 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:20:13.201 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:20:13.201 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:20:13.201 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:20:13.201 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:20:13.202 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:20:13.202 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:20:13.202 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:20:13.202 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:20:13.202 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:20:13.202 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:20:13.202 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:20:13.202 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:20:13.205 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:20:13.205 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:20:13.205 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:20:13.205 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:20:13.205 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:20:13.206 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:20:13.206 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:20:13.206 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:20:13.206 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:20:13.206 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:20:13.206 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:20:13.206 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:20:13.206 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:20:13.206 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:20:13.206 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:20:13.206 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:20:13.206 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:20:13.206 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:20:13.206 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:20:13.206 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:20:13.207 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:20:13.207 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:20:13.207 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:20:13.207 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:20:13.207 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:20:13.207 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:20:13.207 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:20:13.207 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:20:13.207 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:20:13.208 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:20:13.208 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:20:13.208 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:20:13.231 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:13.232 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:13.232 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:13.232 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:20:13.367 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:20:13.367 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:20:13.367 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:13.367 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:13.367 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:13.367 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:20:13.368 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:13.368 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:13.368 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:13.368 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:20:13.368 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:20:13.369 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:20:13.369 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:20:13.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:13.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:20:13.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:13.423 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 11 2024-09-11T06:20:13.423 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 11 2024-09-11T06:20:13.423 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:20:13.423 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=10638], [object_number=47, tag_tid=8, entry_tid=10635]]], state=connected] 2024-09-11T06:20:13.423 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=46, tag_tid=8, entry_tid=8194], [object_number=45, tag_tid=8, entry_tid=8193], [object_number=44, tag_tid=8, entry_tid=8192], [object_number=47, tag_tid=8, entry_tid=8191]]], state=connected] 2024-09-11T06:20:13.427 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:13.428 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-09-11T06:20:13.429 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:20:13.429 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:13.429 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:13.429 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:13.429 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:20:13.430 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:13.430 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:13.430 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:13.430 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:20:13.430 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:20:13.430 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:20:13.430 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:20:13.430 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:13.431 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:20:13.431 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:13.477 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 11 2024-09-11T06:20:13.477 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 11 2024-09-11T06:20:13.477 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:20:13.477 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=10638], [object_number=47, tag_tid=8, entry_tid=10635]]], state=connected] 2024-09-11T06:20:13.477 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=46, tag_tid=8, entry_tid=8194], [object_number=45, tag_tid=8, entry_tid=8193], [object_number=44, tag_tid=8, entry_tid=8192], [object_number=47, tag_tid=8, entry_tid=8191]]], state=connected] 2024-09-11T06:20:13.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:13.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=46, tag_tid=8, entry_tid=8194' 2024-09-11T06:20:13.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=46, tag_tid=8, entry_tid=8194' 2024-09-11T06:20:13.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=46, tag_tid=8, entry_tid=8194' '!=' 'object_number=44, tag_tid=8, entry_tid=8088' 2024-09-11T06:20:13.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:20:13.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=46, tag_tid=8, entry_tid=8194' = 'object_number=44, tag_tid=8, entry_tid=8088' 2024-09-11T06:20:13.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=46, tag_tid=8, entry_tid=8194' 2024-09-11T06:20:13.483 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-09-11T06:20:13.483 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:20:13.483 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:13.483 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=9 2024-09-11T06:20:13.484 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=46, tag_tid=8, entry_tid=8194' 2024-09-11T06:20:13.484 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:20:13.484 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:13.485 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=8 2024-09-11T06:20:13.486 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-09-11T06:20:13.486 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:20:13.486 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:13.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:20:13.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=46, tag_tid=8, entry_tid=8194' 2024-09-11T06:20:13.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:20:13.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:13.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=8194 2024-09-11T06:20:13.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 9 = 8 -a 1 -le 8194 2024-09-11T06:20:13.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:20:13.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:20:13.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:20:13.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:20:13.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:20:13.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:20:13.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:20:13.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:20:13.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:20:13.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:20:13.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:20:13.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:20:13.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:20:13.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:20:13.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:20:13.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:20:13.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:20:13.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:20:13.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:20:13.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:20:13.695 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:20:13.695 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:20:13.696 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:20:13.696 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:20:13.696 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:20:13.696 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:20:13.696 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:20:13.696 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:20:13.696 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:20:13.696 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:20:13.696 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:20:13.696 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:20:13.696 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:20:13.696 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:20:13.697 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:20:13.697 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:20:13.697 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:20:13.697 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:20:13.697 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:20:13.697 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:20:13.697 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:20:13.697 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:20:13.697 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:20:13.697 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:20:13.697 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:20:13.697 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:20:13.697 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:20:13.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:20:13.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:20:13.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:20:13.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:20:13.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:20:13.723 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:13.723 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:13.723 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:13.723 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:20:13.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:20:13.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:20:13.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:13.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:13.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:13.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:20:13.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:13.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:13.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:13.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:20:13.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:20:13.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:20:13.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:20:13.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:13.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:20:13.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:13.898 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 11 2024-09-11T06:20:13.898 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 11 2024-09-11T06:20:13.898 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:20:13.898 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=10638], [object_number=47, tag_tid=8, entry_tid=10635]]], state=connected] 2024-09-11T06:20:13.898 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=47, tag_tid=8, entry_tid=8299], [object_number=46, tag_tid=8, entry_tid=8298], [object_number=45, tag_tid=8, entry_tid=8297], [object_number=44, tag_tid=8, entry_tid=8296]]], state=connected] 2024-09-11T06:20:13.901 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:13.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-09-11T06:20:13.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:20:13.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:13.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:13.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:13.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:20:13.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:13.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:13.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:13.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:20:13.905 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:20:13.905 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:20:13.905 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:20:13.906 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:13.906 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:20:13.906 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:13.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 11 2024-09-11T06:20:13.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 11 2024-09-11T06:20:13.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:20:13.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=10638], [object_number=47, tag_tid=8, entry_tid=10635]]], state=connected] 2024-09-11T06:20:13.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=47, tag_tid=8, entry_tid=8299], [object_number=46, tag_tid=8, entry_tid=8298], [object_number=45, tag_tid=8, entry_tid=8297], [object_number=44, tag_tid=8, entry_tid=8296]]], state=connected] 2024-09-11T06:20:13.954 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:13.956 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=47, tag_tid=8, entry_tid=8299' 2024-09-11T06:20:13.956 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=47, tag_tid=8, entry_tid=8299' 2024-09-11T06:20:13.956 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=47, tag_tid=8, entry_tid=8299' '!=' 'object_number=46, tag_tid=8, entry_tid=8194' 2024-09-11T06:20:13.956 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:20:13.957 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=47, tag_tid=8, entry_tid=8299' = 'object_number=46, tag_tid=8, entry_tid=8194' 2024-09-11T06:20:13.957 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=47, tag_tid=8, entry_tid=8299' 2024-09-11T06:20:13.957 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-09-11T06:20:13.957 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:20:13.957 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:13.957 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=9 2024-09-11T06:20:13.958 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=47, tag_tid=8, entry_tid=8299' 2024-09-11T06:20:13.958 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:20:13.958 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:13.959 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=8 2024-09-11T06:20:13.960 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-09-11T06:20:13.960 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:20:13.960 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:13.961 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:20:13.961 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=47, tag_tid=8, entry_tid=8299' 2024-09-11T06:20:13.962 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:20:13.962 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:13.963 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=8299 2024-09-11T06:20:13.963 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 9 = 8 -a 1 -le 8299 2024-09-11T06:20:13.963 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:20:13.963 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:20:13.963 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:20:14.164 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:20:14.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:20:14.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:20:14.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:20:14.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:20:14.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:20:14.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:20:14.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:20:14.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:20:14.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:20:14.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:20:14.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:20:14.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:20:14.166 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:20:14.166 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:20:14.166 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:20:14.166 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:20:14.168 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:20:14.168 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:20:14.168 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:20:14.168 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:20:14.168 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:20:14.168 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:20:14.168 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:20:14.169 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:20:14.169 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:20:14.169 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:20:14.169 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:20:14.169 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:20:14.169 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:20:14.169 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:20:14.169 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:20:14.169 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:20:14.170 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:20:14.170 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:20:14.170 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:20:14.170 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:20:14.170 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:20:14.170 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:20:14.170 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:20:14.170 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:20:14.171 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:20:14.171 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:20:14.171 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:20:14.171 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:20:14.171 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:20:14.171 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:20:14.171 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:20:14.171 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:20:14.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:14.196 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:14.196 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:14.196 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:20:14.328 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:20:14.330 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:20:14.330 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:14.330 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:14.330 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:14.330 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:20:14.330 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:14.330 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:14.330 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:14.330 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:20:14.330 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:20:14.331 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:20:14.331 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:20:14.332 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:14.332 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:20:14.332 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:14.384 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 11 2024-09-11T06:20:14.384 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 11 2024-09-11T06:20:14.385 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:20:14.385 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=10638], [object_number=47, tag_tid=8, entry_tid=10635]]], state=connected] 2024-09-11T06:20:14.385 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=45, tag_tid=8, entry_tid=8405], [object_number=44, tag_tid=8, entry_tid=8404], [object_number=47, tag_tid=8, entry_tid=8403], [object_number=46, tag_tid=8, entry_tid=8402]]], state=connected] 2024-09-11T06:20:14.388 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:14.389 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-09-11T06:20:14.390 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:20:14.391 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:14.391 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:14.391 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:14.391 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:20:14.391 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:14.391 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:14.391 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:14.391 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:20:14.391 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:20:14.391 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:20:14.391 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:20:14.392 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:14.392 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:20:14.392 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:14.438 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 11 2024-09-11T06:20:14.438 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 11 2024-09-11T06:20:14.438 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:20:14.438 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=10638], [object_number=47, tag_tid=8, entry_tid=10635]]], state=connected] 2024-09-11T06:20:14.438 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=45, tag_tid=8, entry_tid=8405], [object_number=44, tag_tid=8, entry_tid=8404], [object_number=47, tag_tid=8, entry_tid=8403], [object_number=46, tag_tid=8, entry_tid=8402]]], state=connected] 2024-09-11T06:20:14.441 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:14.443 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=45, tag_tid=8, entry_tid=8405' 2024-09-11T06:20:14.443 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=45, tag_tid=8, entry_tid=8405' 2024-09-11T06:20:14.443 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=45, tag_tid=8, entry_tid=8405' '!=' 'object_number=47, tag_tid=8, entry_tid=8299' 2024-09-11T06:20:14.443 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:20:14.443 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=45, tag_tid=8, entry_tid=8405' = 'object_number=47, tag_tid=8, entry_tid=8299' 2024-09-11T06:20:14.443 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=45, tag_tid=8, entry_tid=8405' 2024-09-11T06:20:14.444 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-09-11T06:20:14.444 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:20:14.444 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:14.444 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=9 2024-09-11T06:20:14.445 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=45, tag_tid=8, entry_tid=8405' 2024-09-11T06:20:14.445 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:20:14.445 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:14.446 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=8 2024-09-11T06:20:14.447 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-09-11T06:20:14.447 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:20:14.447 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:14.448 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:20:14.448 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=45, tag_tid=8, entry_tid=8405' 2024-09-11T06:20:14.449 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:20:14.449 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:14.450 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=8405 2024-09-11T06:20:14.450 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 9 = 8 -a 1 -le 8405 2024-09-11T06:20:14.450 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:20:14.450 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:20:14.450 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:20:14.652 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:20:14.652 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:20:14.652 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:20:14.652 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:20:14.652 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:20:14.652 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:20:14.652 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:20:14.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:20:14.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:20:14.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:20:14.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:20:14.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:20:14.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:20:14.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:20:14.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:20:14.654 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:20:14.654 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:20:14.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:20:14.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:20:14.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:20:14.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:20:14.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:20:14.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:20:14.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:20:14.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:20:14.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:20:14.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:20:14.658 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:20:14.658 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:20:14.658 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:20:14.658 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:20:14.658 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:20:14.658 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:20:14.658 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:20:14.658 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:20:14.658 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:20:14.658 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:20:14.658 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:20:14.658 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:20:14.658 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:20:14.658 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:20:14.659 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:20:14.659 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:20:14.659 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:20:14.659 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:20:14.659 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:20:14.659 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:20:14.659 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:20:14.659 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:20:14.683 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:14.683 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:14.683 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:14.683 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:20:14.820 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:20:14.820 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:20:14.820 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:14.820 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:14.820 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:14.820 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:20:14.820 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:14.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:14.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:14.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:20:14.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:20:14.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:20:14.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:20:14.822 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:14.822 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:20:14.822 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:14.875 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 11 2024-09-11T06:20:14.875 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 11 2024-09-11T06:20:14.875 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:20:14.875 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=10638], [object_number=47, tag_tid=8, entry_tid=10635]]], state=connected] 2024-09-11T06:20:14.875 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=47, tag_tid=8, entry_tid=8519], [object_number=46, tag_tid=8, entry_tid=8518], [object_number=45, tag_tid=8, entry_tid=8517], [object_number=44, tag_tid=8, entry_tid=8516]]], state=connected] 2024-09-11T06:20:14.878 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:14.879 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-09-11T06:20:14.881 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:20:14.881 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:14.881 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:14.881 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:14.881 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:20:14.881 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:14.881 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:14.881 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:14.881 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:20:14.881 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:20:14.882 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:20:14.882 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:20:14.882 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:14.882 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:20:14.882 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:14.930 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 11 2024-09-11T06:20:14.930 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 11 2024-09-11T06:20:14.930 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:20:14.930 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=10638], [object_number=47, tag_tid=8, entry_tid=10635]]], state=connected] 2024-09-11T06:20:14.930 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=47, tag_tid=8, entry_tid=8519], [object_number=46, tag_tid=8, entry_tid=8518], [object_number=45, tag_tid=8, entry_tid=8517], [object_number=44, tag_tid=8, entry_tid=8516]]], state=connected] 2024-09-11T06:20:14.933 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:14.935 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=47, tag_tid=8, entry_tid=8519' 2024-09-11T06:20:14.935 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=47, tag_tid=8, entry_tid=8519' 2024-09-11T06:20:14.935 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=47, tag_tid=8, entry_tid=8519' '!=' 'object_number=45, tag_tid=8, entry_tid=8405' 2024-09-11T06:20:14.935 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:20:14.935 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=47, tag_tid=8, entry_tid=8519' = 'object_number=45, tag_tid=8, entry_tid=8405' 2024-09-11T06:20:14.935 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=47, tag_tid=8, entry_tid=8519' 2024-09-11T06:20:14.936 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-09-11T06:20:14.936 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:20:14.936 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:14.936 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=9 2024-09-11T06:20:14.937 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=47, tag_tid=8, entry_tid=8519' 2024-09-11T06:20:14.937 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:20:14.937 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:14.938 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=8 2024-09-11T06:20:14.939 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-09-11T06:20:14.939 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:20:14.939 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:14.940 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:20:14.940 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=47, tag_tid=8, entry_tid=8519' 2024-09-11T06:20:14.940 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:20:14.941 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:14.942 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=8519 2024-09-11T06:20:14.942 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 9 = 8 -a 1 -le 8519 2024-09-11T06:20:14.942 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:20:14.942 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:20:14.942 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:20:15.144 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:20:15.144 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:20:15.144 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:20:15.144 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:20:15.144 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:20:15.144 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:20:15.144 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:20:15.145 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:20:15.145 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:20:15.145 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:20:15.145 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:20:15.145 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:20:15.145 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:20:15.145 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:20:15.145 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:20:15.146 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:20:15.146 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:20:15.148 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:20:15.149 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:20:15.149 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:20:15.149 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:20:15.149 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:20:15.149 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:20:15.149 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:20:15.149 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:20:15.149 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:20:15.149 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:20:15.149 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:20:15.149 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:20:15.149 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:20:15.149 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:20:15.150 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:20:15.150 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:20:15.150 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:20:15.150 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:20:15.150 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:20:15.150 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:20:15.150 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:20:15.150 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:20:15.150 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:20:15.150 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:20:15.150 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:20:15.150 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:20:15.150 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:20:15.150 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:20:15.151 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:20:15.151 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:20:15.151 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:20:15.151 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:20:15.173 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:15.173 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:15.174 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:15.174 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:20:15.306 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:20:15.307 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:20:15.307 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:15.307 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:15.307 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:15.307 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:20:15.307 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:15.307 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:15.307 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:15.307 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:20:15.307 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:20:15.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:20:15.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:20:15.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:15.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:20:15.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:15.356 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 11 2024-09-11T06:20:15.356 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 11 2024-09-11T06:20:15.356 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:20:15.356 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=10638], [object_number=47, tag_tid=8, entry_tid=10635]]], state=connected] 2024-09-11T06:20:15.356 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=45, tag_tid=8, entry_tid=8641], [object_number=44, tag_tid=8, entry_tid=8640], [object_number=47, tag_tid=8, entry_tid=8639], [object_number=46, tag_tid=8, entry_tid=8638]]], state=connected] 2024-09-11T06:20:15.359 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:15.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-09-11T06:20:15.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:20:15.362 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:15.362 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:15.362 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:15.362 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:20:15.362 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:15.362 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:15.362 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:15.362 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:20:15.362 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:20:15.362 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:20:15.362 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:20:15.363 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:15.363 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:20:15.363 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:15.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 11 2024-09-11T06:20:15.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 11 2024-09-11T06:20:15.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:20:15.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=10638], [object_number=47, tag_tid=8, entry_tid=10635]]], state=connected] 2024-09-11T06:20:15.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=45, tag_tid=8, entry_tid=8641], [object_number=44, tag_tid=8, entry_tid=8640], [object_number=47, tag_tid=8, entry_tid=8639], [object_number=46, tag_tid=8, entry_tid=8638]]], state=connected] 2024-09-11T06:20:15.413 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:15.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=45, tag_tid=8, entry_tid=8641' 2024-09-11T06:20:15.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=45, tag_tid=8, entry_tid=8641' 2024-09-11T06:20:15.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=45, tag_tid=8, entry_tid=8641' '!=' 'object_number=47, tag_tid=8, entry_tid=8519' 2024-09-11T06:20:15.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:20:15.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=45, tag_tid=8, entry_tid=8641' = 'object_number=47, tag_tid=8, entry_tid=8519' 2024-09-11T06:20:15.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=45, tag_tid=8, entry_tid=8641' 2024-09-11T06:20:15.416 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-09-11T06:20:15.416 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:20:15.416 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:15.416 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=9 2024-09-11T06:20:15.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=45, tag_tid=8, entry_tid=8641' 2024-09-11T06:20:15.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:20:15.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:15.418 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=8 2024-09-11T06:20:15.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-09-11T06:20:15.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:20:15.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:15.420 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:20:15.420 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=45, tag_tid=8, entry_tid=8641' 2024-09-11T06:20:15.421 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:20:15.421 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:15.422 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=8641 2024-09-11T06:20:15.422 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 9 = 8 -a 1 -le 8641 2024-09-11T06:20:15.422 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:20:15.422 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:20:15.422 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:20:15.624 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:20:15.624 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:20:15.624 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:20:15.624 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:20:15.624 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:20:15.624 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:20:15.624 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:20:15.624 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:20:15.624 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:20:15.625 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:20:15.625 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:20:15.625 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:20:15.625 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:20:15.625 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:20:15.625 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:20:15.625 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:20:15.626 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:20:15.628 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:20:15.628 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:20:15.628 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:20:15.628 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:20:15.628 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:20:15.628 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:20:15.628 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:20:15.629 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:20:15.629 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:20:15.629 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:20:15.629 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:20:15.629 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:20:15.629 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:20:15.629 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:20:15.629 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:20:15.629 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:20:15.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:20:15.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:20:15.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:20:15.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:20:15.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:20:15.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:20:15.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:20:15.631 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:20:15.631 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:20:15.631 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:20:15.631 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:20:15.631 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:20:15.631 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:20:15.631 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:20:15.631 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:20:15.631 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:20:15.654 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:15.655 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:15.655 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:15.655 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:20:15.783 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:20:15.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:20:15.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:15.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:15.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:15.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:20:15.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:15.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:15.785 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:15.785 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:20:15.785 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:20:15.785 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:20:15.785 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:20:15.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:15.787 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:20:15.787 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:15.838 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 11 2024-09-11T06:20:15.838 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 11 2024-09-11T06:20:15.838 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:20:15.838 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=10638], [object_number=47, tag_tid=8, entry_tid=10635]]], state=connected] 2024-09-11T06:20:15.838 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=46, tag_tid=8, entry_tid=8750], [object_number=45, tag_tid=8, entry_tid=8749], [object_number=44, tag_tid=8, entry_tid=8748], [object_number=47, tag_tid=8, entry_tid=8747]]], state=connected] 2024-09-11T06:20:15.842 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:15.843 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-09-11T06:20:15.844 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:20:15.844 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:15.844 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:15.844 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:15.844 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:20:15.844 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:15.844 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:15.845 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:15.845 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:20:15.845 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:20:15.845 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:20:15.845 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:20:15.845 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:15.846 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:20:15.846 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:15.892 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 11 2024-09-11T06:20:15.892 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 11 2024-09-11T06:20:15.892 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:20:15.892 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=10638], [object_number=47, tag_tid=8, entry_tid=10635]]], state=connected] 2024-09-11T06:20:15.892 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=46, tag_tid=8, entry_tid=8750], [object_number=45, tag_tid=8, entry_tid=8749], [object_number=44, tag_tid=8, entry_tid=8748], [object_number=47, tag_tid=8, entry_tid=8747]]], state=connected] 2024-09-11T06:20:15.895 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:15.897 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=46, tag_tid=8, entry_tid=8750' 2024-09-11T06:20:15.897 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=46, tag_tid=8, entry_tid=8750' 2024-09-11T06:20:15.897 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=46, tag_tid=8, entry_tid=8750' '!=' 'object_number=45, tag_tid=8, entry_tid=8641' 2024-09-11T06:20:15.898 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:20:15.898 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=46, tag_tid=8, entry_tid=8750' = 'object_number=45, tag_tid=8, entry_tid=8641' 2024-09-11T06:20:15.898 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=46, tag_tid=8, entry_tid=8750' 2024-09-11T06:20:15.898 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-09-11T06:20:15.898 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:20:15.898 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:15.898 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=9 2024-09-11T06:20:15.899 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=46, tag_tid=8, entry_tid=8750' 2024-09-11T06:20:15.899 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:20:15.899 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:15.900 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=8 2024-09-11T06:20:15.901 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-09-11T06:20:15.901 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:20:15.901 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:15.902 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:20:15.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=46, tag_tid=8, entry_tid=8750' 2024-09-11T06:20:15.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:20:15.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:15.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=8750 2024-09-11T06:20:15.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 9 = 8 -a 1 -le 8750 2024-09-11T06:20:15.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:20:15.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:20:15.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:20:16.106 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:20:16.106 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:20:16.106 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:20:16.106 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:20:16.106 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:20:16.106 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:20:16.106 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:20:16.107 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:20:16.107 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:20:16.107 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:20:16.107 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:20:16.107 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:20:16.107 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:20:16.107 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:20:16.107 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:20:16.108 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:20:16.108 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:20:16.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:20:16.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:20:16.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:20:16.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:20:16.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:20:16.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:20:16.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:20:16.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:20:16.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:20:16.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:20:16.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:20:16.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:20:16.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:20:16.112 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:20:16.112 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:20:16.112 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:20:16.112 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:20:16.112 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:20:16.112 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:20:16.112 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:20:16.112 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:20:16.112 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:20:16.112 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:20:16.112 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:20:16.112 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:20:16.113 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:20:16.113 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:20:16.113 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:20:16.113 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:20:16.113 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:20:16.113 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:20:16.113 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:20:16.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:16.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:16.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:16.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:20:16.261 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:20:16.262 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:20:16.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:16.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:16.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:16.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:20:16.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:16.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:16.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:16.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:20:16.264 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:20:16.264 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:20:16.264 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:20:16.265 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:16.265 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:20:16.265 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:16.526 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 11 2024-09-11T06:20:16.526 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 11 2024-09-11T06:20:16.526 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:20:16.527 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=10638], [object_number=47, tag_tid=8, entry_tid=10635]]], state=connected] 2024-09-11T06:20:16.527 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=46, tag_tid=8, entry_tid=8850], [object_number=45, tag_tid=8, entry_tid=8849], [object_number=44, tag_tid=8, entry_tid=8848], [object_number=47, tag_tid=8, entry_tid=8847]]], state=connected] 2024-09-11T06:20:16.534 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:16.534 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-09-11T06:20:16.534 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:20:16.534 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:16.535 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:16.535 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:16.535 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:20:16.535 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:16.535 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:16.535 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:16.535 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:20:16.535 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:20:16.535 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:20:16.535 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:20:16.536 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:16.536 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:20:16.536 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:16.586 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 11 2024-09-11T06:20:16.586 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 11 2024-09-11T06:20:16.586 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:20:16.586 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=10638], [object_number=47, tag_tid=8, entry_tid=10635]]], state=connected] 2024-09-11T06:20:16.586 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=46, tag_tid=8, entry_tid=8850], [object_number=45, tag_tid=8, entry_tid=8849], [object_number=44, tag_tid=8, entry_tid=8848], [object_number=47, tag_tid=8, entry_tid=8847]]], state=connected] 2024-09-11T06:20:16.589 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:16.592 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=46, tag_tid=8, entry_tid=8850' 2024-09-11T06:20:16.592 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=46, tag_tid=8, entry_tid=8850' 2024-09-11T06:20:16.592 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=46, tag_tid=8, entry_tid=8850' '!=' 'object_number=46, tag_tid=8, entry_tid=8750' 2024-09-11T06:20:16.592 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:20:16.592 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=46, tag_tid=8, entry_tid=8850' = 'object_number=46, tag_tid=8, entry_tid=8750' 2024-09-11T06:20:16.592 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=46, tag_tid=8, entry_tid=8850' 2024-09-11T06:20:16.592 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-09-11T06:20:16.592 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:20:16.592 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:16.593 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=9 2024-09-11T06:20:16.593 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=46, tag_tid=8, entry_tid=8850' 2024-09-11T06:20:16.594 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:20:16.594 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:16.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=8 2024-09-11T06:20:16.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-09-11T06:20:16.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:20:16.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:16.596 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:20:16.597 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=46, tag_tid=8, entry_tid=8850' 2024-09-11T06:20:16.597 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:20:16.597 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:16.598 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=8850 2024-09-11T06:20:16.599 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 9 = 8 -a 1 -le 8850 2024-09-11T06:20:16.599 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:20:16.599 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:20:16.599 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:20:16.800 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:20:16.800 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:20:16.800 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:20:16.800 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:20:16.801 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:20:16.801 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:20:16.801 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:20:16.801 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:20:16.801 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:20:16.801 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:20:16.801 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:20:16.801 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:20:16.801 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:20:16.801 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:20:16.801 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:20:16.802 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:20:16.802 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:20:16.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:20:16.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:20:16.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:20:16.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:20:16.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:20:16.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:20:16.806 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:20:16.806 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:20:16.806 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:20:16.806 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:20:16.806 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:20:16.806 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:20:16.806 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:20:16.806 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:20:16.806 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:20:16.806 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:20:16.806 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:20:16.806 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:20:16.806 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:20:16.806 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:20:16.807 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:20:16.807 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:20:16.807 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:20:16.807 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:20:16.807 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:20:16.807 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:20:16.807 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:20:16.808 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:20:16.808 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:20:16.808 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:20:16.808 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:20:16.808 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:20:16.831 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:16.831 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:16.831 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:16.831 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:20:16.958 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:20:16.959 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:20:16.960 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:16.960 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:16.960 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:16.960 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:20:16.960 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:16.960 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:16.960 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:16.960 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:20:16.960 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:20:16.961 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:20:16.961 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:20:16.962 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:16.962 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:20:16.962 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:17.018 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 11 2024-09-11T06:20:17.018 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 11 2024-09-11T06:20:17.018 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:20:17.018 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=10638], [object_number=47, tag_tid=8, entry_tid=10635]]], state=connected] 2024-09-11T06:20:17.019 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=45, tag_tid=8, entry_tid=8997], [object_number=44, tag_tid=8, entry_tid=8996], [object_number=47, tag_tid=8, entry_tid=8995], [object_number=46, tag_tid=8, entry_tid=8994]]], state=connected] 2024-09-11T06:20:17.022 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:17.023 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-09-11T06:20:17.024 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:20:17.024 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:17.024 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:17.024 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:17.025 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:20:17.025 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:17.025 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:17.025 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:17.025 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:20:17.025 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:20:17.025 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:20:17.025 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:20:17.025 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:17.026 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:20:17.026 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:17.072 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 11 2024-09-11T06:20:17.072 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 11 2024-09-11T06:20:17.072 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:20:17.072 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=10638], [object_number=47, tag_tid=8, entry_tid=10635]]], state=connected] 2024-09-11T06:20:17.072 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=45, tag_tid=8, entry_tid=8997], [object_number=44, tag_tid=8, entry_tid=8996], [object_number=47, tag_tid=8, entry_tid=8995], [object_number=46, tag_tid=8, entry_tid=8994]]], state=connected] 2024-09-11T06:20:17.076 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:17.078 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=45, tag_tid=8, entry_tid=8997' 2024-09-11T06:20:17.078 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=45, tag_tid=8, entry_tid=8997' 2024-09-11T06:20:17.078 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=45, tag_tid=8, entry_tid=8997' '!=' 'object_number=46, tag_tid=8, entry_tid=8850' 2024-09-11T06:20:17.078 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:20:17.078 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=45, tag_tid=8, entry_tid=8997' = 'object_number=46, tag_tid=8, entry_tid=8850' 2024-09-11T06:20:17.079 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=45, tag_tid=8, entry_tid=8997' 2024-09-11T06:20:17.079 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-09-11T06:20:17.079 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:20:17.079 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:17.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=9 2024-09-11T06:20:17.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=45, tag_tid=8, entry_tid=8997' 2024-09-11T06:20:17.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:20:17.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:17.081 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=8 2024-09-11T06:20:17.082 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-09-11T06:20:17.082 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:20:17.082 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:17.083 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:20:17.083 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=45, tag_tid=8, entry_tid=8997' 2024-09-11T06:20:17.084 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:20:17.084 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:17.085 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=8997 2024-09-11T06:20:17.085 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 9 = 8 -a 1 -le 8997 2024-09-11T06:20:17.085 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:20:17.085 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:20:17.085 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:20:17.287 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:20:17.287 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:20:17.287 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:20:17.287 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:20:17.287 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:20:17.287 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:20:17.287 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:20:17.287 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:20:17.287 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:20:17.287 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:20:17.287 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:20:17.287 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:20:17.288 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:20:17.288 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:20:17.288 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:20:17.288 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:20:17.288 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:20:17.291 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:20:17.291 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:20:17.291 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:20:17.291 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:20:17.291 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:20:17.291 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:20:17.291 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:20:17.291 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:20:17.291 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:20:17.291 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:20:17.291 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:20:17.291 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:20:17.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:20:17.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:20:17.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:20:17.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:20:17.293 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:20:17.293 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:20:17.293 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:20:17.293 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:20:17.293 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:20:17.293 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:20:17.294 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:20:17.294 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:20:17.294 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:20:17.294 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:20:17.294 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:20:17.294 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:20:17.294 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:20:17.294 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:20:17.294 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:20:17.294 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:20:17.317 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:17.318 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:17.318 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:17.318 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:20:17.454 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:20:17.454 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:20:17.454 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:17.454 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:17.454 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:17.454 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:20:17.454 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:17.454 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:17.454 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:17.455 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:20:17.455 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:20:17.455 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:20:17.455 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:20:17.456 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:17.457 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:20:17.457 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:17.510 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 11 2024-09-11T06:20:17.510 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 11 2024-09-11T06:20:17.510 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:20:17.510 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=10638], [object_number=47, tag_tid=8, entry_tid=10635]]], state=connected] 2024-09-11T06:20:17.510 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=45, tag_tid=8, entry_tid=9105], [object_number=44, tag_tid=8, entry_tid=9104], [object_number=47, tag_tid=8, entry_tid=9103], [object_number=46, tag_tid=8, entry_tid=9102]]], state=connected] 2024-09-11T06:20:17.513 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:17.514 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-09-11T06:20:17.515 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:20:17.515 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:17.515 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:17.516 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:17.516 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:20:17.516 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:17.516 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:17.516 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:17.516 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:20:17.516 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:20:17.516 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:20:17.516 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:20:17.516 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:17.517 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:20:17.517 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:17.563 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 11 2024-09-11T06:20:17.564 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 11 2024-09-11T06:20:17.564 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:20:17.564 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=10638], [object_number=47, tag_tid=8, entry_tid=10635]]], state=connected] 2024-09-11T06:20:17.564 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=45, tag_tid=8, entry_tid=9105], [object_number=44, tag_tid=8, entry_tid=9104], [object_number=47, tag_tid=8, entry_tid=9103], [object_number=46, tag_tid=8, entry_tid=9102]]], state=connected] 2024-09-11T06:20:17.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:17.569 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=45, tag_tid=8, entry_tid=9105' 2024-09-11T06:20:17.569 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=45, tag_tid=8, entry_tid=9105' 2024-09-11T06:20:17.569 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=45, tag_tid=8, entry_tid=9105' '!=' 'object_number=45, tag_tid=8, entry_tid=8997' 2024-09-11T06:20:17.569 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:20:17.569 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=45, tag_tid=8, entry_tid=9105' = 'object_number=45, tag_tid=8, entry_tid=8997' 2024-09-11T06:20:17.569 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=45, tag_tid=8, entry_tid=9105' 2024-09-11T06:20:17.570 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-09-11T06:20:17.570 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:20:17.570 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:17.570 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=9 2024-09-11T06:20:17.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=45, tag_tid=8, entry_tid=9105' 2024-09-11T06:20:17.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:20:17.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:17.572 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=8 2024-09-11T06:20:17.573 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-09-11T06:20:17.573 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:20:17.573 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:17.574 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:20:17.574 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=45, tag_tid=8, entry_tid=9105' 2024-09-11T06:20:17.574 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:20:17.574 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:17.576 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=9105 2024-09-11T06:20:17.576 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 9 = 8 -a 1 -le 9105 2024-09-11T06:20:17.576 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:20:17.576 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:20:17.576 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:20:17.777 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:20:17.777 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:20:17.777 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:20:17.778 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:20:17.778 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:20:17.778 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:20:17.778 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:20:17.778 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:20:17.778 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:20:17.778 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:20:17.778 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:20:17.779 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:20:17.779 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:20:17.779 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:20:17.779 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:20:17.779 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:20:17.780 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:20:17.782 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:20:17.782 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:20:17.782 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:20:17.782 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:20:17.783 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:20:17.783 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:20:17.783 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:20:17.783 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:20:17.783 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:20:17.783 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:20:17.783 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:20:17.783 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:20:17.783 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:20:17.783 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:20:17.783 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:20:17.783 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:20:17.783 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:20:17.783 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:20:17.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:20:17.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:20:17.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:20:17.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:20:17.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:20:17.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:20:17.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:20:17.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:20:17.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:20:17.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:20:17.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:20:17.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:20:17.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:20:17.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:20:17.807 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:17.808 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:17.808 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:17.808 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:20:17.938 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:20:17.939 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:20:17.940 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:17.940 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:17.940 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:17.940 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:20:17.940 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:17.940 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:17.940 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:17.940 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:20:17.941 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:20:17.941 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:20:17.941 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:20:17.942 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:17.942 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:20:17.943 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:17.995 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 11 2024-09-11T06:20:17.995 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 11 2024-09-11T06:20:17.995 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:20:17.995 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=10638], [object_number=47, tag_tid=8, entry_tid=10635]]], state=connected] 2024-09-11T06:20:17.995 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=44, tag_tid=8, entry_tid=9224], [object_number=47, tag_tid=8, entry_tid=9223], [object_number=46, tag_tid=8, entry_tid=9222], [object_number=45, tag_tid=8, entry_tid=9221]]], state=connected] 2024-09-11T06:20:17.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:17.999 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-09-11T06:20:18.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:20:18.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:18.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:18.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:18.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:20:18.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:18.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:18.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:18.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:20:18.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:20:18.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:20:18.002 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:20:18.002 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:18.002 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:20:18.002 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:18.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 11 2024-09-11T06:20:18.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 11 2024-09-11T06:20:18.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:20:18.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=10638], [object_number=47, tag_tid=8, entry_tid=10635]]], state=connected] 2024-09-11T06:20:18.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=44, tag_tid=8, entry_tid=9224], [object_number=47, tag_tid=8, entry_tid=9223], [object_number=46, tag_tid=8, entry_tid=9222], [object_number=45, tag_tid=8, entry_tid=9221]]], state=connected] 2024-09-11T06:20:18.051 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:18.053 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=44, tag_tid=8, entry_tid=9224' 2024-09-11T06:20:18.053 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=44, tag_tid=8, entry_tid=9224' 2024-09-11T06:20:18.053 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=44, tag_tid=8, entry_tid=9224' '!=' 'object_number=45, tag_tid=8, entry_tid=9105' 2024-09-11T06:20:18.053 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:20:18.053 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=44, tag_tid=8, entry_tid=9224' = 'object_number=45, tag_tid=8, entry_tid=9105' 2024-09-11T06:20:18.054 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=44, tag_tid=8, entry_tid=9224' 2024-09-11T06:20:18.054 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-09-11T06:20:18.054 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:20:18.054 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:18.054 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=9 2024-09-11T06:20:18.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=44, tag_tid=8, entry_tid=9224' 2024-09-11T06:20:18.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:20:18.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:18.056 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=8 2024-09-11T06:20:18.057 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-09-11T06:20:18.057 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:20:18.057 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:18.058 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:20:18.058 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=44, tag_tid=8, entry_tid=9224' 2024-09-11T06:20:18.058 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:20:18.058 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:18.060 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=9224 2024-09-11T06:20:18.060 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 9 = 8 -a 1 -le 9224 2024-09-11T06:20:18.060 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:20:18.060 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:20:18.060 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:20:18.261 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:20:18.262 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:20:18.262 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:20:18.262 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:20:18.262 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:20:18.262 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:20:18.262 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:20:18.262 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:20:18.262 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:20:18.262 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:20:18.262 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:20:18.262 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:20:18.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:20:18.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:20:18.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:20:18.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:20:18.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:20:18.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:20:18.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:20:18.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:20:18.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:20:18.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:20:18.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:20:18.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:20:18.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:20:18.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:20:18.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:20:18.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:20:18.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:20:18.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:20:18.269 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:20:18.269 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:20:18.269 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:20:18.269 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:20:18.269 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:20:18.269 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:20:18.269 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:20:18.269 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:20:18.269 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:20:18.269 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:20:18.269 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:20:18.269 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:20:18.270 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:20:18.270 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:20:18.270 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:20:18.270 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:20:18.270 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:20:18.270 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:20:18.270 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:20:18.291 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:18.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:18.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:18.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:20:18.425 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:20:18.425 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:20:18.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:18.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:18.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:18.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:20:18.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:18.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:18.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:18.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:20:18.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:20:18.427 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:20:18.427 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:20:18.428 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:18.428 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:20:18.429 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:18.479 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 11 2024-09-11T06:20:18.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 11 2024-09-11T06:20:18.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:20:18.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=10638], [object_number=47, tag_tid=8, entry_tid=10635]]], state=connected] 2024-09-11T06:20:18.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=47, tag_tid=8, entry_tid=9343], [object_number=46, tag_tid=8, entry_tid=9342], [object_number=45, tag_tid=8, entry_tid=9341], [object_number=44, tag_tid=8, entry_tid=9340]]], state=connected] 2024-09-11T06:20:18.483 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:18.484 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-09-11T06:20:18.486 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:20:18.486 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:18.486 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:18.486 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:18.486 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:20:18.486 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:18.486 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:18.486 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:18.486 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:20:18.486 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:20:18.486 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:20:18.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:20:18.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:18.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:20:18.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:18.534 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 11 2024-09-11T06:20:18.534 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 11 2024-09-11T06:20:18.534 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:20:18.534 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=10638], [object_number=47, tag_tid=8, entry_tid=10635]]], state=connected] 2024-09-11T06:20:18.534 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=47, tag_tid=8, entry_tid=9343], [object_number=46, tag_tid=8, entry_tid=9342], [object_number=45, tag_tid=8, entry_tid=9341], [object_number=44, tag_tid=8, entry_tid=9340]]], state=connected] 2024-09-11T06:20:18.537 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:18.539 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=47, tag_tid=8, entry_tid=9343' 2024-09-11T06:20:18.539 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=47, tag_tid=8, entry_tid=9343' 2024-09-11T06:20:18.539 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=47, tag_tid=8, entry_tid=9343' '!=' 'object_number=44, tag_tid=8, entry_tid=9224' 2024-09-11T06:20:18.539 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:20:18.540 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=47, tag_tid=8, entry_tid=9343' = 'object_number=44, tag_tid=8, entry_tid=9224' 2024-09-11T06:20:18.540 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=47, tag_tid=8, entry_tid=9343' 2024-09-11T06:20:18.540 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-09-11T06:20:18.540 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:20:18.540 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:18.540 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=9 2024-09-11T06:20:18.541 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=47, tag_tid=8, entry_tid=9343' 2024-09-11T06:20:18.541 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:20:18.541 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:18.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=8 2024-09-11T06:20:18.543 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-09-11T06:20:18.543 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:20:18.543 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:18.544 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:20:18.545 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=47, tag_tid=8, entry_tid=9343' 2024-09-11T06:20:18.545 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:20:18.545 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:18.546 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=9343 2024-09-11T06:20:18.546 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 9 = 8 -a 1 -le 9343 2024-09-11T06:20:18.546 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:20:18.546 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:20:18.546 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:20:18.748 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:20:18.748 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:20:18.748 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:20:18.748 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:20:18.748 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:20:18.748 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:20:18.748 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:20:18.749 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:20:18.749 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:20:18.749 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:20:18.749 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:20:18.749 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:20:18.749 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:20:18.749 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:20:18.749 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:20:18.750 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:20:18.750 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:20:18.753 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:20:18.753 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:20:18.753 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:20:18.753 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:20:18.754 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:20:18.754 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:20:18.754 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:20:18.754 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:20:18.754 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:20:18.754 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:20:18.754 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:20:18.754 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:20:18.754 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:20:18.754 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:20:18.754 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:20:18.754 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:20:18.754 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:20:18.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:20:18.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:20:18.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:20:18.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:20:18.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:20:18.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:20:18.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:20:18.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:20:18.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:20:18.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:20:18.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:20:18.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:20:18.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:20:18.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:20:18.756 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:20:18.779 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:18.779 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:18.780 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:18.780 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:20:18.906 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:20:18.907 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:20:18.907 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:18.907 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:18.907 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:18.907 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:20:18.908 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:18.908 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:18.908 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:18.908 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:20:18.908 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:20:18.908 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:20:18.909 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:20:18.909 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:18.910 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:20:18.910 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:18.966 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 11 2024-09-11T06:20:18.966 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 11 2024-09-11T06:20:18.966 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:20:18.966 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=10638], [object_number=47, tag_tid=8, entry_tid=10635]]], state=connected] 2024-09-11T06:20:18.966 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=47, tag_tid=8, entry_tid=9467], [object_number=46, tag_tid=8, entry_tid=9466], [object_number=45, tag_tid=8, entry_tid=9465], [object_number=44, tag_tid=8, entry_tid=9464]]], state=connected] 2024-09-11T06:20:18.969 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:18.970 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-09-11T06:20:18.971 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:20:18.971 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:18.971 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:18.971 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:18.972 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:20:18.972 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:18.972 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:18.972 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:18.972 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:20:18.972 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:20:18.972 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:20:18.973 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:20:18.973 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:18.973 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:20:18.973 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:19.021 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 11 2024-09-11T06:20:19.021 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 11 2024-09-11T06:20:19.021 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:20:19.022 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=10638], [object_number=47, tag_tid=8, entry_tid=10635]]], state=connected] 2024-09-11T06:20:19.022 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=47, tag_tid=8, entry_tid=9467], [object_number=46, tag_tid=8, entry_tid=9466], [object_number=45, tag_tid=8, entry_tid=9465], [object_number=44, tag_tid=8, entry_tid=9464]]], state=connected] 2024-09-11T06:20:19.024 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:19.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=47, tag_tid=8, entry_tid=9467' 2024-09-11T06:20:19.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=47, tag_tid=8, entry_tid=9467' 2024-09-11T06:20:19.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=47, tag_tid=8, entry_tid=9467' '!=' 'object_number=47, tag_tid=8, entry_tid=9343' 2024-09-11T06:20:19.028 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:20:19.028 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=47, tag_tid=8, entry_tid=9467' = 'object_number=47, tag_tid=8, entry_tid=9343' 2024-09-11T06:20:19.028 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=47, tag_tid=8, entry_tid=9467' 2024-09-11T06:20:19.028 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-09-11T06:20:19.028 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:20:19.028 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:19.030 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=9 2024-09-11T06:20:19.030 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=47, tag_tid=8, entry_tid=9467' 2024-09-11T06:20:19.031 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:20:19.031 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:19.031 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=8 2024-09-11T06:20:19.031 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-09-11T06:20:19.031 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:20:19.031 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:19.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:20:19.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=47, tag_tid=8, entry_tid=9467' 2024-09-11T06:20:19.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:20:19.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:19.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=9467 2024-09-11T06:20:19.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 9 = 8 -a 1 -le 9467 2024-09-11T06:20:19.035 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:20:19.035 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:20:19.035 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:20:19.235 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:20:19.235 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:20:19.235 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:20:19.235 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:20:19.235 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:20:19.235 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:20:19.235 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:20:19.235 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:20:19.236 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:20:19.236 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:20:19.236 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:20:19.236 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:20:19.236 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:20:19.236 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:20:19.236 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:20:19.236 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:20:19.236 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:20:19.238 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:20:19.238 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:20:19.238 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:20:19.238 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:20:19.238 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:20:19.238 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:20:19.238 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:20:19.238 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:20:19.239 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:20:19.239 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:20:19.239 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:20:19.239 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:20:19.239 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:20:19.239 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:20:19.239 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:20:19.239 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:20:19.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:20:19.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:20:19.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:20:19.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:20:19.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:20:19.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:20:19.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:20:19.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:20:19.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:20:19.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:20:19.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:20:19.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:20:19.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:20:19.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:20:19.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:20:19.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:20:19.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:19.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:19.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:19.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:20:19.397 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:20:19.398 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:20:19.398 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:19.398 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:19.398 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:19.398 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:20:19.398 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:19.398 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:19.398 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:19.398 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:20:19.399 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:20:19.399 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:20:19.399 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:20:19.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:19.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:20:19.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:19.458 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 11 2024-09-11T06:20:19.458 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 11 2024-09-11T06:20:19.458 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:20:19.458 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=10638], [object_number=47, tag_tid=8, entry_tid=10635]]], state=connected] 2024-09-11T06:20:19.458 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=44, tag_tid=8, entry_tid=9576], [object_number=47, tag_tid=8, entry_tid=9575], [object_number=46, tag_tid=8, entry_tid=9574], [object_number=45, tag_tid=8, entry_tid=9573]]], state=connected] 2024-09-11T06:20:19.462 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:19.463 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-09-11T06:20:19.464 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:20:19.464 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:19.464 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:19.464 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:19.465 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:20:19.465 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:19.465 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:19.465 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:19.465 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:20:19.465 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:20:19.465 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:20:19.465 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:20:19.466 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:19.466 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:20:19.466 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:19.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 11 2024-09-11T06:20:19.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 11 2024-09-11T06:20:19.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:20:19.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=10638], [object_number=47, tag_tid=8, entry_tid=10635]]], state=connected] 2024-09-11T06:20:19.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=44, tag_tid=8, entry_tid=9576], [object_number=47, tag_tid=8, entry_tid=9575], [object_number=46, tag_tid=8, entry_tid=9574], [object_number=45, tag_tid=8, entry_tid=9573]]], state=connected] 2024-09-11T06:20:19.515 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:19.517 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=44, tag_tid=8, entry_tid=9576' 2024-09-11T06:20:19.517 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=44, tag_tid=8, entry_tid=9576' 2024-09-11T06:20:19.517 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=44, tag_tid=8, entry_tid=9576' '!=' 'object_number=47, tag_tid=8, entry_tid=9467' 2024-09-11T06:20:19.518 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:20:19.518 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=44, tag_tid=8, entry_tid=9576' = 'object_number=47, tag_tid=8, entry_tid=9467' 2024-09-11T06:20:19.518 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=44, tag_tid=8, entry_tid=9576' 2024-09-11T06:20:19.518 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-09-11T06:20:19.518 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:20:19.518 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:19.518 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=9 2024-09-11T06:20:19.519 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=44, tag_tid=8, entry_tid=9576' 2024-09-11T06:20:19.519 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:20:19.519 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:19.520 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=8 2024-09-11T06:20:19.521 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-09-11T06:20:19.521 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:20:19.521 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:19.522 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:20:19.522 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=44, tag_tid=8, entry_tid=9576' 2024-09-11T06:20:19.523 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:20:19.523 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:19.524 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=9576 2024-09-11T06:20:19.524 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 9 = 8 -a 1 -le 9576 2024-09-11T06:20:19.524 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:20:19.524 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:20:19.524 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:20:19.726 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:20:19.726 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:20:19.726 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:20:19.727 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:20:19.727 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:20:19.727 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:20:19.727 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:20:19.727 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:20:19.727 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:20:19.727 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:20:19.727 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:20:19.727 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:20:19.727 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:20:19.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:20:19.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:20:19.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:20:19.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:20:19.731 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:20:19.731 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:20:19.731 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:20:19.731 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:20:19.731 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:20:19.731 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:20:19.731 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:20:19.731 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:20:19.731 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:20:19.731 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:20:19.732 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:20:19.732 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:20:19.732 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:20:19.732 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:20:19.732 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:20:19.732 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:20:19.732 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:20:19.732 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:20:19.732 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:20:19.732 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:20:19.732 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:20:19.732 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:20:19.732 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:20:19.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:20:19.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:20:19.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:20:19.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:20:19.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:20:19.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:20:19.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:20:19.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:20:19.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:20:19.756 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:19.756 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:19.757 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:19.757 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:20:19.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:20:19.889 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:20:19.889 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:19.889 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:19.889 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:19.889 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:20:19.889 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:19.889 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:19.889 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:19.889 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:20:19.890 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:20:19.890 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:20:19.890 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:20:19.891 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:19.892 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:20:19.892 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:19.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 11 2024-09-11T06:20:19.946 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 11 2024-09-11T06:20:19.946 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:20:19.946 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=10638], [object_number=47, tag_tid=8, entry_tid=10635]]], state=connected] 2024-09-11T06:20:19.946 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=47, tag_tid=8, entry_tid=9687], [object_number=46, tag_tid=8, entry_tid=9686], [object_number=45, tag_tid=8, entry_tid=9685], [object_number=44, tag_tid=8, entry_tid=9684]]], state=connected] 2024-09-11T06:20:19.949 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:19.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-09-11T06:20:19.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:20:19.952 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:19.952 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:19.952 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:19.952 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:20:19.952 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:19.952 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:19.952 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:19.952 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:20:19.952 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:20:19.952 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:20:19.952 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:20:19.953 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:19.953 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:20:19.953 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:19.999 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 11 2024-09-11T06:20:19.999 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 11 2024-09-11T06:20:20.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:20:20.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=10638], [object_number=47, tag_tid=8, entry_tid=10635]]], state=connected] 2024-09-11T06:20:20.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=47, tag_tid=8, entry_tid=9687], [object_number=46, tag_tid=8, entry_tid=9686], [object_number=45, tag_tid=8, entry_tid=9685], [object_number=44, tag_tid=8, entry_tid=9684]]], state=connected] 2024-09-11T06:20:20.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:20.007 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=47, tag_tid=8, entry_tid=9687' 2024-09-11T06:20:20.008 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=47, tag_tid=8, entry_tid=9687' 2024-09-11T06:20:20.008 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=47, tag_tid=8, entry_tid=9687' '!=' 'object_number=44, tag_tid=8, entry_tid=9576' 2024-09-11T06:20:20.008 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:20:20.008 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=47, tag_tid=8, entry_tid=9687' = 'object_number=44, tag_tid=8, entry_tid=9576' 2024-09-11T06:20:20.008 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=47, tag_tid=8, entry_tid=9687' 2024-09-11T06:20:20.008 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-09-11T06:20:20.009 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:20:20.009 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:20.009 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=9 2024-09-11T06:20:20.010 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=47, tag_tid=8, entry_tid=9687' 2024-09-11T06:20:20.010 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:20:20.010 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:20.011 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=8 2024-09-11T06:20:20.011 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-09-11T06:20:20.011 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:20:20.012 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:20.012 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:20:20.013 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=47, tag_tid=8, entry_tid=9687' 2024-09-11T06:20:20.013 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:20:20.013 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:20.014 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=9687 2024-09-11T06:20:20.014 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 9 = 8 -a 1 -le 9687 2024-09-11T06:20:20.015 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:20:20.015 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:20:20.015 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:20:20.216 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:20:20.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:20:20.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:20:20.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:20:20.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:20:20.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:20:20.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:20:20.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:20:20.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:20:20.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:20:20.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:20:20.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:20:20.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:20:20.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:20:20.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:20:20.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:20:20.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:20:20.222 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:20:20.222 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:20:20.222 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:20:20.222 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:20:20.222 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:20:20.222 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:20:20.222 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:20:20.222 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:20:20.222 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:20:20.223 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:20:20.223 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:20:20.223 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:20:20.223 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:20:20.223 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:20:20.223 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:20:20.223 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:20:20.223 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:20:20.223 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:20:20.223 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:20:20.223 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:20:20.223 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:20:20.223 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:20:20.224 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:20:20.224 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:20:20.224 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:20:20.224 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:20:20.224 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:20:20.224 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:20:20.224 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:20:20.224 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:20:20.224 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:20:20.224 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:20:20.248 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:20.248 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:20.248 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:20.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:20:20.379 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:20:20.380 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:20:20.380 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:20.380 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:20.380 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:20.380 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:20:20.380 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:20.381 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:20.381 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:20.381 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:20:20.381 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:20:20.381 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:20:20.381 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:20:20.383 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:20.383 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:20.384 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:20:20.436 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 11 2024-09-11T06:20:20.437 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 11 2024-09-11T06:20:20.437 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:20:20.437 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=10638], [object_number=47, tag_tid=8, entry_tid=10635]]], state=connected] 2024-09-11T06:20:20.437 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=47, tag_tid=8, entry_tid=9803], [object_number=46, tag_tid=8, entry_tid=9802], [object_number=45, tag_tid=8, entry_tid=9801], [object_number=44, tag_tid=8, entry_tid=9800]]], state=connected] 2024-09-11T06:20:20.440 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:20.441 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-09-11T06:20:20.442 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:20:20.443 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:20.443 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:20.443 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:20.443 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:20:20.443 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:20.443 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:20.443 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:20.443 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:20:20.443 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:20:20.443 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:20:20.443 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:20:20.444 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:20.444 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:20:20.444 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:20.490 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 11 2024-09-11T06:20:20.490 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 11 2024-09-11T06:20:20.490 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:20:20.490 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=10638], [object_number=47, tag_tid=8, entry_tid=10635]]], state=connected] 2024-09-11T06:20:20.490 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=47, tag_tid=8, entry_tid=9803], [object_number=46, tag_tid=8, entry_tid=9802], [object_number=45, tag_tid=8, entry_tid=9801], [object_number=44, tag_tid=8, entry_tid=9800]]], state=connected] 2024-09-11T06:20:20.493 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:20.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=47, tag_tid=8, entry_tid=9803' 2024-09-11T06:20:20.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=47, tag_tid=8, entry_tid=9803' 2024-09-11T06:20:20.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=47, tag_tid=8, entry_tid=9803' '!=' 'object_number=47, tag_tid=8, entry_tid=9687' 2024-09-11T06:20:20.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:20:20.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=47, tag_tid=8, entry_tid=9803' = 'object_number=47, tag_tid=8, entry_tid=9687' 2024-09-11T06:20:20.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=47, tag_tid=8, entry_tid=9803' 2024-09-11T06:20:20.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-09-11T06:20:20.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:20:20.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:20.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=9 2024-09-11T06:20:20.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=47, tag_tid=8, entry_tid=9803' 2024-09-11T06:20:20.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:20:20.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:20.498 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=8 2024-09-11T06:20:20.498 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-09-11T06:20:20.498 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:20:20.499 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:20.499 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:20:20.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=47, tag_tid=8, entry_tid=9803' 2024-09-11T06:20:20.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:20:20.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:20.501 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=9803 2024-09-11T06:20:20.501 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 9 = 8 -a 1 -le 9803 2024-09-11T06:20:20.501 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:20:20.502 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:20:20.502 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:20:20.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:20:20.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:20:20.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:20:20.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:20:20.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:20:20.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:20:20.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:20:20.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:20:20.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:20:20.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:20:20.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:20:20.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:20:20.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:20:20.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:20:20.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:20:20.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:20:20.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:20:20.708 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:20:20.708 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:20:20.708 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:20:20.708 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:20:20.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:20:20.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:20:20.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:20:20.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:20:20.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:20:20.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:20:20.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:20:20.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:20:20.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:20:20.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:20:20.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:20:20.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:20:20.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:20:20.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:20:20.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:20:20.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:20:20.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:20:20.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:20:20.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:20:20.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:20:20.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:20:20.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:20:20.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:20:20.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:20:20.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:20:20.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:20:20.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:20:20.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:20:20.734 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:20.734 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:20.734 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:20.734 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:20:20.863 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:20:20.863 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:20:20.864 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:20.864 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:20.864 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:20.864 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:20:20.864 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:20.864 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:20.864 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:20.864 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:20:20.865 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:20:20.865 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:20:20.865 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:20:20.867 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:20.868 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:20:20.868 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:20.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 11 2024-09-11T06:20:20.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 11 2024-09-11T06:20:20.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:20:20.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=10638], [object_number=47, tag_tid=8, entry_tid=10635]]], state=connected] 2024-09-11T06:20:20.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=46, tag_tid=8, entry_tid=9910], [object_number=45, tag_tid=8, entry_tid=9909], [object_number=44, tag_tid=8, entry_tid=9908], [object_number=47, tag_tid=8, entry_tid=9907]]], state=connected] 2024-09-11T06:20:20.922 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:20.923 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-09-11T06:20:20.924 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:20:20.924 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:20.925 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:20.925 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:20.925 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:20:20.925 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:20.925 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:20.925 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:20.925 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:20:20.925 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:20:20.925 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:20:20.925 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:20:20.926 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:20.926 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:20:20.926 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:20.971 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 11 2024-09-11T06:20:20.971 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 11 2024-09-11T06:20:20.971 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:20:20.971 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=10638], [object_number=47, tag_tid=8, entry_tid=10635]]], state=connected] 2024-09-11T06:20:20.971 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=46, tag_tid=8, entry_tid=9910], [object_number=45, tag_tid=8, entry_tid=9909], [object_number=44, tag_tid=8, entry_tid=9908], [object_number=47, tag_tid=8, entry_tid=9907]]], state=connected] 2024-09-11T06:20:20.974 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:20.976 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=46, tag_tid=8, entry_tid=9910' 2024-09-11T06:20:20.976 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=46, tag_tid=8, entry_tid=9910' 2024-09-11T06:20:20.976 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=46, tag_tid=8, entry_tid=9910' '!=' 'object_number=47, tag_tid=8, entry_tid=9803' 2024-09-11T06:20:20.976 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:20:20.976 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=46, tag_tid=8, entry_tid=9910' = 'object_number=47, tag_tid=8, entry_tid=9803' 2024-09-11T06:20:20.977 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=46, tag_tid=8, entry_tid=9910' 2024-09-11T06:20:20.977 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-09-11T06:20:20.977 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:20:20.977 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:20.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=9 2024-09-11T06:20:20.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=46, tag_tid=8, entry_tid=9910' 2024-09-11T06:20:20.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:20:20.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:20.979 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=8 2024-09-11T06:20:20.980 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-09-11T06:20:20.980 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:20:20.980 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:20.981 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:20:20.982 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=46, tag_tid=8, entry_tid=9910' 2024-09-11T06:20:20.982 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:20:20.982 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:20.983 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=9910 2024-09-11T06:20:20.983 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 9 = 8 -a 1 -le 9910 2024-09-11T06:20:20.983 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:20:20.983 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:20:20.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:20:21.185 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:20:21.186 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:20:21.186 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:20:21.186 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:20:21.186 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:20:21.186 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:20:21.186 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:20:21.186 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:20:21.186 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:20:21.186 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:20:21.186 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:20:21.186 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:20:21.187 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:20:21.187 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:20:21.187 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:20:21.187 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:20:21.187 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:20:21.190 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:20:21.190 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:20:21.190 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:20:21.190 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:20:21.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:20:21.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:20:21.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:20:21.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:20:21.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:20:21.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:20:21.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:20:21.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:20:21.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:20:21.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:20:21.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:20:21.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:20:21.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:20:21.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:20:21.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:20:21.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:20:21.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:20:21.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:20:21.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:20:21.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:20:21.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:20:21.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:20:21.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:20:21.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:20:21.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:20:21.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:20:21.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:20:21.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:20:21.216 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:21.216 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:21.216 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:21.216 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:20:21.343 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:20:21.344 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:20:21.344 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:21.344 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:21.344 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:21.344 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:20:21.345 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:21.345 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:21.345 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:21.345 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:20:21.345 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:20:21.345 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:20:21.345 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:20:21.347 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:21.348 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:20:21.348 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:21.399 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 11 2024-09-11T06:20:21.399 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 11 2024-09-11T06:20:21.399 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:20:21.399 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=10638], [object_number=47, tag_tid=8, entry_tid=10635]]], state=connected] 2024-09-11T06:20:21.399 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=47, tag_tid=8, entry_tid=10007], [object_number=46, tag_tid=8, entry_tid=10006], [object_number=45, tag_tid=8, entry_tid=10005], [object_number=44, tag_tid=8, entry_tid=10004]]], state=connected] 2024-09-11T06:20:21.403 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:21.404 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-09-11T06:20:21.405 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:20:21.405 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:21.405 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:21.405 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:21.406 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:20:21.406 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:21.406 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:21.406 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:21.406 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:20:21.406 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:20:21.406 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:20:21.406 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:20:21.407 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:21.407 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:20:21.407 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:21.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 11 2024-09-11T06:20:21.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 11 2024-09-11T06:20:21.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:20:21.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=10638], [object_number=47, tag_tid=8, entry_tid=10635]]], state=connected] 2024-09-11T06:20:21.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=47, tag_tid=8, entry_tid=10007], [object_number=46, tag_tid=8, entry_tid=10006], [object_number=45, tag_tid=8, entry_tid=10005], [object_number=44, tag_tid=8, entry_tid=10004]]], state=connected] 2024-09-11T06:20:21.456 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:21.458 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=47, tag_tid=8, entry_tid=10007' 2024-09-11T06:20:21.458 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=47, tag_tid=8, entry_tid=10007' 2024-09-11T06:20:21.458 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=47, tag_tid=8, entry_tid=10007' '!=' 'object_number=46, tag_tid=8, entry_tid=9910' 2024-09-11T06:20:21.458 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:20:21.459 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=47, tag_tid=8, entry_tid=10007' = 'object_number=46, tag_tid=8, entry_tid=9910' 2024-09-11T06:20:21.459 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=47, tag_tid=8, entry_tid=10007' 2024-09-11T06:20:21.459 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-09-11T06:20:21.459 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:20:21.459 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:21.459 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=9 2024-09-11T06:20:21.460 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=47, tag_tid=8, entry_tid=10007' 2024-09-11T06:20:21.460 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:20:21.460 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:21.461 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=8 2024-09-11T06:20:21.462 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-09-11T06:20:21.462 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:20:21.462 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:21.463 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:20:21.463 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=47, tag_tid=8, entry_tid=10007' 2024-09-11T06:20:21.463 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:20:21.464 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:21.465 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=10007 2024-09-11T06:20:21.465 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 9 = 8 -a 1 -le 10007 2024-09-11T06:20:21.465 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:20:21.465 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:20:21.465 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:20:21.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:20:21.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:20:21.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:20:21.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:20:21.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:20:21.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:20:21.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:20:21.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:20:21.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:20:21.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:20:21.668 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:20:21.668 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:20:21.668 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:20:21.668 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:20:21.668 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:20:21.668 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:20:21.668 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:20:21.672 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:20:21.672 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:20:21.672 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:20:21.672 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:20:21.672 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:20:21.672 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:20:21.672 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:20:21.672 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:20:21.672 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:20:21.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:20:21.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:20:21.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:20:21.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:20:21.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:20:21.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:20:21.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:20:21.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:20:21.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:20:21.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:20:21.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:20:21.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:20:21.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:20:21.674 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:20:21.674 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:20:21.674 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:20:21.674 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:20:21.674 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:20:21.674 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:20:21.674 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:20:21.674 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:20:21.674 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:20:21.674 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:20:21.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:21.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:21.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:21.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:20:21.830 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:20:21.831 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:20:21.831 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:21.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:21.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:21.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:20:21.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:21.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:21.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:21.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:20:21.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:20:21.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:20:21.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:20:21.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:21.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:20:21.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:21.884 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 11 2024-09-11T06:20:21.884 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 11 2024-09-11T06:20:21.884 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:20:21.885 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=10638], [object_number=47, tag_tid=8, entry_tid=10635]]], state=connected] 2024-09-11T06:20:21.885 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=47, tag_tid=8, entry_tid=10107], [object_number=46, tag_tid=8, entry_tid=10106], [object_number=45, tag_tid=8, entry_tid=10105], [object_number=44, tag_tid=8, entry_tid=10104]]], state=connected] 2024-09-11T06:20:21.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:21.889 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-09-11T06:20:21.890 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:20:21.891 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:21.891 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:21.891 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:21.891 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:20:21.891 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:21.891 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:21.891 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:21.891 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:20:21.891 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:20:21.891 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:20:21.891 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:20:21.892 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:21.892 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:20:21.892 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:21.939 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 11 2024-09-11T06:20:21.939 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 11 2024-09-11T06:20:21.939 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:20:21.939 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=10638], [object_number=47, tag_tid=8, entry_tid=10635]]], state=connected] 2024-09-11T06:20:21.939 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=47, tag_tid=8, entry_tid=10107], [object_number=46, tag_tid=8, entry_tid=10106], [object_number=45, tag_tid=8, entry_tid=10105], [object_number=44, tag_tid=8, entry_tid=10104]]], state=connected] 2024-09-11T06:20:21.942 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:21.944 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=47, tag_tid=8, entry_tid=10107' 2024-09-11T06:20:21.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=47, tag_tid=8, entry_tid=10107' 2024-09-11T06:20:21.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=47, tag_tid=8, entry_tid=10107' '!=' 'object_number=47, tag_tid=8, entry_tid=10007' 2024-09-11T06:20:21.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:20:21.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=47, tag_tid=8, entry_tid=10107' = 'object_number=47, tag_tid=8, entry_tid=10007' 2024-09-11T06:20:21.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=47, tag_tid=8, entry_tid=10107' 2024-09-11T06:20:21.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-09-11T06:20:21.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:20:21.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:21.946 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=9 2024-09-11T06:20:21.946 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=47, tag_tid=8, entry_tid=10107' 2024-09-11T06:20:21.946 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:20:21.946 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:21.947 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=8 2024-09-11T06:20:21.948 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-09-11T06:20:21.948 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:20:21.948 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:21.949 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:20:21.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=47, tag_tid=8, entry_tid=10107' 2024-09-11T06:20:21.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:20:21.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:21.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=10107 2024-09-11T06:20:21.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 9 = 8 -a 1 -le 10107 2024-09-11T06:20:21.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:20:21.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:20:21.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:20:22.153 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:20:22.153 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:20:22.153 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:20:22.153 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:20:22.153 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:20:22.153 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:20:22.153 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:20:22.154 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:20:22.154 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:20:22.154 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:20:22.154 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:20:22.154 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:20:22.154 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:20:22.154 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:20:22.154 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:20:22.154 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:20:22.155 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:20:22.157 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:20:22.157 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:20:22.157 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:20:22.157 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:20:22.157 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:20:22.157 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:20:22.157 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:20:22.157 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:20:22.157 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:20:22.157 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:20:22.157 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:20:22.157 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:20:22.157 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:20:22.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:20:22.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:20:22.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:20:22.159 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:20:22.159 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:20:22.159 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:20:22.159 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:20:22.159 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:20:22.159 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:20:22.159 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:20:22.159 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:20:22.160 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:20:22.160 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:20:22.160 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:20:22.160 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:20:22.160 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:20:22.160 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:20:22.160 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:20:22.160 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:20:22.184 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:22.184 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:22.184 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:22.184 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:20:22.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:20:22.317 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:20:22.317 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:22.317 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:22.317 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:22.317 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:20:22.317 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:22.317 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:22.318 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:22.318 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:20:22.318 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:20:22.318 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:20:22.318 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:20:22.320 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:22.321 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:20:22.321 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:22.368 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 11 2024-09-11T06:20:22.368 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 11 2024-09-11T06:20:22.368 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:20:22.368 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=10638], [object_number=47, tag_tid=8, entry_tid=10635]]], state=connected] 2024-09-11T06:20:22.368 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=47, tag_tid=8, entry_tid=10211], [object_number=46, tag_tid=8, entry_tid=10210], [object_number=45, tag_tid=8, entry_tid=10209], [object_number=44, tag_tid=8, entry_tid=10208]]], state=connected] 2024-09-11T06:20:22.372 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:22.373 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-09-11T06:20:22.374 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:20:22.374 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:22.374 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:22.374 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:22.374 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:20:22.375 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:22.375 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:22.375 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:22.375 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:20:22.375 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:20:22.375 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:20:22.375 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:20:22.376 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:22.376 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:20:22.376 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:22.421 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 11 2024-09-11T06:20:22.421 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 11 2024-09-11T06:20:22.421 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:20:22.421 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=10638], [object_number=47, tag_tid=8, entry_tid=10635]]], state=connected] 2024-09-11T06:20:22.421 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=47, tag_tid=8, entry_tid=10211], [object_number=46, tag_tid=8, entry_tid=10210], [object_number=45, tag_tid=8, entry_tid=10209], [object_number=44, tag_tid=8, entry_tid=10208]]], state=connected] 2024-09-11T06:20:22.424 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:22.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=47, tag_tid=8, entry_tid=10211' 2024-09-11T06:20:22.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=47, tag_tid=8, entry_tid=10211' 2024-09-11T06:20:22.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=47, tag_tid=8, entry_tid=10211' '!=' 'object_number=47, tag_tid=8, entry_tid=10107' 2024-09-11T06:20:22.427 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:20:22.427 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=47, tag_tid=8, entry_tid=10211' = 'object_number=47, tag_tid=8, entry_tid=10107' 2024-09-11T06:20:22.427 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=47, tag_tid=8, entry_tid=10211' 2024-09-11T06:20:22.427 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-09-11T06:20:22.427 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:20:22.427 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:22.427 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=9 2024-09-11T06:20:22.428 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=47, tag_tid=8, entry_tid=10211' 2024-09-11T06:20:22.428 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:20:22.428 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:22.429 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=8 2024-09-11T06:20:22.430 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-09-11T06:20:22.430 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:20:22.430 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:22.431 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:20:22.431 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=47, tag_tid=8, entry_tid=10211' 2024-09-11T06:20:22.432 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:20:22.432 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:22.433 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=10211 2024-09-11T06:20:22.433 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 9 = 8 -a 1 -le 10211 2024-09-11T06:20:22.433 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:20:22.433 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:20:22.433 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:20:22.635 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:20:22.635 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:20:22.635 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:20:22.635 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:20:22.635 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:20:22.636 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:20:22.636 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:20:22.636 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:20:22.636 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:20:22.636 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:20:22.636 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:20:22.636 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:20:22.636 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:20:22.636 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:20:22.636 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:20:22.637 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:20:22.637 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:20:22.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:20:22.640 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:20:22.640 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:20:22.640 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:20:22.640 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:20:22.640 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:20:22.640 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:20:22.640 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:20:22.640 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:20:22.640 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:20:22.640 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:20:22.640 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:20:22.640 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:20:22.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:20:22.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:20:22.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:20:22.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:20:22.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:20:22.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:20:22.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:20:22.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:20:22.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:20:22.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:20:22.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:20:22.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:20:22.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:20:22.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:20:22.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:20:22.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:20:22.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:20:22.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:20:22.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:20:22.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:22.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:22.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:22.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:20:22.795 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:20:22.797 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:20:22.797 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:22.797 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:22.797 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:22.797 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:20:22.797 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:22.797 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:22.797 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:22.797 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:20:22.797 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:20:22.798 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:20:22.798 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:20:22.800 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:22.800 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:20:22.800 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:22.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 11 2024-09-11T06:20:22.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 11 2024-09-11T06:20:22.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:20:22.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=10638], [object_number=47, tag_tid=8, entry_tid=10635]]], state=connected] 2024-09-11T06:20:22.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=46, tag_tid=8, entry_tid=10310], [object_number=45, tag_tid=8, entry_tid=10309], [object_number=44, tag_tid=8, entry_tid=10308], [object_number=47, tag_tid=8, entry_tid=10307]]], state=connected] 2024-09-11T06:20:22.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:22.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-09-11T06:20:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:20:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:20:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:20:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:20:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:20:22.854 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:20:22.854 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:22.854 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:20:22.854 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:22.899 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 11 2024-09-11T06:20:22.899 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 11 2024-09-11T06:20:22.900 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:20:22.900 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=10638], [object_number=47, tag_tid=8, entry_tid=10635]]], state=connected] 2024-09-11T06:20:22.900 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=46, tag_tid=8, entry_tid=10310], [object_number=45, tag_tid=8, entry_tid=10309], [object_number=44, tag_tid=8, entry_tid=10308], [object_number=47, tag_tid=8, entry_tid=10307]]], state=connected] 2024-09-11T06:20:22.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:22.905 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=46, tag_tid=8, entry_tid=10310' 2024-09-11T06:20:22.905 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=46, tag_tid=8, entry_tid=10310' 2024-09-11T06:20:22.905 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=46, tag_tid=8, entry_tid=10310' '!=' 'object_number=47, tag_tid=8, entry_tid=10211' 2024-09-11T06:20:22.905 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:20:22.905 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=46, tag_tid=8, entry_tid=10310' = 'object_number=47, tag_tid=8, entry_tid=10211' 2024-09-11T06:20:22.905 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=46, tag_tid=8, entry_tid=10310' 2024-09-11T06:20:22.906 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-09-11T06:20:22.906 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:20:22.906 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:22.906 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=9 2024-09-11T06:20:22.907 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=46, tag_tid=8, entry_tid=10310' 2024-09-11T06:20:22.907 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:20:22.907 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:22.908 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=8 2024-09-11T06:20:22.909 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-09-11T06:20:22.909 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:20:22.909 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:22.909 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:20:22.910 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=46, tag_tid=8, entry_tid=10310' 2024-09-11T06:20:22.910 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:20:22.910 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:22.911 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=10310 2024-09-11T06:20:22.912 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 9 = 8 -a 1 -le 10310 2024-09-11T06:20:22.912 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:20:22.912 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:20:22.912 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:20:23.113 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:20:23.113 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:20:23.113 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:20:23.114 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:20:23.114 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:20:23.114 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:20:23.114 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:20:23.114 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:20:23.114 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:20:23.114 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:20:23.114 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:20:23.114 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:20:23.114 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:20:23.114 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:20:23.114 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:20:23.115 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:20:23.115 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:20:23.117 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:20:23.117 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:20:23.117 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:20:23.117 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:20:23.117 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:20:23.117 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:20:23.117 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:20:23.118 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:20:23.118 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:20:23.118 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:20:23.118 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:20:23.118 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:20:23.118 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:20:23.118 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:20:23.118 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:20:23.118 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:20:23.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:20:23.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:20:23.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:20:23.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:20:23.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:20:23.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:20:23.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:20:23.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:20:23.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:20:23.120 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:20:23.120 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:20:23.120 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:20:23.120 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:20:23.120 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:20:23.120 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:20:23.120 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:20:23.143 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:23.144 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:23.144 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:23.144 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:20:23.275 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:20:23.277 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:20:23.277 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:23.277 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:23.277 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:23.277 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:20:23.277 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:23.277 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:23.277 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:23.277 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:20:23.278 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:20:23.278 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:20:23.278 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:20:23.280 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:23.280 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:20:23.280 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:23.326 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 11 2024-09-11T06:20:23.327 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 11 2024-09-11T06:20:23.327 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:20:23.327 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=10638], [object_number=47, tag_tid=8, entry_tid=10635]]], state=connected] 2024-09-11T06:20:23.327 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=44, tag_tid=8, entry_tid=10396], [object_number=47, tag_tid=8, entry_tid=10395], [object_number=46, tag_tid=8, entry_tid=10394], [object_number=45, tag_tid=8, entry_tid=10393]]], state=connected] 2024-09-11T06:20:23.330 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:23.331 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-09-11T06:20:23.332 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:20:23.332 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:23.332 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:23.333 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:23.333 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:20:23.333 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:23.333 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:23.333 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:23.333 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:20:23.333 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:20:23.333 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:20:23.333 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:20:23.334 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:23.334 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:20:23.334 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:23.379 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 11 2024-09-11T06:20:23.379 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 11 2024-09-11T06:20:23.379 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:20:23.379 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=10638], [object_number=47, tag_tid=8, entry_tid=10635]]], state=connected] 2024-09-11T06:20:23.379 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=44, tag_tid=8, entry_tid=10396], [object_number=47, tag_tid=8, entry_tid=10395], [object_number=46, tag_tid=8, entry_tid=10394], [object_number=45, tag_tid=8, entry_tid=10393]]], state=connected] 2024-09-11T06:20:23.382 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:23.385 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=44, tag_tid=8, entry_tid=10396' 2024-09-11T06:20:23.385 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=44, tag_tid=8, entry_tid=10396' 2024-09-11T06:20:23.385 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=44, tag_tid=8, entry_tid=10396' '!=' 'object_number=46, tag_tid=8, entry_tid=10310' 2024-09-11T06:20:23.385 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:20:23.385 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=44, tag_tid=8, entry_tid=10396' = 'object_number=46, tag_tid=8, entry_tid=10310' 2024-09-11T06:20:23.385 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=44, tag_tid=8, entry_tid=10396' 2024-09-11T06:20:23.385 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-09-11T06:20:23.385 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:20:23.385 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:23.386 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=9 2024-09-11T06:20:23.386 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=44, tag_tid=8, entry_tid=10396' 2024-09-11T06:20:23.386 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:20:23.386 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:23.388 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=8 2024-09-11T06:20:23.388 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-09-11T06:20:23.388 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:20:23.388 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:23.389 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:20:23.390 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=44, tag_tid=8, entry_tid=10396' 2024-09-11T06:20:23.390 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:20:23.390 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:23.391 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=10396 2024-09-11T06:20:23.391 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 9 = 8 -a 1 -le 10396 2024-09-11T06:20:23.391 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:20:23.391 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:20:23.391 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:20:23.593 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:20:23.593 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:20:23.593 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:20:23.593 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:20:23.593 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:20:23.594 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:20:23.594 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:20:23.594 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:20:23.594 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:20:23.594 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:20:23.594 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:20:23.594 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:20:23.594 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:20:23.594 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:20:23.594 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:20:23.594 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:20:23.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:20:23.597 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:20:23.597 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:20:23.597 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:20:23.597 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:20:23.597 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:20:23.597 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:20:23.597 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:20:23.597 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:20:23.598 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:20:23.598 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:20:23.598 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:20:23.598 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:20:23.598 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:20:23.598 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:20:23.598 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:20:23.598 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:20:23.600 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:20:23.600 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:20:23.600 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:20:23.600 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:20:23.600 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:20:23.600 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:20:23.600 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:20:23.600 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:20:23.601 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:20:23.601 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:20:23.601 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:20:23.601 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:20:23.601 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:20:23.601 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:20:23.601 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:20:23.601 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:20:23.621 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:23.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:23.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:23.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:20:23.750 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:20:23.751 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:20:23.751 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:23.751 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:23.751 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:23.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:20:23.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:23.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:23.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:23.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:20:23.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:20:23.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:20:23.753 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:20:23.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:23.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:20:23.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:24.011 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 11 2024-09-11T06:20:24.011 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 11 2024-09-11T06:20:24.011 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:20:24.011 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=10638], [object_number=47, tag_tid=8, entry_tid=10635]]], state=connected] 2024-09-11T06:20:24.011 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=46, tag_tid=8, entry_tid=10486], [object_number=45, tag_tid=8, entry_tid=10485], [object_number=44, tag_tid=8, entry_tid=10484], [object_number=47, tag_tid=8, entry_tid=10483]]], state=connected] 2024-09-11T06:20:24.015 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:24.017 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-09-11T06:20:24.018 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:20:24.018 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:24.018 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:24.018 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:24.018 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:20:24.018 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:24.018 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:24.018 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:24.018 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:20:24.018 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:20:24.018 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:20:24.019 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:20:24.019 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:24.020 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:20:24.020 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:24.069 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 11 2024-09-11T06:20:24.069 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 11 2024-09-11T06:20:24.069 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:20:24.069 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=10638], [object_number=47, tag_tid=8, entry_tid=10635]]], state=connected] 2024-09-11T06:20:24.069 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=46, tag_tid=8, entry_tid=10486], [object_number=45, tag_tid=8, entry_tid=10485], [object_number=44, tag_tid=8, entry_tid=10484], [object_number=47, tag_tid=8, entry_tid=10483]]], state=connected] 2024-09-11T06:20:24.072 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:24.074 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=46, tag_tid=8, entry_tid=10486' 2024-09-11T06:20:24.074 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=46, tag_tid=8, entry_tid=10486' 2024-09-11T06:20:24.074 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=46, tag_tid=8, entry_tid=10486' '!=' 'object_number=44, tag_tid=8, entry_tid=10396' 2024-09-11T06:20:24.074 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:20:24.074 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=46, tag_tid=8, entry_tid=10486' = 'object_number=44, tag_tid=8, entry_tid=10396' 2024-09-11T06:20:24.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=46, tag_tid=8, entry_tid=10486' 2024-09-11T06:20:24.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-09-11T06:20:24.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:20:24.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:24.076 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=9 2024-09-11T06:20:24.077 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=46, tag_tid=8, entry_tid=10486' 2024-09-11T06:20:24.077 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:20:24.077 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:24.078 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=8 2024-09-11T06:20:24.078 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-09-11T06:20:24.078 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:20:24.079 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:24.079 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:20:24.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=46, tag_tid=8, entry_tid=10486' 2024-09-11T06:20:24.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:20:24.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:24.081 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=10486 2024-09-11T06:20:24.082 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 9 = 8 -a 1 -le 10486 2024-09-11T06:20:24.082 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:20:24.082 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:20:24.082 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:20:24.283 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:20:24.284 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:20:24.284 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:20:24.284 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:20:24.284 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:20:24.284 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:20:24.284 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:20:24.284 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:20:24.284 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:20:24.284 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:20:24.284 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:20:24.284 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:20:24.284 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:20:24.284 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:20:24.285 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:20:24.285 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:20:24.285 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:20:24.287 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:20:24.288 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:20:24.288 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:20:24.288 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:20:24.288 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:20:24.288 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:20:24.288 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:20:24.288 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:20:24.288 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:20:24.288 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:20:24.288 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:20:24.288 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:20:24.288 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:20:24.288 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:20:24.289 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:20:24.289 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:20:24.289 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:20:24.289 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:20:24.289 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:20:24.289 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:20:24.289 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:20:24.289 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:20:24.289 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:20:24.289 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:20:24.289 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:20:24.289 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:20:24.289 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:20:24.289 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:20:24.290 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:20:24.290 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:20:24.290 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:20:24.290 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:20:24.313 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:24.313 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:24.314 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:24.314 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:20:24.449 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:20:24.451 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:20:24.451 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:24.451 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:24.451 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:24.451 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:20:24.451 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:24.451 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:24.451 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:24.451 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:20:24.452 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:20:24.452 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:20:24.452 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:20:24.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:24.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:20:24.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:24.499 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 11 2024-09-11T06:20:24.499 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 11 2024-09-11T06:20:24.499 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:20:24.499 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=10638], [object_number=47, tag_tid=8, entry_tid=10635]]], state=connected] 2024-09-11T06:20:24.499 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=46, tag_tid=8, entry_tid=10606], [object_number=45, tag_tid=8, entry_tid=10605], [object_number=44, tag_tid=8, entry_tid=10604], [object_number=47, tag_tid=8, entry_tid=10603]]], state=connected] 2024-09-11T06:20:24.502 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:24.503 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-09-11T06:20:24.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:20:24.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:24.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:24.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:24.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:20:24.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:24.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:24.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:24.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:20:24.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:20:24.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:20:24.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:20:24.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:24.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:20:24.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:24.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 11 2024-09-11T06:20:24.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 11 2024-09-11T06:20:24.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:20:24.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=10638], [object_number=47, tag_tid=8, entry_tid=10635]]], state=connected] 2024-09-11T06:20:24.553 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=46, tag_tid=8, entry_tid=10606], [object_number=45, tag_tid=8, entry_tid=10605], [object_number=44, tag_tid=8, entry_tid=10604], [object_number=47, tag_tid=8, entry_tid=10603]]], state=connected] 2024-09-11T06:20:24.556 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:24.558 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=46, tag_tid=8, entry_tid=10606' 2024-09-11T06:20:24.558 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=46, tag_tid=8, entry_tid=10606' 2024-09-11T06:20:24.558 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=46, tag_tid=8, entry_tid=10606' '!=' 'object_number=46, tag_tid=8, entry_tid=10486' 2024-09-11T06:20:24.558 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:20:24.558 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=46, tag_tid=8, entry_tid=10606' = 'object_number=46, tag_tid=8, entry_tid=10486' 2024-09-11T06:20:24.558 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=46, tag_tid=8, entry_tid=10606' 2024-09-11T06:20:24.558 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-09-11T06:20:24.558 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:20:24.558 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:24.559 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=9 2024-09-11T06:20:24.559 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=46, tag_tid=8, entry_tid=10606' 2024-09-11T06:20:24.560 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:20:24.560 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:24.560 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=8 2024-09-11T06:20:24.561 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-09-11T06:20:24.561 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:20:24.561 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:24.562 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:20:24.563 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=46, tag_tid=8, entry_tid=10606' 2024-09-11T06:20:24.563 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:20:24.563 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:24.564 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=10606 2024-09-11T06:20:24.564 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 9 = 8 -a 1 -le 10606 2024-09-11T06:20:24.564 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:20:24.564 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:20:24.565 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:20:24.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:20:24.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:20:24.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:20:24.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:20:24.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:20:24.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:20:24.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:20:24.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:20:24.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:20:24.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:20:24.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:20:24.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:20:24.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:20:24.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:20:24.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:20:24.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:20:24.768 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:20:24.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:20:24.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:20:24.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:20:24.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:20:24.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:20:24.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:20:24.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:20:24.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:20:24.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:20:24.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:20:24.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:20:24.772 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:20:24.772 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:20:24.772 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:20:24.772 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:20:24.772 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:20:24.773 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:20:24.773 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:20:24.773 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:20:24.773 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:20:24.773 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:20:24.773 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:20:24.773 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:20:24.773 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:20:24.773 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:20:24.774 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:20:24.774 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:20:24.774 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:20:24.774 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:20:24.774 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:20:24.774 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:20:24.774 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:20:24.798 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:24.799 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:24.799 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:24.799 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:20:24.925 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:20:24.926 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:20:24.926 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:24.926 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:24.926 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:24.926 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:20:24.926 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:24.926 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:24.926 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:24.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:20:24.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:20:24.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:20:24.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:20:24.929 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:24.929 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:20:24.929 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:24.983 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 11 2024-09-11T06:20:24.983 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 11 2024-09-11T06:20:24.983 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:20:24.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=10638], [object_number=47, tag_tid=8, entry_tid=10635]]], state=connected] 2024-09-11T06:20:24.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=46, tag_tid=8, entry_tid=10638], [object_number=45, tag_tid=8, entry_tid=10637], [object_number=44, tag_tid=8, entry_tid=10636], [object_number=47, tag_tid=8, entry_tid=10635]]], state=connected] 2024-09-11T06:20:24.987 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:24.988 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-09-11T06:20:24.989 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:20:24.989 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:24.990 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:24.990 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:24.990 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:20:24.990 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:24.990 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:24.990 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:24.990 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:20:24.990 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:20:24.990 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:20:24.990 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:20:24.991 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:24.991 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:20:24.991 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:25.037 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 11 2024-09-11T06:20:25.037 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 11 2024-09-11T06:20:25.037 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:20:25.037 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=10638], [object_number=47, tag_tid=8, entry_tid=10635]]], state=connected] 2024-09-11T06:20:25.037 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=46, tag_tid=8, entry_tid=10638], [object_number=45, tag_tid=8, entry_tid=10637], [object_number=44, tag_tid=8, entry_tid=10636], [object_number=47, tag_tid=8, entry_tid=10635]]], state=connected] 2024-09-11T06:20:25.040 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:25.041 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=46, tag_tid=8, entry_tid=10638' 2024-09-11T06:20:25.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=46, tag_tid=8, entry_tid=10638' 2024-09-11T06:20:25.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=46, tag_tid=8, entry_tid=10638' '!=' 'object_number=46, tag_tid=8, entry_tid=10606' 2024-09-11T06:20:25.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:20:25.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=46, tag_tid=8, entry_tid=10638' = 'object_number=46, tag_tid=8, entry_tid=10606' 2024-09-11T06:20:25.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=46, tag_tid=8, entry_tid=10638' 2024-09-11T06:20:25.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-09-11T06:20:25.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:20:25.043 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:25.043 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=9 2024-09-11T06:20:25.044 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=46, tag_tid=8, entry_tid=10638' 2024-09-11T06:20:25.044 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:20:25.044 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:25.045 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=8 2024-09-11T06:20:25.045 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=45, tag_tid=9, entry_tid=1' 2024-09-11T06:20:25.046 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:20:25.046 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:25.046 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:20:25.047 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=46, tag_tid=8, entry_tid=10638' 2024-09-11T06:20:25.047 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:20:25.047 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:20:25.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=10638 2024-09-11T06:20:25.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 9 = 8 -a 1 -le 10638 2024-09-11T06:20:25.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:20:25.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:20:25.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:20:25.250 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:20:25.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:20:25.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:20:25.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:20:25.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:20:25.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:20:25.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:20:25.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:20:25.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:20:25.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:20:25.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:20:25.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:20:25.252 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:20:25.252 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:20:25.252 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:20:25.252 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:20:25.252 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:20:25.256 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:20:25.256 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:20:25.257 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:20:25.257 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:20:25.257 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:20:25.257 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:20:25.257 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:20:25.257 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:20:25.257 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:20:25.257 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:20:25.257 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:20:25.257 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:20:25.257 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:20:25.257 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:20:25.257 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:20:25.258 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:20:25.258 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:20:25.258 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:20:25.258 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:20:25.258 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:20:25.258 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:20:25.258 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:20:25.258 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:20:25.258 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:20:25.258 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:20:25.258 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:20:25.258 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:20:25.259 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:20:25.259 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:20:25.259 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:20:25.259 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:20:25.259 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:20:25.282 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:25.283 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:25.283 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:25.283 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:20:25.399 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:20:25.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:20:25.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:25.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:25.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:25.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:20:25.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:25.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:25.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:25.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:20:25.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:20:25.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:20:25.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:20:25.405 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:25.405 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:20:25.405 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:25.452 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 11 2024-09-11T06:20:25.452 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 11 2024-09-11T06:20:25.452 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:20:25.452 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=10638], [object_number=47, tag_tid=8, entry_tid=10635]]], state=connected] 2024-09-11T06:20:25.452 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=46, tag_tid=8, entry_tid=10638], [object_number=45, tag_tid=8, entry_tid=10637], [object_number=44, tag_tid=8, entry_tid=10636], [object_number=47, tag_tid=8, entry_tid=10635]]], state=connected] 2024-09-11T06:20:25.455 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:25.457 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-09-11T06:20:25.458 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:20:25.458 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:25.458 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:25.458 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:25.458 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:20:25.458 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:25.459 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:25.459 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:25.459 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:20:25.459 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:20:25.459 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:20:25.459 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:20:25.459 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:25.460 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:20:25.460 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:25.504 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 11 2024-09-11T06:20:25.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 11 2024-09-11T06:20:25.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:20:25.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=10638], [object_number=47, tag_tid=8, entry_tid=10635]]], state=connected] 2024-09-11T06:20:25.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=46, tag_tid=8, entry_tid=10638], [object_number=45, tag_tid=8, entry_tid=10637], [object_number=44, tag_tid=8, entry_tid=10636], [object_number=47, tag_tid=8, entry_tid=10635]]], state=connected] 2024-09-11T06:20:25.508 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:25.510 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=46, tag_tid=8, entry_tid=10638' 2024-09-11T06:20:25.510 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=46, tag_tid=8, entry_tid=10638' 2024-09-11T06:20:25.510 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=46, tag_tid=8, entry_tid=10638' '!=' 'object_number=46, tag_tid=8, entry_tid=10638' 2024-09-11T06:20:25.510 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:20:25.510 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.4 2024-09-11T06:20:25.912 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:20:25.912 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:20:25.912 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:20:25.912 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:20:25.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:20:25.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:20:25.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:20:25.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:20:25.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:20:25.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:20:25.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:20:25.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:20:25.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:20:25.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:20:25.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:20:25.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:20:25.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:20:25.917 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:20:25.917 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:20:25.917 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:20:25.917 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:20:25.917 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:20:25.917 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:20:25.917 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:20:25.917 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:20:25.917 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:20:25.917 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:20:25.918 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:20:25.918 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:20:25.918 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:20:25.918 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:20:25.918 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:20:25.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:20:25.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:20:25.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:20:25.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:20:25.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:20:25.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:20:25.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:20:25.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:20:25.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:20:25.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:20:25.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:20:25.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:20:25.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:20:25.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:20:25.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:20:25.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:20:25.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:20:25.942 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:25.944 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:25.944 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:25.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:20:26.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:20:26.063 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:20:26.063 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:26.063 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:26.063 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:26.064 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:20:26.064 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:26.064 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:26.064 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:26.064 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:20:26.064 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:20:26.064 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:20:26.064 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:20:26.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:26.067 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:20:26.067 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:26.118 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 11 2024-09-11T06:20:26.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 11 2024-09-11T06:20:26.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:20:26.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=10638], [object_number=47, tag_tid=8, entry_tid=10635]]], state=connected] 2024-09-11T06:20:26.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=46, tag_tid=8, entry_tid=10638], [object_number=45, tag_tid=8, entry_tid=10637], [object_number=44, tag_tid=8, entry_tid=10636], [object_number=47, tag_tid=8, entry_tid=10635]]], state=connected] 2024-09-11T06:20:26.122 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:26.123 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-09-11T06:20:26.124 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:20:26.124 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:26.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:26.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:26.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:20:26.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:26.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:26.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:26.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:20:26.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:20:26.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:20:26.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:20:26.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:26.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:20:26.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:26.172 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 11 2024-09-11T06:20:26.172 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 11 2024-09-11T06:20:26.172 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:20:26.172 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=10638], [object_number=47, tag_tid=8, entry_tid=10635]]], state=connected] 2024-09-11T06:20:26.172 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=46, tag_tid=8, entry_tid=10638], [object_number=45, tag_tid=8, entry_tid=10637], [object_number=44, tag_tid=8, entry_tid=10636], [object_number=47, tag_tid=8, entry_tid=10635]]], state=connected] 2024-09-11T06:20:26.176 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:26.177 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=46, tag_tid=8, entry_tid=10638' 2024-09-11T06:20:26.177 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=46, tag_tid=8, entry_tid=10638' 2024-09-11T06:20:26.177 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=46, tag_tid=8, entry_tid=10638' '!=' 'object_number=46, tag_tid=8, entry_tid=10638' 2024-09-11T06:20:26.177 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:20:26.177 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.8 2024-09-11T06:20:26.979 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:20:26.979 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:20:26.979 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:20:26.980 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:20:26.980 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:20:26.980 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:20:26.980 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:20:26.980 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:20:26.980 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:20:26.980 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:20:26.980 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:20:26.980 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:20:26.980 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:20:26.980 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:20:26.980 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:20:26.981 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:20:26.981 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:20:26.983 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:20:26.983 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:20:26.983 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:20:26.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:20:26.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:20:26.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:20:26.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:20:26.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:20:26.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:20:26.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:20:26.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:20:26.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:20:26.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:20:26.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:20:26.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:20:26.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:20:26.986 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:20:26.986 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:20:26.986 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:20:26.986 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:20:26.986 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:20:26.986 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:20:26.986 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:20:26.986 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:20:26.986 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:20:26.986 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:20:26.987 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:20:26.987 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:20:26.987 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:20:26.987 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:20:26.987 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:20:26.987 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:20:27.009 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:27.009 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:27.009 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:27.010 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:20:27.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:20:27.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:20:27.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:27.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:27.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:27.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:20:27.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:27.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:27.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:27.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:20:27.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:20:27.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:20:27.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:20:27.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:27.139 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:20:27.139 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:27.184 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 11 2024-09-11T06:20:27.185 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 11 2024-09-11T06:20:27.185 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:20:27.185 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=10638], [object_number=47, tag_tid=8, entry_tid=10635]]], state=connected] 2024-09-11T06:20:27.185 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=46, tag_tid=8, entry_tid=10638], [object_number=45, tag_tid=8, entry_tid=10637], [object_number=44, tag_tid=8, entry_tid=10636], [object_number=47, tag_tid=8, entry_tid=10635]]], state=connected] 2024-09-11T06:20:27.188 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:27.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-09-11T06:20:27.190 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:20:27.190 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:27.190 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:27.190 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:27.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:20:27.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:27.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:27.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:27.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:20:27.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:20:27.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:20:27.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:20:27.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:27.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:20:27.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:27.237 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 11 2024-09-11T06:20:27.237 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 11 2024-09-11T06:20:27.237 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:20:27.237 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=10638], [object_number=47, tag_tid=8, entry_tid=10635]]], state=connected] 2024-09-11T06:20:27.238 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=46, tag_tid=8, entry_tid=10638], [object_number=45, tag_tid=8, entry_tid=10637], [object_number=44, tag_tid=8, entry_tid=10636], [object_number=47, tag_tid=8, entry_tid=10635]]], state=connected] 2024-09-11T06:20:27.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:27.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=46, tag_tid=8, entry_tid=10638' 2024-09-11T06:20:27.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=46, tag_tid=8, entry_tid=10638' 2024-09-11T06:20:27.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=46, tag_tid=8, entry_tid=10638' '!=' 'object_number=46, tag_tid=8, entry_tid=10638' 2024-09-11T06:20:27.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:20:27.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1.6 2024-09-11T06:20:28.844 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:20:28.844 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:20:28.844 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:20:28.844 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:20:28.845 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:20:28.845 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:20:28.845 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:20:28.845 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:20:28.845 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:20:28.845 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:20:28.845 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:20:28.845 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:20:28.845 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:20:28.845 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:20:28.845 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:20:28.846 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:20:28.846 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:20:28.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:20:28.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:20:28.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:20:28.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:20:28.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:20:28.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:20:28.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:20:28.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:20:28.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:20:28.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:20:28.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:20:28.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:20:28.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:20:28.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:20:28.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:20:28.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:20:28.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:20:28.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:20:28.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:20:28.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:20:28.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:20:28.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:20:28.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:20:28.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:20:28.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:20:28.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:20:28.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:20:28.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:20:28.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:20:28.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:20:28.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:20:28.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:20:28.874 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:28.875 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:28.875 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:20:28.875 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:20:29.003 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:20:29.004 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:20:29.004 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:29.004 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:29.004 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:29.004 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:20:29.004 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:29.004 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:29.005 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:29.005 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:20:29.005 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:20:29.005 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:20:29.005 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:20:29.007 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:29.007 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:20:29.007 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:29.058 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 11 2024-09-11T06:20:29.058 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 11 2024-09-11T06:20:29.058 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:20:29.058 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=10638], [object_number=47, tag_tid=8, entry_tid=10635]]], state=connected] 2024-09-11T06:20:29.058 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=10638], [object_number=47, tag_tid=8, entry_tid=10635]]], state=connected] 2024-09-11T06:20:29.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:29.063 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-09-11T06:20:29.064 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:20:29.064 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:29.064 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:29.064 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:29.065 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:20:29.065 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:20:29.065 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:20:29.065 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:20:29.065 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:20:29.065 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:20:29.065 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:20:29.065 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:20:29.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:29.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:20:29.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:29.113 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 11 2024-09-11T06:20:29.113 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 11 2024-09-11T06:20:29.113 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:20:29.113 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=10638], [object_number=47, tag_tid=8, entry_tid=10635]]], state=connected] 2024-09-11T06:20:29.113 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=45, tag_tid=9, entry_tid=1], [object_number=44, tag_tid=9, entry_tid=0], [object_number=46, tag_tid=8, entry_tid=10638], [object_number=47, tag_tid=8, entry_tid=10635]]], state=connected] 2024-09-11T06:20:29.116 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:20:29.118 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=45, tag_tid=9, entry_tid=1' 2024-09-11T06:20:29.118 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=45, tag_tid=9, entry_tid=1' -a 'object_number=45, tag_tid=9, entry_tid=1' = 'object_number=45, tag_tid=9, entry_tid=1' 2024-09-11T06:20:29.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:20:29.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_snap_present cluster1 mirror test snap3 2024-09-11T06:20:29.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:20:29.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:20:29.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:20:29.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap3 2024-09-11T06:20:29.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:20:29.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T06:20:29.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:20:30.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 info mirror/test@snap3 2024-09-11T06:20:30.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:rbd image 'test': 2024-09-11T06:20:30.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: size 512 MiB in 128 objects 2024-09-11T06:20:30.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: order 22 (4 MiB objects) 2024-09-11T06:20:30.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: snapshot_count: 5 2024-09-11T06:20:30.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: id: 10a618cfe337 2024-09-11T06:20:30.190 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: block_name_prefix: rbd_data.10a618cfe337 2024-09-11T06:20:30.190 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: format: 2 2024-09-11T06:20:30.190 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: features: layering, exclusive-lock, journaling 2024-09-11T06:20:30.190 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: op_features: 2024-09-11T06:20:30.190 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: flags: 2024-09-11T06:20:30.190 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: create_timestamp: Wed Sep 11 06:16:46 2024 2024-09-11T06:20:30.190 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: access_timestamp: Wed Sep 11 06:19:23 2024 2024-09-11T06:20:30.190 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: modify_timestamp: Wed Sep 11 06:19:49 2024 2024-09-11T06:20:30.190 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: protected: False 2024-09-11T06:20:30.190 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: journal: 10a618cfe337 2024-09-11T06:20:30.190 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring state: enabled 2024-09-11T06:20:30.190 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring mode: journal 2024-09-11T06:20:30.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring global id: a0f7d515-874e-4c67-a26b-4958c7ab3083 2024-09-11T06:20:30.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring primary: false 2024-09-11T06:20:30.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:20:30.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n snap2-clean ']' 2024-09-11T06:20:30.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ compare_image_snaps mirror test snap2-clean 2024-09-11T06:20:30.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:20:30.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:20:30.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap2-clean 2024-09-11T06:20:30.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local ret=0 2024-09-11T06:20:30.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local rmt_export=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export 2024-09-11T06:20:30.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loc_export=/tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:20:30.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:20:30.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2-clean /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export 2024-09-11T06:20:30.773 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T06:20:30.783 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2-clean /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:20:32.152 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T06:20:32.172 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmp /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:20:32.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:20:32.590 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:20:32.590 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ compare_image_snaps mirror test snap3 2024-09-11T06:20:32.590 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:20:32.590 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:20:32.590 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap3 2024-09-11T06:20:32.590 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local ret=0 2024-09-11T06:20:32.590 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local rmt_export=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export 2024-09-11T06:20:32.590 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loc_export=/tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:20:32.590 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:20:32.591 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3 /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export 2024-09-11T06:20:33.234 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T06:20:33.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3 /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:20:34.603 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T06:20:34.625 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmp /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:20:34.906 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:20:35.044 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:20:35.044 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ clean_snap_name=snap3-clean 2024-09-11T06:20:35.044 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_snap cluster2 mirror test snap3-clean 2024-09-11T06:20:35.045 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:20:35.045 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:20:35.045 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:20:35.045 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap3-clean 2024-09-11T06:20:35.045 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap3-clean --debug-rbd=20 --debug-journaler=20 2024-09-11T06:20:35.367 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 10` 2024-09-11T06:20:35.367 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ stress_write_image cluster2 mirror test 2024-09-11T06:20:35.367 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:20:35.367 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:20:35.367 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:20:35.367 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T06:20:35.369 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local duration=11 2024-09-11T06:20:35.369 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set +e 2024-09-11T06:20:35.369 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-11T06:20:36.318 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:100 IOs, 12612608 bytes 2024-09-11T06:20:38.038 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:200 IOs, 45449216 bytes 2024-09-11T06:20:39.338 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:300 IOs, 71232512 bytes 2024-09-11T06:20:41.058 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:400 IOs, 108672000 bytes 2024-09-11T06:20:41.663 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:500 IOs, 121603072 bytes 2024-09-11T06:20:42.706 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:600 IOs, 143536128 bytes 2024-09-11T06:20:45.181 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:700 IOs, 223155200 bytes 2024-09-11T06:20:46.376 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ error_code=124 2024-09-11T06:20:46.376 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set -e 2024-09-11T06:20:46.376 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T06:20:46.376 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:20:46.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-11T06:20:46.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:20:46.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:20:46.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:20:46.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state_pattern=up+replaying 2024-09-11T06:20:46.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local description_pattern= 2024-09-11T06:20:46.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local service_pattern= 2024-09-11T06:20:46.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:20:46.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:20:47.378 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-11T06:20:47.378 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:20:47.378 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:20:47.378 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:20:47.378 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state_pattern=up+replaying 2024-09-11T06:20:47.378 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local description_pattern= 2024-09-11T06:20:47.379 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local service_pattern= 2024-09-11T06:20:47.379 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local status 2024-09-11T06:20:47.379 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ CEPH_ARGS= 2024-09-11T06:20:47.379 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-11T06:20:47.466 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ status='test: 2024-09-11T06:20:47.466 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: global_id: a0f7d515-874e-4c67-a26b-4958c7ab3083 2024-09-11T06:20:47.466 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: state: up+replaying 2024-09-11T06:20:47.466 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: description: replaying, {"bytes_per_second":1857473.73,"entries_behind_primary":1,"entries_per_second":115.03,"non_primary_position":{"entry_tid":1,"object_number":45,"tag_tid":9},"primary_position":{"entry_tid":1,"object_number":45,"tag_tid":10},"seconds_until_synced":0} 2024-09-11T06:20:47.466 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: service: mirror.0 on smithi112 2024-09-11T06:20:47.466 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: last_update: 2024-09-11 06:20:44 2024-09-11T06:20:47.466 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: peer_sites: 2024-09-11T06:20:47.466 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: name: cluster2 2024-09-11T06:20:47.466 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: state: up+stopped 2024-09-11T06:20:47.466 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: description: local image is primary 2024-09-11T06:20:47.467 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: last_update: 2024-09-11 06:20:44' 2024-09-11T06:20:47.467 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ grep '^ state: .*up+replaying' 2024-09-11T06:20:47.468 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: state: up+replaying 2024-09-11T06:20:47.468 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ grep '^ description: .*' 2024-09-11T06:20:47.468 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: description: replaying, {"bytes_per_second":1857473.73,"entries_behind_primary":1,"entries_per_second":115.03,"non_primary_position":{"entry_tid":1,"object_number":45,"tag_tid":9},"primary_position":{"entry_tid":1,"object_number":45,"tag_tid":10},"seconds_until_synced":0} 2024-09-11T06:20:47.469 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n '' ']' 2024-09-11T06:20:47.469 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ grep '^up+' 2024-09-11T06:20:47.471 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ echo up+replaying 2024-09-11T06:20:47.471 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:up+replaying 2024-09-11T06:20:47.472 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ grep 'service: *mirror..* on ' 2024-09-11T06:20:47.473 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: service: mirror.0 on smithi112 2024-09-11T06:20:47.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local last_update 2024-09-11T06:20:47.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-11T06:20:47.476 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_update='2024-09-11 06:20:44' 2024-09-11T06:20:47.476 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-11 06:20:44' 2024-09-11T06:20:47.476 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:20:47.476 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:20:47.476 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:20:47.476 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state_pattern=up+replaying 2024-09-11T06:20:47.477 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'prev_last_update=2024-09-11 06:20:44' 2024-09-11T06:20:47.477 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local status 2024-09-11T06:20:47.477 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ CEPH_ARGS= 2024-09-11T06:20:47.477 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-11T06:20:47.577 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ status='OKOKOK142554262mirror.0smithi11219.1.1-298-gb53b6d3etrueOK42594265mirror.1smithi11219.1.1-298-gb53b6d3efalseOKtesta0f7d515-874e-4c67-a26b-4958c7ab3083up+replayingreplaying, {"bytes_per_second":1857473.73,"entries_behind_primary":1,"entries_per_second":115.03,"non_primary_position":{"entry_tid":1,"object_number":45,"tag_tid":9},"primary_position":{"entry_tid":1,"object_number":45,"tag_tid":10},"seconds_until_synced":0}42554262mirror.0smithi1122024-09-11 06:20:44cluster20bfc6eda-f3c5-424a-8ff7-5c4d3d9fd488up+stoppedlocal image is primary2024-09-11 06:20:44' 2024-09-11T06:20:47.577 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local last_update state 2024-09-11T06:20:47.577 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-11T06:20:47.580 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_update='2024-09-11 06:20:44' 2024-09-11T06:20:47.581 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-11T06:20:47.584 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ state=up+replaying 2024-09-11T06:20:47.584 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ echo up+replaying 2024-09-11T06:20:47.585 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ grep up+replaying 2024-09-11T06:20:47.585 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:up+replaying 2024-09-11T06:20:47.586 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:20:47.586 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:20:47.586 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ snap_name=snap4 2024-09-11T06:20:47.586 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_snap cluster2 mirror test snap4 2024-09-11T06:20:47.586 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:20:47.587 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:20:47.587 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:20:47.587 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap4 2024-09-11T06:20:47.587 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap4 --debug-rbd=20 --debug-journaler=20 2024-09-11T06:21:17.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-11T06:21:17.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:17.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:21:17.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:21:17.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-11T06:21:17.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:17.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:21:17.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:21:17.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=started 2024-09-11T06:21:17.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:21:17.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:21:17.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:21:18.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-11T06:21:18.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:18.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:21:18.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:21:18.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=started 2024-09-11T06:21:18.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local status_result 2024-09-11T06:21:18.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=stopped 2024-09-11T06:21:18.483 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-11T06:21:18.483 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -i state 2024-09-11T06:21:18.483 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_instance=cluster1 2024-09-11T06:21:18.483 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:21:18.483 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:21:18.483 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance=cluster1 2024-09-11T06:21:18.483 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local loop_instance 2024-09-11T06:21:18.483 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:21:18.483 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sleep 0 2024-09-11T06:21:18.484 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:21:18.484 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ seq 0 1 2024-09-11T06:21:18.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:21:18.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-11T06:21:18.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1:0 2024-09-11T06:21:18.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:21:18.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:21:18.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:21:18.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:21:18.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=cluster 2024-09-11T06:21:18.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:21:18.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:18.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:18.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:18.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval cluster=cluster1 2024-09-11T06:21:18.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:21:18.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:21:18.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:21:18.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:21:18.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local local_cluster=cluster1:0 2024-09-11T06:21:18.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster=cluster1 2024-09-11T06:21:18.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance 2024-09-11T06:21:18.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:21:18.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local val=cluster1:0 2024-09-11T06:21:18.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T06:21:18.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance_var_name=instance 2024-09-11T06:21:18.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:21:18.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:21:18.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:18.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval local_cluster=cluster1 2024-09-11T06:21:18.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ local_cluster=cluster1 2024-09-11T06:21:18.490 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval instance=0 2024-09-11T06:21:18.490 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ instance=0 2024-09-11T06:21:18.490 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:21:18.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:18.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:18.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:18.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-11T06:21:18.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ return 0 2024-09-11T06:21:18.631 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T06:21:18.632 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ grep -i Replaying 2024-09-11T06:21:18.632 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ echo ' "state": "Replaying"' 2024-09-11T06:21:18.633 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: "state": "Replaying" 2024-09-11T06:21:18.634 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_state=started 2024-09-11T06:21:18.634 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test started = started 2024-09-11T06:21:18.634 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:21:18.634 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-11T06:21:18.634 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:21:18.635 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:21:18.635 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:21:18.635 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:21:18.635 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' journal = journal ']' 2024-09-11T06:21:18.635 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-09-11T06:21:18.635 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:21:18.635 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:21:18.635 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:21:18.635 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:21:18.635 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-11T06:21:18.635 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-11T06:21:18.635 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:21:18.635 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:21:18.635 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:21:18.836 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:21:18.836 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:18.836 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:21:18.836 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:21:18.837 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:21:18.837 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:21:18.837 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:21:18.837 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:21:18.837 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:21:18.837 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:18.837 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:18.837 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:21:18.838 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:21:18.838 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:21:18.838 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:21:18.841 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:21:18.841 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:21:18.841 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:21:18.841 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:21:18.841 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:21:18.841 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:18.842 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:21:18.842 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:21:18.842 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:21:18.842 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:21:18.842 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:21:18.842 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:21:18.842 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:21:18.842 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:18.842 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:21:18.842 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:18.842 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:21:18.842 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:18.842 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:21:18.843 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:21:18.843 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:21:18.843 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:21:18.843 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:21:18.843 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:21:18.843 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:21:18.843 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:21:18.843 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:18.843 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:18.843 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:18.843 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:21:18.843 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:21:18.843 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:21:18.843 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:21:18.844 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:21:18.866 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:18.867 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:18.867 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:18.867 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:21:18.991 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:21:18.992 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:21:18.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:18.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:18.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:18.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:21:18.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:18.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:18.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:18.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:21:18.994 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:18.994 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:18.994 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:18.995 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:18.996 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:18.996 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:19.047 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 13 2024-09-11T06:21:19.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:19.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:19.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:19.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=52, tag_tid=11, entry_tid=7220], [object_number=55, tag_tid=11, entry_tid=7219], [object_number=54, tag_tid=11, entry_tid=7218], [object_number=53, tag_tid=11, entry_tid=7217]]], state=connected] 2024-09-11T06:21:19.051 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:19.052 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:19.053 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:21:19.053 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:19.054 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:19.054 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:19.054 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:21:19.054 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:19.054 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:19.054 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:19.054 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:21:19.054 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:19.054 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:19.054 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:19.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:19.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:19.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:19.106 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 13 2024-09-11T06:21:19.106 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:19.106 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:19.106 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:19.106 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=52, tag_tid=11, entry_tid=7220], [object_number=55, tag_tid=11, entry_tid=7219], [object_number=54, tag_tid=11, entry_tid=7218], [object_number=53, tag_tid=11, entry_tid=7217]]], state=connected] 2024-09-11T06:21:19.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:19.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=52, tag_tid=11, entry_tid=7220' 2024-09-11T06:21:19.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=52, tag_tid=11, entry_tid=7220' 2024-09-11T06:21:19.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=52, tag_tid=11, entry_tid=7220' '!=' '' 2024-09-11T06:21:19.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:21:19.112 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=52, tag_tid=11, entry_tid=7220' = '' 2024-09-11T06:21:19.112 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=52, tag_tid=11, entry_tid=7220' 2024-09-11T06:21:19.112 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:19.112 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:19.112 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:19.112 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=12 2024-09-11T06:21:19.113 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=52, tag_tid=11, entry_tid=7220' 2024-09-11T06:21:19.113 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:19.113 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:19.114 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=11 2024-09-11T06:21:19.115 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:19.115 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:19.115 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:19.116 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:21:19.116 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=52, tag_tid=11, entry_tid=7220' 2024-09-11T06:21:19.116 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:19.117 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:19.118 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=7220 2024-09-11T06:21:19.118 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 12 = 11 -a 1 -le 7220 2024-09-11T06:21:19.118 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:21:19.118 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:21:19.118 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:21:19.320 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:21:19.320 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:19.320 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:21:19.320 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:21:19.320 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:21:19.320 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:21:19.320 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:21:19.320 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:21:19.320 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:21:19.320 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:19.321 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:19.321 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:21:19.321 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:21:19.321 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:21:19.321 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:21:19.321 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:21:19.321 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:21:19.323 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:21:19.323 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:21:19.324 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:21:19.324 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:19.324 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:21:19.324 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:21:19.324 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:21:19.324 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:21:19.324 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:21:19.324 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:21:19.324 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:21:19.324 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:19.324 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:21:19.324 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:19.324 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:21:19.324 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:19.326 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:21:19.326 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:21:19.326 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:21:19.326 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:21:19.326 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:21:19.326 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:21:19.326 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:21:19.326 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:21:19.326 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:19.326 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:19.326 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:19.327 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:21:19.327 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:21:19.327 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:21:19.327 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:21:19.327 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:21:19.351 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:19.351 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:19.351 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:19.351 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:21:19.476 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:21:19.477 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:21:19.477 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:19.477 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:19.477 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:19.477 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:21:19.477 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:19.478 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:19.478 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:19.478 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:21:19.478 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:19.478 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:19.478 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:19.479 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:19.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:19.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:19.536 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 13 2024-09-11T06:21:19.536 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:19.537 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:19.537 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:19.537 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=56, tag_tid=11, entry_tid=7268], [object_number=55, tag_tid=11, entry_tid=7267], [object_number=54, tag_tid=11, entry_tid=7266], [object_number=57, tag_tid=11, entry_tid=7265]]], state=connected] 2024-09-11T06:21:19.540 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:19.541 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:19.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:21:19.543 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:19.543 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:19.543 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:19.543 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:21:19.543 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:19.543 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:19.543 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:19.543 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:21:19.543 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:19.543 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:19.543 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:19.544 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:19.544 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:19.544 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:19.590 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 13 2024-09-11T06:21:19.590 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:19.590 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:19.590 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:19.590 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=56, tag_tid=11, entry_tid=7268], [object_number=55, tag_tid=11, entry_tid=7267], [object_number=54, tag_tid=11, entry_tid=7266], [object_number=57, tag_tid=11, entry_tid=7265]]], state=connected] 2024-09-11T06:21:19.593 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:19.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=56, tag_tid=11, entry_tid=7268' 2024-09-11T06:21:19.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=56, tag_tid=11, entry_tid=7268' 2024-09-11T06:21:19.596 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=56, tag_tid=11, entry_tid=7268' '!=' 'object_number=52, tag_tid=11, entry_tid=7220' 2024-09-11T06:21:19.596 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:21:19.596 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=56, tag_tid=11, entry_tid=7268' = 'object_number=52, tag_tid=11, entry_tid=7220' 2024-09-11T06:21:19.596 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=56, tag_tid=11, entry_tid=7268' 2024-09-11T06:21:19.596 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:19.596 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:19.596 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:19.596 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=12 2024-09-11T06:21:19.597 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=56, tag_tid=11, entry_tid=7268' 2024-09-11T06:21:19.597 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:19.597 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:19.598 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=11 2024-09-11T06:21:19.599 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:19.599 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:19.599 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:19.600 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:21:19.601 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=56, tag_tid=11, entry_tid=7268' 2024-09-11T06:21:19.601 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:19.601 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:19.602 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=7268 2024-09-11T06:21:19.602 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 12 = 11 -a 1 -le 7268 2024-09-11T06:21:19.602 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:21:19.602 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:21:19.602 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:21:19.803 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:21:19.804 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:19.804 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:21:19.804 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:21:19.804 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:21:19.804 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:21:19.804 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:21:19.804 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:21:19.804 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:21:19.804 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:19.804 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:19.804 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:21:19.804 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:21:19.804 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:21:19.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:21:19.807 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:21:19.807 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:21:19.807 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:21:19.807 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:21:19.807 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:21:19.807 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:19.807 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:21:19.807 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:21:19.808 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:21:19.808 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:21:19.808 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:21:19.808 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:21:19.808 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:21:19.808 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:19.808 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:21:19.808 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:19.808 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:21:19.808 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:19.808 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:21:19.808 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:21:19.808 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:21:19.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:21:19.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:21:19.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:21:19.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:21:19.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:21:19.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:19.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:19.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:19.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:21:19.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:21:19.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:21:19.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:21:19.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:21:19.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:19.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:19.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:19.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:21:19.960 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:21:19.961 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:21:19.961 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:19.961 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:19.961 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:19.961 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:21:19.961 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:19.962 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:19.962 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:19.962 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:21:19.962 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:19.962 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:19.962 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:19.964 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:19.964 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:19.964 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:20.010 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 13 2024-09-11T06:21:20.010 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:20.010 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:20.010 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:20.010 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=55, tag_tid=11, entry_tid=7347], [object_number=58, tag_tid=11, entry_tid=7346], [object_number=57, tag_tid=11, entry_tid=7345], [object_number=56, tag_tid=11, entry_tid=7344]]], state=connected] 2024-09-11T06:21:20.013 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:20.014 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:20.016 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:21:20.016 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:20.016 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:20.016 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:20.016 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:21:20.016 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:20.016 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:20.016 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:20.016 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:21:20.017 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:20.017 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:20.017 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:20.017 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:20.018 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:20.018 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:20.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 13 2024-09-11T06:21:20.063 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:20.063 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:20.063 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:20.063 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=55, tag_tid=11, entry_tid=7347], [object_number=58, tag_tid=11, entry_tid=7346], [object_number=57, tag_tid=11, entry_tid=7345], [object_number=56, tag_tid=11, entry_tid=7344]]], state=connected] 2024-09-11T06:21:20.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:20.068 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=55, tag_tid=11, entry_tid=7347' 2024-09-11T06:21:20.068 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=55, tag_tid=11, entry_tid=7347' 2024-09-11T06:21:20.068 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=55, tag_tid=11, entry_tid=7347' '!=' 'object_number=56, tag_tid=11, entry_tid=7268' 2024-09-11T06:21:20.068 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:21:20.068 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=55, tag_tid=11, entry_tid=7347' = 'object_number=56, tag_tid=11, entry_tid=7268' 2024-09-11T06:21:20.068 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=55, tag_tid=11, entry_tid=7347' 2024-09-11T06:21:20.069 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:20.069 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:20.069 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:20.069 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=12 2024-09-11T06:21:20.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=55, tag_tid=11, entry_tid=7347' 2024-09-11T06:21:20.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:20.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:20.071 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=11 2024-09-11T06:21:20.072 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:20.072 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:20.072 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:20.073 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:21:20.074 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=55, tag_tid=11, entry_tid=7347' 2024-09-11T06:21:20.074 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:20.074 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:20.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=7347 2024-09-11T06:21:20.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 12 = 11 -a 1 -le 7347 2024-09-11T06:21:20.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:21:20.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:21:20.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:21:20.277 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:21:20.277 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:20.278 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:21:20.278 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:21:20.278 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:21:20.278 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:21:20.278 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:21:20.278 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:21:20.278 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:21:20.278 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:20.278 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:20.278 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:21:20.278 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:21:20.278 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:21:20.279 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:21:20.279 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:21:20.281 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:21:20.282 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:21:20.282 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:21:20.282 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:21:20.282 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:20.282 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:21:20.282 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:21:20.282 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:21:20.282 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:21:20.282 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:21:20.282 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:21:20.282 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:21:20.282 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:20.282 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:21:20.282 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:20.283 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:21:20.283 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:20.283 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:21:20.283 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:21:20.283 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:21:20.283 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:21:20.283 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:21:20.283 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:21:20.283 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:21:20.283 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:21:20.283 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:20.283 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:20.283 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:20.283 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:21:20.283 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:21:20.284 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:21:20.284 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:21:20.284 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:21:20.306 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:20.307 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:20.307 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:20.307 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:21:20.437 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:21:20.438 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:21:20.438 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:20.438 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:20.438 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:20.438 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:21:20.438 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:20.438 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:20.438 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:20.438 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:21:20.438 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:20.439 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:20.439 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:20.440 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:20.440 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:20.440 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:20.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 13 2024-09-11T06:21:20.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:20.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:20.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:20.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=58, tag_tid=11, entry_tid=7466], [object_number=57, tag_tid=11, entry_tid=7465], [object_number=56, tag_tid=11, entry_tid=7464], [object_number=55, tag_tid=11, entry_tid=7463]]], state=connected] 2024-09-11T06:21:20.492 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:20.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:20.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:21:20.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:20.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:20.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:20.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:21:20.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:20.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:20.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:20.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:21:20.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:20.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:20.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:20.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:20.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:20.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:20.543 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 13 2024-09-11T06:21:20.543 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:20.544 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:20.544 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:20.544 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=58, tag_tid=11, entry_tid=7466], [object_number=57, tag_tid=11, entry_tid=7465], [object_number=56, tag_tid=11, entry_tid=7464], [object_number=55, tag_tid=11, entry_tid=7463]]], state=connected] 2024-09-11T06:21:20.547 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:20.549 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=58, tag_tid=11, entry_tid=7466' 2024-09-11T06:21:20.549 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=58, tag_tid=11, entry_tid=7466' 2024-09-11T06:21:20.549 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=58, tag_tid=11, entry_tid=7466' '!=' 'object_number=55, tag_tid=11, entry_tid=7347' 2024-09-11T06:21:20.550 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:21:20.550 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=58, tag_tid=11, entry_tid=7466' = 'object_number=55, tag_tid=11, entry_tid=7347' 2024-09-11T06:21:20.550 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=58, tag_tid=11, entry_tid=7466' 2024-09-11T06:21:20.550 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:20.550 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:20.550 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:20.550 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=12 2024-09-11T06:21:20.551 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=58, tag_tid=11, entry_tid=7466' 2024-09-11T06:21:20.551 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:20.551 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:20.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=11 2024-09-11T06:21:20.553 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:20.553 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:20.553 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:20.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:21:20.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=58, tag_tid=11, entry_tid=7466' 2024-09-11T06:21:20.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:20.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:20.555 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=7466 2024-09-11T06:21:20.556 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 12 = 11 -a 1 -le 7466 2024-09-11T06:21:20.556 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:21:20.556 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:21:20.556 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:21:20.758 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:21:20.758 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:20.758 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:21:20.758 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:21:20.758 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:21:20.758 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:21:20.758 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:21:20.758 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:21:20.758 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:21:20.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:20.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:20.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:21:20.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:21:20.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:21:20.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:21:20.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:21:20.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:21:20.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:21:20.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:21:20.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:21:20.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:20.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:21:20.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:21:20.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:21:20.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:21:20.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:21:20.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:21:20.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:21:20.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:20.764 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:21:20.764 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:20.764 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:21:20.764 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:20.764 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:21:20.764 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:21:20.764 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:21:20.764 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:21:20.764 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:21:20.764 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:21:20.764 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:21:20.764 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:21:20.764 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:20.764 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:20.765 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:20.765 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:21:20.765 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:21:20.765 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:21:20.765 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:21:20.765 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:21:20.788 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:20.788 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:20.788 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:20.788 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:21:20.925 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:21:20.926 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:21:20.926 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:20.926 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:20.926 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:20.926 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:21:20.926 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:20.926 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:20.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:20.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:21:20.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:20.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:20.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:20.928 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:20.929 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:20.929 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:20.990 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 14 2024-09-11T06:21:20.990 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:20.990 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:20.990 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:20.990 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=58, tag_tid=11, entry_tid=7554], [object_number=57, tag_tid=11, entry_tid=7553], [object_number=56, tag_tid=11, entry_tid=7552], [object_number=59, tag_tid=11, entry_tid=7551]]], state=connected] 2024-09-11T06:21:20.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:20.995 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:20.996 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:21:20.996 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:20.996 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:20.996 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:20.996 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:21:20.996 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:20.996 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:20.996 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:20.996 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:21:20.997 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:20.997 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:20.997 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:20.997 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:20.997 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:20.997 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:21.046 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 14 2024-09-11T06:21:21.046 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:21.046 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:21.046 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:21.046 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=58, tag_tid=11, entry_tid=7554], [object_number=57, tag_tid=11, entry_tid=7553], [object_number=56, tag_tid=11, entry_tid=7552], [object_number=59, tag_tid=11, entry_tid=7551]]], state=connected] 2024-09-11T06:21:21.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:21.051 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=58, tag_tid=11, entry_tid=7554' 2024-09-11T06:21:21.051 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=58, tag_tid=11, entry_tid=7554' 2024-09-11T06:21:21.051 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=58, tag_tid=11, entry_tid=7554' '!=' 'object_number=58, tag_tid=11, entry_tid=7466' 2024-09-11T06:21:21.051 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:21:21.051 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=58, tag_tid=11, entry_tid=7554' = 'object_number=58, tag_tid=11, entry_tid=7466' 2024-09-11T06:21:21.051 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=58, tag_tid=11, entry_tid=7554' 2024-09-11T06:21:21.052 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:21.052 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:21.052 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:21.053 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=12 2024-09-11T06:21:21.053 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=58, tag_tid=11, entry_tid=7554' 2024-09-11T06:21:21.053 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:21.053 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:21.054 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=11 2024-09-11T06:21:21.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:21.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:21.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:21.056 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:21:21.057 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=58, tag_tid=11, entry_tid=7554' 2024-09-11T06:21:21.057 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:21.057 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:21.058 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=7554 2024-09-11T06:21:21.058 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 12 = 11 -a 1 -le 7554 2024-09-11T06:21:21.058 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:21:21.058 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:21:21.058 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:21:21.259 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:21:21.260 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:21.260 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:21:21.260 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:21:21.260 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:21:21.260 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:21:21.260 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:21:21.260 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:21:21.260 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:21:21.260 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:21.260 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:21.260 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:21:21.260 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:21:21.261 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:21:21.261 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:21:21.261 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:21:21.261 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:21:21.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:21:21.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:21:21.264 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:21:21.264 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:21.264 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:21:21.264 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:21:21.264 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:21:21.264 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:21:21.264 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:21:21.264 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:21:21.264 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:21:21.264 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:21.264 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:21:21.264 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:21.265 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:21:21.265 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:21.266 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:21:21.266 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:21:21.266 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:21:21.266 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:21:21.266 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:21:21.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:21:21.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:21:21.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:21:21.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:21.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:21.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:21.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:21:21.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:21:21.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:21:21.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:21:21.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:21:21.290 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:21.291 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:21.291 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:21.291 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:21:21.421 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:21:21.422 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:21:21.422 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:21.422 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:21.423 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:21.423 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:21:21.423 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:21.423 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:21.423 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:21.423 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:21:21.424 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:21.424 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:21.424 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:21.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:21.427 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:21.427 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:21.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 14 2024-09-11T06:21:21.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:21.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:21.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:21.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=59, tag_tid=11, entry_tid=7667], [object_number=58, tag_tid=11, entry_tid=7666], [object_number=57, tag_tid=11, entry_tid=7665], [object_number=56, tag_tid=11, entry_tid=7664]]], state=connected] 2024-09-11T06:21:21.485 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:21.486 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:21.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:21:21.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:21.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:21.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:21.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:21:21.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:21.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:21.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:21.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:21:21.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:21.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:21.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:21.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:21.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:21.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:21.536 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 14 2024-09-11T06:21:21.536 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:21.536 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:21.536 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:21.536 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=59, tag_tid=11, entry_tid=7667], [object_number=58, tag_tid=11, entry_tid=7666], [object_number=57, tag_tid=11, entry_tid=7665], [object_number=56, tag_tid=11, entry_tid=7664]]], state=connected] 2024-09-11T06:21:21.539 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:21.541 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=59, tag_tid=11, entry_tid=7667' 2024-09-11T06:21:21.541 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=59, tag_tid=11, entry_tid=7667' 2024-09-11T06:21:21.541 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=59, tag_tid=11, entry_tid=7667' '!=' 'object_number=58, tag_tid=11, entry_tid=7554' 2024-09-11T06:21:21.541 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:21:21.541 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=59, tag_tid=11, entry_tid=7667' = 'object_number=58, tag_tid=11, entry_tid=7554' 2024-09-11T06:21:21.541 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=59, tag_tid=11, entry_tid=7667' 2024-09-11T06:21:21.541 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:21.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:21.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:21.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=12 2024-09-11T06:21:21.543 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=59, tag_tid=11, entry_tid=7667' 2024-09-11T06:21:21.543 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:21.543 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:21.544 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=11 2024-09-11T06:21:21.544 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:21.545 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:21.545 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:21.546 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:21:21.546 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=59, tag_tid=11, entry_tid=7667' 2024-09-11T06:21:21.546 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:21.546 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:21.548 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=7667 2024-09-11T06:21:21.548 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 12 = 11 -a 1 -le 7667 2024-09-11T06:21:21.548 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:21:21.548 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:21:21.548 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:21:21.749 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:21:21.750 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:21.750 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:21:21.750 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:21:21.750 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:21:21.750 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:21:21.750 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:21:21.750 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:21:21.750 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:21:21.750 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:21.750 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:21.750 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:21:21.750 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:21:21.751 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:21:21.751 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:21:21.751 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:21:21.751 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:21:21.754 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:21:21.754 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:21:21.754 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:21:21.754 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:21.754 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:21:21.754 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:21:21.754 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:21:21.754 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:21:21.754 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:21:21.754 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:21:21.754 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:21:21.754 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:21.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:21:21.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:21.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:21:21.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:21.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:21:21.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:21:21.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:21:21.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:21:21.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:21:21.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:21:21.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:21:21.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:21:21.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:21.756 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:21.756 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:21.756 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:21:21.756 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:21:21.756 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:21:21.756 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:21:21.756 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:21:21.781 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:21.781 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:21.781 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:21.781 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:21:21.906 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:21:21.907 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:21:21.907 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:21.907 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:21.907 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:21.907 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:21:21.907 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:21.907 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:21.907 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:21.907 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:21:21.907 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:21.908 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:21.908 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:21.910 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:21.910 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:21.910 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:21.966 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 14 2024-09-11T06:21:21.966 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:21.966 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:21.966 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:21.966 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=58, tag_tid=11, entry_tid=7774], [object_number=57, tag_tid=11, entry_tid=7773], [object_number=56, tag_tid=11, entry_tid=7772], [object_number=59, tag_tid=11, entry_tid=7771]]], state=connected] 2024-09-11T06:21:21.969 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:21.971 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:21.972 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:21:21.972 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:21.972 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:21.972 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:21.972 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:21:21.972 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:21.972 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:21.972 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:21.972 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:21:21.972 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:21.973 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:21.973 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:21.973 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:21.973 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:21.973 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:22.021 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 14 2024-09-11T06:21:22.021 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:22.022 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:22.022 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:22.022 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=58, tag_tid=11, entry_tid=7774], [object_number=57, tag_tid=11, entry_tid=7773], [object_number=56, tag_tid=11, entry_tid=7772], [object_number=59, tag_tid=11, entry_tid=7771]]], state=connected] 2024-09-11T06:21:22.025 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:22.026 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=58, tag_tid=11, entry_tid=7774' 2024-09-11T06:21:22.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=58, tag_tid=11, entry_tid=7774' 2024-09-11T06:21:22.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=58, tag_tid=11, entry_tid=7774' '!=' 'object_number=59, tag_tid=11, entry_tid=7667' 2024-09-11T06:21:22.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:21:22.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=58, tag_tid=11, entry_tid=7774' = 'object_number=59, tag_tid=11, entry_tid=7667' 2024-09-11T06:21:22.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=58, tag_tid=11, entry_tid=7774' 2024-09-11T06:21:22.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:22.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:22.028 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:22.028 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=12 2024-09-11T06:21:22.029 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=58, tag_tid=11, entry_tid=7774' 2024-09-11T06:21:22.029 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:22.029 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:22.030 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=11 2024-09-11T06:21:22.030 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:22.030 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:22.031 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:22.031 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:21:22.032 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=58, tag_tid=11, entry_tid=7774' 2024-09-11T06:21:22.032 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:22.032 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:22.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=7774 2024-09-11T06:21:22.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 12 = 11 -a 1 -le 7774 2024-09-11T06:21:22.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:21:22.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:21:22.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:21:22.236 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:21:22.236 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:22.236 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:21:22.236 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:21:22.236 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:21:22.236 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:21:22.236 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:21:22.236 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:21:22.236 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:21:22.236 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:22.236 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:22.237 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:21:22.237 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:21:22.237 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:21:22.237 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:21:22.237 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:21:22.237 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:21:22.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:21:22.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:21:22.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:21:22.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:22.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:21:22.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:21:22.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:21:22.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:21:22.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:21:22.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:21:22.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:21:22.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:22.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:21:22.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:22.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:21:22.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:22.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:21:22.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:21:22.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:21:22.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:21:22.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:21:22.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:21:22.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:21:22.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:21:22.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:22.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:22.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:22.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:21:22.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:21:22.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:21:22.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:21:22.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:21:22.265 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:22.266 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:22.266 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:22.266 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:21:22.395 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:21:22.396 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:21:22.396 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:22.396 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:22.396 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:22.396 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:21:22.396 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:22.396 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:22.397 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:22.397 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:21:22.397 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:22.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:22.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:22.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:22.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:22.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:22.447 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 14 2024-09-11T06:21:22.447 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:22.448 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:22.448 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:22.448 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=57, tag_tid=11, entry_tid=7877], [object_number=56, tag_tid=11, entry_tid=7876], [object_number=59, tag_tid=11, entry_tid=7875], [object_number=58, tag_tid=11, entry_tid=7874]]], state=connected] 2024-09-11T06:21:22.451 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:22.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:22.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:21:22.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:22.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:22.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:22.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:21:22.454 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:22.454 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:22.454 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:22.454 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:21:22.454 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:22.454 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:22.454 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:22.454 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:22.455 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:22.455 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:22.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 14 2024-09-11T06:21:22.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:22.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:22.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:22.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=57, tag_tid=11, entry_tid=7877], [object_number=56, tag_tid=11, entry_tid=7876], [object_number=59, tag_tid=11, entry_tid=7875], [object_number=58, tag_tid=11, entry_tid=7874]]], state=connected] 2024-09-11T06:21:22.503 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:22.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=57, tag_tid=11, entry_tid=7877' 2024-09-11T06:21:22.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=57, tag_tid=11, entry_tid=7877' 2024-09-11T06:21:22.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=57, tag_tid=11, entry_tid=7877' '!=' 'object_number=58, tag_tid=11, entry_tid=7774' 2024-09-11T06:21:22.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:21:22.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=57, tag_tid=11, entry_tid=7877' = 'object_number=58, tag_tid=11, entry_tid=7774' 2024-09-11T06:21:22.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=57, tag_tid=11, entry_tid=7877' 2024-09-11T06:21:22.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:22.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:22.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:22.507 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=12 2024-09-11T06:21:22.507 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=57, tag_tid=11, entry_tid=7877' 2024-09-11T06:21:22.507 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:22.508 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:22.508 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=11 2024-09-11T06:21:22.509 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:22.509 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:22.509 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:22.510 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:21:22.511 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=57, tag_tid=11, entry_tid=7877' 2024-09-11T06:21:22.511 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:22.511 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:22.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=7877 2024-09-11T06:21:22.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 12 = 11 -a 1 -le 7877 2024-09-11T06:21:22.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:21:22.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:21:22.513 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:21:22.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:21:22.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:22.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:21:22.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:21:22.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:21:22.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:21:22.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:21:22.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:21:22.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:21:22.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:22.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:22.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:21:22.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:21:22.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:21:22.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:21:22.716 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:21:22.716 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:21:22.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:21:22.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:21:22.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:21:22.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:22.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:21:22.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:21:22.721 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:21:22.721 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:21:22.721 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:21:22.721 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:21:22.721 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:21:22.721 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:22.721 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:21:22.721 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:22.721 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:21:22.721 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:22.721 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:21:22.721 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:21:22.721 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:21:22.721 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:21:22.722 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:21:22.722 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:21:22.722 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:21:22.722 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:21:22.722 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:22.722 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:22.722 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:22.722 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:21:22.722 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:21:22.722 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:21:22.722 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:21:22.722 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:21:22.745 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:22.746 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:22.746 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:22.746 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:21:22.883 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:21:22.884 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:21:22.884 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:22.884 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:22.884 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:22.884 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:21:22.884 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:22.884 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:22.885 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:22.885 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:21:22.885 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:22.885 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:22.885 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:22.887 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:22.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:22.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:22.939 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 14 2024-09-11T06:21:22.939 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:22.939 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:22.939 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:22.939 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=57, tag_tid=11, entry_tid=7973], [object_number=56, tag_tid=11, entry_tid=7972], [object_number=59, tag_tid=11, entry_tid=7971], [object_number=58, tag_tid=11, entry_tid=7970]]], state=connected] 2024-09-11T06:21:22.942 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:22.943 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:22.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:21:22.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:22.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:22.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:22.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:21:22.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:22.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:22.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:22.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:21:22.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:22.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:22.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:22.946 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:22.946 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:22.946 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:22.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 14 2024-09-11T06:21:22.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:22.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:22.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:22.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=57, tag_tid=11, entry_tid=7973], [object_number=56, tag_tid=11, entry_tid=7972], [object_number=59, tag_tid=11, entry_tid=7971], [object_number=58, tag_tid=11, entry_tid=7970]]], state=connected] 2024-09-11T06:21:22.996 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:22.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=57, tag_tid=11, entry_tid=7973' 2024-09-11T06:21:22.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=57, tag_tid=11, entry_tid=7973' 2024-09-11T06:21:22.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=57, tag_tid=11, entry_tid=7973' '!=' 'object_number=57, tag_tid=11, entry_tid=7877' 2024-09-11T06:21:22.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:21:22.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=57, tag_tid=11, entry_tid=7973' = 'object_number=57, tag_tid=11, entry_tid=7877' 2024-09-11T06:21:22.999 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=57, tag_tid=11, entry_tid=7973' 2024-09-11T06:21:22.999 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:22.999 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:22.999 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:23.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=12 2024-09-11T06:21:23.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=57, tag_tid=11, entry_tid=7973' 2024-09-11T06:21:23.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:23.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:23.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=11 2024-09-11T06:21:23.002 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:23.002 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:23.002 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:23.003 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:21:23.004 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=57, tag_tid=11, entry_tid=7973' 2024-09-11T06:21:23.004 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:23.004 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:23.005 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=7973 2024-09-11T06:21:23.005 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 12 = 11 -a 1 -le 7973 2024-09-11T06:21:23.005 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:21:23.005 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:21:23.005 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:21:23.207 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:21:23.207 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:23.207 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:21:23.207 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:21:23.207 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:21:23.207 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:21:23.207 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:21:23.207 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:21:23.207 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:21:23.207 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:23.207 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:23.208 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:21:23.208 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:21:23.208 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:21:23.208 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:21:23.208 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:21:23.209 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:21:23.211 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:21:23.211 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:21:23.211 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:21:23.211 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:23.211 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:21:23.211 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:21:23.211 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:21:23.211 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:21:23.211 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:21:23.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:21:23.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:21:23.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:23.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:21:23.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:23.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:21:23.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:23.214 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:21:23.214 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:21:23.214 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:21:23.214 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:21:23.214 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:21:23.214 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:21:23.214 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:21:23.214 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:21:23.214 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:23.214 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:23.214 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:23.214 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:21:23.215 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:21:23.215 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:21:23.215 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:21:23.215 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:21:23.237 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:23.238 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:23.238 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:23.238 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:21:23.368 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:21:23.369 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:21:23.369 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:23.369 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:23.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:23.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:21:23.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:23.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:23.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:23.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:21:23.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:23.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:23.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:23.373 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:23.373 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:23.374 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:23.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 14 2024-09-11T06:21:23.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:23.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:23.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:23.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=58, tag_tid=11, entry_tid=8074], [object_number=57, tag_tid=11, entry_tid=8073], [object_number=56, tag_tid=11, entry_tid=8072], [object_number=59, tag_tid=11, entry_tid=8071]]], state=connected] 2024-09-11T06:21:23.422 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:23.424 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:23.425 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:21:23.425 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:23.425 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:23.425 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:23.425 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:21:23.425 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:23.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:23.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:23.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:21:23.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:23.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:23.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:23.427 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:23.427 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:23.427 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:23.472 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 14 2024-09-11T06:21:23.472 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:23.472 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:23.472 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:23.473 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=58, tag_tid=11, entry_tid=8074], [object_number=57, tag_tid=11, entry_tid=8073], [object_number=56, tag_tid=11, entry_tid=8072], [object_number=59, tag_tid=11, entry_tid=8071]]], state=connected] 2024-09-11T06:21:23.476 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:23.477 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=58, tag_tid=11, entry_tid=8074' 2024-09-11T06:21:23.477 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=58, tag_tid=11, entry_tid=8074' 2024-09-11T06:21:23.477 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=58, tag_tid=11, entry_tid=8074' '!=' 'object_number=57, tag_tid=11, entry_tid=7973' 2024-09-11T06:21:23.478 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:21:23.478 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=58, tag_tid=11, entry_tid=8074' = 'object_number=57, tag_tid=11, entry_tid=7973' 2024-09-11T06:21:23.478 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=58, tag_tid=11, entry_tid=8074' 2024-09-11T06:21:23.478 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:23.478 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:23.478 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:23.479 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=12 2024-09-11T06:21:23.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=58, tag_tid=11, entry_tid=8074' 2024-09-11T06:21:23.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:23.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:23.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=11 2024-09-11T06:21:23.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:23.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:23.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:23.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:21:23.483 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=58, tag_tid=11, entry_tid=8074' 2024-09-11T06:21:23.483 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:23.483 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:23.484 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=8074 2024-09-11T06:21:23.484 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 12 = 11 -a 1 -le 8074 2024-09-11T06:21:23.484 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:21:23.484 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:21:23.484 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:21:23.686 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:21:23.686 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:23.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:21:23.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:21:23.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:21:23.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:21:23.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:21:23.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:21:23.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:21:23.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:23.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:23.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:21:23.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:21:23.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:21:23.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:21:23.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:21:23.690 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:21:23.690 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:21:23.690 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:21:23.690 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:21:23.690 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:23.690 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:21:23.690 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:21:23.690 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:21:23.690 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:21:23.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:21:23.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:21:23.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:21:23.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:23.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:21:23.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:23.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:21:23.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:23.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:21:23.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:21:23.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:21:23.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:21:23.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:21:23.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:21:23.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:21:23.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:21:23.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:23.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:23.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:23.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:21:23.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:21:23.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:21:23.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:21:23.693 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:21:23.716 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:23.716 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:23.716 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:23.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:21:23.846 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:21:23.848 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:21:23.848 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:23.848 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:23.848 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:23.848 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:21:23.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:23.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:23.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:23.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:21:23.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:23.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:23.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:23.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:23.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:23.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:23.897 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 14 2024-09-11T06:21:23.897 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:23.897 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:23.897 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:23.897 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=58, tag_tid=11, entry_tid=8166], [object_number=57, tag_tid=11, entry_tid=8165], [object_number=56, tag_tid=11, entry_tid=8164], [object_number=59, tag_tid=11, entry_tid=8163]]], state=connected] 2024-09-11T06:21:23.900 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:23.901 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:23.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:21:23.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:23.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:23.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:23.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:21:23.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:23.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:23.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:23.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:21:23.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:23.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:23.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:23.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:23.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:23.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:23.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 14 2024-09-11T06:21:23.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:23.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:23.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:23.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=58, tag_tid=11, entry_tid=8166], [object_number=57, tag_tid=11, entry_tid=8165], [object_number=56, tag_tid=11, entry_tid=8164], [object_number=59, tag_tid=11, entry_tid=8163]]], state=connected] 2024-09-11T06:21:23.954 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:23.956 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=58, tag_tid=11, entry_tid=8166' 2024-09-11T06:21:23.956 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=58, tag_tid=11, entry_tid=8166' 2024-09-11T06:21:23.956 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=58, tag_tid=11, entry_tid=8166' '!=' 'object_number=58, tag_tid=11, entry_tid=8074' 2024-09-11T06:21:23.956 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:21:23.956 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=58, tag_tid=11, entry_tid=8166' = 'object_number=58, tag_tid=11, entry_tid=8074' 2024-09-11T06:21:23.957 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=58, tag_tid=11, entry_tid=8166' 2024-09-11T06:21:23.957 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:23.957 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:23.957 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:23.958 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=12 2024-09-11T06:21:23.958 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=58, tag_tid=11, entry_tid=8166' 2024-09-11T06:21:23.958 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:23.959 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:23.959 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=11 2024-09-11T06:21:23.960 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:23.960 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:23.960 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:23.960 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:21:23.961 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=58, tag_tid=11, entry_tid=8166' 2024-09-11T06:21:23.961 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:23.961 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:23.962 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=8166 2024-09-11T06:21:23.962 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 12 = 11 -a 1 -le 8166 2024-09-11T06:21:23.963 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:21:23.963 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:21:23.963 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:21:24.164 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:21:24.164 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:24.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:21:24.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:21:24.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:21:24.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:21:24.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:21:24.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:21:24.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:21:24.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:24.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:24.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:21:24.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:21:24.166 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:21:24.166 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:21:24.166 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:21:24.166 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:21:24.169 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:21:24.169 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:21:24.170 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:21:24.170 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:24.170 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:21:24.170 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:21:24.170 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:21:24.170 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:21:24.170 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:21:24.170 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:21:24.170 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:21:24.170 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:24.170 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:21:24.170 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:24.171 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:21:24.171 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:24.171 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:21:24.171 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:21:24.171 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:21:24.171 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:21:24.171 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:21:24.171 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:21:24.171 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:21:24.171 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:21:24.171 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:24.171 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:24.172 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:24.172 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:21:24.172 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:21:24.172 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:21:24.172 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:21:24.172 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:21:24.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:24.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:24.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:24.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:21:24.323 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:21:24.324 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:21:24.325 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:24.325 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:24.325 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:24.325 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:21:24.325 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:24.325 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:24.325 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:24.325 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:21:24.326 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:24.326 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:24.326 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:24.328 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:24.328 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:24.328 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:24.374 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 14 2024-09-11T06:21:24.374 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:24.375 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:24.375 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:24.375 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=56, tag_tid=11, entry_tid=8256], [object_number=59, tag_tid=11, entry_tid=8255], [object_number=58, tag_tid=11, entry_tid=8254], [object_number=57, tag_tid=11, entry_tid=8253]]], state=connected] 2024-09-11T06:21:24.378 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:24.379 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:24.380 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:21:24.380 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:24.381 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:24.381 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:24.381 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:21:24.381 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:24.381 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:24.381 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:24.381 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:21:24.381 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:24.381 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:24.381 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:24.382 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:24.382 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:24.382 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:24.634 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 14 2024-09-11T06:21:24.634 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:24.634 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:24.634 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:24.635 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=56, tag_tid=11, entry_tid=8256], [object_number=59, tag_tid=11, entry_tid=8255], [object_number=58, tag_tid=11, entry_tid=8254], [object_number=57, tag_tid=11, entry_tid=8253]]], state=connected] 2024-09-11T06:21:24.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:24.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=56, tag_tid=11, entry_tid=8256' 2024-09-11T06:21:24.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=56, tag_tid=11, entry_tid=8256' 2024-09-11T06:21:24.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=56, tag_tid=11, entry_tid=8256' '!=' 'object_number=58, tag_tid=11, entry_tid=8166' 2024-09-11T06:21:24.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:21:24.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=56, tag_tid=11, entry_tid=8256' = 'object_number=58, tag_tid=11, entry_tid=8166' 2024-09-11T06:21:24.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=56, tag_tid=11, entry_tid=8256' 2024-09-11T06:21:24.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:24.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:24.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:24.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=12 2024-09-11T06:21:24.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=56, tag_tid=11, entry_tid=8256' 2024-09-11T06:21:24.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:24.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:24.645 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=11 2024-09-11T06:21:24.646 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:24.646 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:24.646 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:24.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:21:24.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=56, tag_tid=11, entry_tid=8256' 2024-09-11T06:21:24.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:24.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:24.650 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=8256 2024-09-11T06:21:24.650 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 12 = 11 -a 1 -le 8256 2024-09-11T06:21:24.650 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:21:24.650 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:21:24.650 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:21:24.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:21:24.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:24.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:21:24.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:21:24.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:21:24.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:21:24.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:21:24.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:21:24.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:21:24.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:24.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:24.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:21:24.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:21:24.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:21:24.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:21:24.854 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:21:24.854 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:21:24.855 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:21:24.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:21:24.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:21:24.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:24.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:21:24.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:21:24.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:21:24.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:21:24.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:21:24.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:21:24.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:21:24.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:24.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:21:24.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:24.857 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:21:24.857 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:24.857 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:21:24.858 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:21:24.858 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:21:24.858 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:21:24.858 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:21:24.858 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:21:24.858 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:21:24.858 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:21:24.858 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:24.858 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:24.858 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:24.858 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:21:24.858 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:21:24.858 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:21:24.858 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:21:24.859 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:21:24.882 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:24.883 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:24.883 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:24.883 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:21:25.020 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:21:25.022 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:21:25.022 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:25.022 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:25.022 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:25.022 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:21:25.022 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:25.022 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:25.022 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:25.022 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:21:25.022 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:25.023 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:25.023 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:25.025 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:25.025 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:25.025 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:25.071 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 14 2024-09-11T06:21:25.072 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:25.072 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:25.072 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:25.072 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=56, tag_tid=11, entry_tid=8380], [object_number=59, tag_tid=11, entry_tid=8379], [object_number=58, tag_tid=11, entry_tid=8378], [object_number=57, tag_tid=11, entry_tid=8377]]], state=connected] 2024-09-11T06:21:25.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:25.076 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:25.077 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:21:25.077 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:25.077 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:25.077 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:25.077 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:21:25.077 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:25.077 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:25.077 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:25.078 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:21:25.078 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:25.078 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:25.078 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:25.078 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:25.079 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:25.079 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:25.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 14 2024-09-11T06:21:25.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:25.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:25.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:25.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=56, tag_tid=11, entry_tid=8380], [object_number=59, tag_tid=11, entry_tid=8379], [object_number=58, tag_tid=11, entry_tid=8378], [object_number=57, tag_tid=11, entry_tid=8377]]], state=connected] 2024-09-11T06:21:25.128 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:25.130 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=56, tag_tid=11, entry_tid=8380' 2024-09-11T06:21:25.130 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=56, tag_tid=11, entry_tid=8380' 2024-09-11T06:21:25.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=56, tag_tid=11, entry_tid=8380' '!=' 'object_number=56, tag_tid=11, entry_tid=8256' 2024-09-11T06:21:25.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:21:25.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=56, tag_tid=11, entry_tid=8380' = 'object_number=56, tag_tid=11, entry_tid=8256' 2024-09-11T06:21:25.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=56, tag_tid=11, entry_tid=8380' 2024-09-11T06:21:25.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:25.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:25.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:25.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=12 2024-09-11T06:21:25.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=56, tag_tid=11, entry_tid=8380' 2024-09-11T06:21:25.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:25.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:25.133 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=11 2024-09-11T06:21:25.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:25.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:25.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:25.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:21:25.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=56, tag_tid=11, entry_tid=8380' 2024-09-11T06:21:25.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:25.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:25.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=8380 2024-09-11T06:21:25.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 12 = 11 -a 1 -le 8380 2024-09-11T06:21:25.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:21:25.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:21:25.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:21:25.339 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:21:25.339 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:25.339 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:21:25.339 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:21:25.339 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:21:25.339 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:21:25.339 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:21:25.339 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:21:25.340 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:21:25.340 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:25.340 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:25.340 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:21:25.340 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:21:25.340 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:21:25.340 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:21:25.340 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:21:25.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:21:25.343 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:21:25.344 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:21:25.344 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:21:25.344 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:25.344 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:21:25.344 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:21:25.344 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:21:25.344 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:21:25.344 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:21:25.344 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:21:25.344 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:21:25.344 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:25.344 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:21:25.345 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:25.345 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:21:25.345 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:25.345 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:21:25.346 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:21:25.346 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:21:25.346 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:21:25.346 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:21:25.346 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:21:25.346 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:21:25.346 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:21:25.346 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:25.346 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:25.346 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:25.346 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:21:25.346 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:21:25.346 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:21:25.347 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:21:25.347 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:21:25.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:25.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:25.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:25.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:21:25.514 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:21:25.516 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:21:25.516 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:25.516 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:25.516 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:25.516 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:21:25.516 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:25.516 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:25.516 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:25.516 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:21:25.517 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:25.517 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:25.517 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:25.519 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:25.519 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:25.519 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:25.564 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 14 2024-09-11T06:21:25.565 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:25.565 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:25.565 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:25.565 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=56, tag_tid=11, entry_tid=8476], [object_number=59, tag_tid=11, entry_tid=8475], [object_number=58, tag_tid=11, entry_tid=8474], [object_number=57, tag_tid=11, entry_tid=8473]]], state=connected] 2024-09-11T06:21:25.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:25.569 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:25.570 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:21:25.570 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:25.570 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:25.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:25.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:21:25.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:25.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:25.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:25.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:21:25.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:25.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:25.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:25.572 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:25.572 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:25.572 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:25.618 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 14 2024-09-11T06:21:25.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:25.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:25.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:25.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=56, tag_tid=11, entry_tid=8476], [object_number=59, tag_tid=11, entry_tid=8475], [object_number=58, tag_tid=11, entry_tid=8474], [object_number=57, tag_tid=11, entry_tid=8473]]], state=connected] 2024-09-11T06:21:25.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:25.624 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=56, tag_tid=11, entry_tid=8476' 2024-09-11T06:21:25.624 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=56, tag_tid=11, entry_tid=8476' 2024-09-11T06:21:25.624 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=56, tag_tid=11, entry_tid=8476' '!=' 'object_number=56, tag_tid=11, entry_tid=8380' 2024-09-11T06:21:25.624 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:21:25.624 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=56, tag_tid=11, entry_tid=8476' = 'object_number=56, tag_tid=11, entry_tid=8380' 2024-09-11T06:21:25.624 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=56, tag_tid=11, entry_tid=8476' 2024-09-11T06:21:25.624 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:25.624 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:25.625 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:25.625 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=12 2024-09-11T06:21:25.626 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=56, tag_tid=11, entry_tid=8476' 2024-09-11T06:21:25.626 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:25.626 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:25.627 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=11 2024-09-11T06:21:25.627 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:25.627 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:25.628 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:25.628 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:21:25.629 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=56, tag_tid=11, entry_tid=8476' 2024-09-11T06:21:25.629 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:25.629 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:25.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=8476 2024-09-11T06:21:25.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 12 = 11 -a 1 -le 8476 2024-09-11T06:21:25.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:21:25.631 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:21:25.631 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:21:25.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:21:25.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:25.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:21:25.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:21:25.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:21:25.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:21:25.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:21:25.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:21:25.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:21:25.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:25.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:25.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:21:25.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:21:25.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:21:25.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:21:25.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:21:25.837 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:21:25.837 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:21:25.837 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:21:25.837 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:21:25.837 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:25.837 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:21:25.837 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:21:25.837 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:21:25.837 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:21:25.838 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:21:25.838 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:21:25.838 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:21:25.838 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:25.838 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:21:25.838 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:25.838 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:21:25.838 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:25.838 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:21:25.838 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:21:25.838 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:21:25.838 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:21:25.838 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:21:25.838 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:21:25.839 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:21:25.839 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:21:25.839 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:25.839 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:25.839 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:25.839 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:21:25.839 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:21:25.839 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:21:25.839 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:21:25.839 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:21:25.863 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:25.863 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:25.864 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:25.864 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:21:25.990 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:21:25.992 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:21:25.992 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:25.992 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:25.992 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:25.992 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:21:25.992 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:25.992 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:25.992 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:25.992 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:21:25.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:25.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:25.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:25.994 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:25.994 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:25.995 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:26.045 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 14 2024-09-11T06:21:26.045 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:26.045 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:26.045 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:26.045 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=58, tag_tid=11, entry_tid=8574], [object_number=57, tag_tid=11, entry_tid=8573], [object_number=56, tag_tid=11, entry_tid=8572], [object_number=59, tag_tid=11, entry_tid=8571]]], state=connected] 2024-09-11T06:21:26.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:26.051 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:26.051 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:21:26.051 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:26.051 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:26.052 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:26.052 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:21:26.052 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:26.052 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:26.052 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:26.052 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:21:26.052 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:26.052 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:26.052 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:26.053 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:26.053 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:26.053 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:26.099 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 14 2024-09-11T06:21:26.100 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:26.100 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:26.100 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:26.100 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=58, tag_tid=11, entry_tid=8574], [object_number=57, tag_tid=11, entry_tid=8573], [object_number=56, tag_tid=11, entry_tid=8572], [object_number=59, tag_tid=11, entry_tid=8571]]], state=connected] 2024-09-11T06:21:26.103 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:26.105 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=58, tag_tid=11, entry_tid=8574' 2024-09-11T06:21:26.105 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=58, tag_tid=11, entry_tid=8574' 2024-09-11T06:21:26.105 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=58, tag_tid=11, entry_tid=8574' '!=' 'object_number=56, tag_tid=11, entry_tid=8476' 2024-09-11T06:21:26.105 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:21:26.106 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=58, tag_tid=11, entry_tid=8574' = 'object_number=56, tag_tid=11, entry_tid=8476' 2024-09-11T06:21:26.106 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=58, tag_tid=11, entry_tid=8574' 2024-09-11T06:21:26.106 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:26.106 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:26.106 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:26.106 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=12 2024-09-11T06:21:26.107 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=58, tag_tid=11, entry_tid=8574' 2024-09-11T06:21:26.107 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:26.107 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:26.108 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=11 2024-09-11T06:21:26.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:26.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:26.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:26.110 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:21:26.110 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=58, tag_tid=11, entry_tid=8574' 2024-09-11T06:21:26.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:26.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:26.112 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=8574 2024-09-11T06:21:26.112 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 12 = 11 -a 1 -le 8574 2024-09-11T06:21:26.112 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:21:26.112 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:21:26.112 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:21:26.314 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:21:26.314 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:26.314 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:21:26.314 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:21:26.314 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:21:26.314 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:21:26.314 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:21:26.314 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:21:26.314 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:21:26.314 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:26.315 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:26.315 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:21:26.315 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:21:26.315 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:21:26.315 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:21:26.315 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:21:26.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:21:26.318 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:21:26.318 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:21:26.318 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:21:26.318 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:26.319 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:21:26.319 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:21:26.319 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:21:26.319 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:21:26.319 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:21:26.319 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:21:26.319 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:21:26.319 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:26.319 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:21:26.319 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:26.319 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:21:26.319 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:26.320 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:21:26.320 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:21:26.320 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:21:26.320 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:21:26.321 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:21:26.321 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:21:26.321 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:21:26.321 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:21:26.321 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:26.321 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:26.321 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:26.321 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:21:26.321 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:21:26.321 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:21:26.321 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:21:26.321 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:21:26.345 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:26.345 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:26.345 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:26.345 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:21:26.472 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:21:26.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:21:26.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:26.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:26.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:26.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:21:26.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:26.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:26.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:26.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:21:26.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:26.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:26.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:26.476 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:26.476 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:26.477 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:26.528 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 14 2024-09-11T06:21:26.528 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:26.528 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:26.528 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:26.528 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=56, tag_tid=11, entry_tid=8672], [object_number=59, tag_tid=11, entry_tid=8671], [object_number=58, tag_tid=11, entry_tid=8670], [object_number=57, tag_tid=11, entry_tid=8669]]], state=connected] 2024-09-11T06:21:26.531 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:26.533 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:26.534 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:21:26.534 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:26.534 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:26.534 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:26.534 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:21:26.534 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:26.534 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:26.534 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:26.534 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:21:26.535 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:26.535 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:26.535 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:26.535 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:26.535 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:26.535 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:26.582 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 14 2024-09-11T06:21:26.582 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:26.582 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:26.582 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:26.582 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=56, tag_tid=11, entry_tid=8672], [object_number=59, tag_tid=11, entry_tid=8671], [object_number=58, tag_tid=11, entry_tid=8670], [object_number=57, tag_tid=11, entry_tid=8669]]], state=connected] 2024-09-11T06:21:26.585 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:26.587 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=56, tag_tid=11, entry_tid=8672' 2024-09-11T06:21:26.587 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=56, tag_tid=11, entry_tid=8672' 2024-09-11T06:21:26.588 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=56, tag_tid=11, entry_tid=8672' '!=' 'object_number=58, tag_tid=11, entry_tid=8574' 2024-09-11T06:21:26.588 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:21:26.588 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=56, tag_tid=11, entry_tid=8672' = 'object_number=58, tag_tid=11, entry_tid=8574' 2024-09-11T06:21:26.588 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=56, tag_tid=11, entry_tid=8672' 2024-09-11T06:21:26.588 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:26.588 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:26.588 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:26.588 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=12 2024-09-11T06:21:26.589 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=56, tag_tid=11, entry_tid=8672' 2024-09-11T06:21:26.589 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:26.589 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:26.590 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=11 2024-09-11T06:21:26.591 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:26.591 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:26.591 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:26.592 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:21:26.593 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=56, tag_tid=11, entry_tid=8672' 2024-09-11T06:21:26.593 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:26.593 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:26.594 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=8672 2024-09-11T06:21:26.594 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 12 = 11 -a 1 -le 8672 2024-09-11T06:21:26.594 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:21:26.594 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:21:26.594 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:21:26.796 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:21:26.796 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:26.796 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:21:26.796 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:21:26.796 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:21:26.796 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:21:26.796 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:21:26.796 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:21:26.797 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:21:26.797 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:26.797 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:26.797 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:21:26.797 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:21:26.797 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:21:26.797 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:21:26.797 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:21:26.798 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:21:26.800 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:21:26.800 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:21:26.800 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:21:26.801 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:26.801 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:21:26.801 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:21:26.801 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:21:26.801 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:21:26.801 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:21:26.801 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:21:26.801 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:21:26.801 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:26.801 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:21:26.801 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:26.801 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:21:26.801 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:26.802 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:21:26.802 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:21:26.802 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:21:26.802 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:21:26.802 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:21:26.802 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:21:26.802 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:21:26.802 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:21:26.802 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:26.802 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:26.802 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:26.802 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:21:26.802 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:21:26.802 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:21:26.802 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:21:26.803 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:21:26.826 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:26.826 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:26.826 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:26.826 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:21:26.956 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:21:26.958 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:21:26.958 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:26.958 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:26.958 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:26.958 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:21:26.958 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:26.958 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:26.958 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:26.958 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:21:26.959 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:26.959 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:26.959 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:26.961 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:26.962 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:26.963 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:27.008 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 14 2024-09-11T06:21:27.008 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:27.008 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:27.009 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:27.009 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=56, tag_tid=11, entry_tid=8764], [object_number=59, tag_tid=11, entry_tid=8763], [object_number=58, tag_tid=11, entry_tid=8762], [object_number=57, tag_tid=11, entry_tid=8761]]], state=connected] 2024-09-11T06:21:27.011 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:27.015 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:27.015 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:21:27.015 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:27.015 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:27.015 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:27.016 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:21:27.016 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:27.016 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:27.016 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:27.016 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:21:27.016 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:27.017 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:27.017 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:27.018 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:27.018 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:27.018 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:27.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 14 2024-09-11T06:21:27.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:27.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:27.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:27.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=56, tag_tid=11, entry_tid=8764], [object_number=59, tag_tid=11, entry_tid=8763], [object_number=58, tag_tid=11, entry_tid=8762], [object_number=57, tag_tid=11, entry_tid=8761]]], state=connected] 2024-09-11T06:21:27.064 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:27.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=56, tag_tid=11, entry_tid=8764' 2024-09-11T06:21:27.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=56, tag_tid=11, entry_tid=8764' 2024-09-11T06:21:27.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=56, tag_tid=11, entry_tid=8764' '!=' 'object_number=56, tag_tid=11, entry_tid=8672' 2024-09-11T06:21:27.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:21:27.067 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=56, tag_tid=11, entry_tid=8764' = 'object_number=56, tag_tid=11, entry_tid=8672' 2024-09-11T06:21:27.067 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=56, tag_tid=11, entry_tid=8764' 2024-09-11T06:21:27.067 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:27.067 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:27.067 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:27.067 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=12 2024-09-11T06:21:27.068 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=56, tag_tid=11, entry_tid=8764' 2024-09-11T06:21:27.068 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:27.068 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:27.069 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=11 2024-09-11T06:21:27.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:27.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:27.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:27.071 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:21:27.071 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=56, tag_tid=11, entry_tid=8764' 2024-09-11T06:21:27.071 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:27.071 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:27.073 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=8764 2024-09-11T06:21:27.073 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 12 = 11 -a 1 -le 8764 2024-09-11T06:21:27.073 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:21:27.073 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:21:27.073 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:21:27.275 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:21:27.275 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:27.275 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:21:27.275 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:21:27.275 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:21:27.275 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:21:27.275 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:21:27.275 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:21:27.275 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:21:27.276 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:27.276 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:27.276 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:21:27.276 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:21:27.276 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:21:27.276 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:21:27.276 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:21:27.276 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:21:27.279 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:21:27.279 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:21:27.279 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:21:27.279 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:27.279 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:21:27.279 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:21:27.280 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:21:27.280 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:21:27.280 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:21:27.280 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:21:27.280 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:21:27.280 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:27.280 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:21:27.280 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:27.280 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:21:27.280 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:27.281 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:21:27.281 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:21:27.281 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:21:27.281 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:21:27.281 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:21:27.281 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:21:27.281 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:21:27.282 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:21:27.282 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:27.282 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:27.282 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:27.282 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:21:27.282 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:21:27.282 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:21:27.283 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:21:27.283 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:21:27.305 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:27.305 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:27.305 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:27.306 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:21:27.438 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:21:27.439 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:21:27.439 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:27.439 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:27.440 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:27.440 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:21:27.440 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:27.440 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:27.440 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:27.440 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:21:27.440 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:27.441 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:27.441 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:27.441 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:27.442 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:27.442 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:27.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 14 2024-09-11T06:21:27.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:27.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:27.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:27.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=56, tag_tid=11, entry_tid=8856], [object_number=59, tag_tid=11, entry_tid=8855], [object_number=58, tag_tid=11, entry_tid=8854], [object_number=57, tag_tid=11, entry_tid=8853]]], state=connected] 2024-09-11T06:21:27.491 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:27.492 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:27.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:21:27.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:27.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:27.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:27.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:21:27.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:27.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:27.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:27.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:21:27.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:27.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:27.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:27.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:27.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:27.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:27.541 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 14 2024-09-11T06:21:27.541 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:27.541 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:27.541 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:27.541 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=56, tag_tid=11, entry_tid=8856], [object_number=59, tag_tid=11, entry_tid=8855], [object_number=58, tag_tid=11, entry_tid=8854], [object_number=57, tag_tid=11, entry_tid=8853]]], state=connected] 2024-09-11T06:21:27.544 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:27.545 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=56, tag_tid=11, entry_tid=8856' 2024-09-11T06:21:27.546 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=56, tag_tid=11, entry_tid=8856' 2024-09-11T06:21:27.546 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=56, tag_tid=11, entry_tid=8856' '!=' 'object_number=56, tag_tid=11, entry_tid=8764' 2024-09-11T06:21:27.546 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:21:27.546 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=56, tag_tid=11, entry_tid=8856' = 'object_number=56, tag_tid=11, entry_tid=8764' 2024-09-11T06:21:27.546 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=56, tag_tid=11, entry_tid=8856' 2024-09-11T06:21:27.546 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:27.546 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:27.547 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:27.547 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=12 2024-09-11T06:21:27.547 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=56, tag_tid=11, entry_tid=8856' 2024-09-11T06:21:27.548 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:27.548 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:27.549 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=11 2024-09-11T06:21:27.549 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:27.549 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:27.549 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:27.550 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:21:27.551 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=56, tag_tid=11, entry_tid=8856' 2024-09-11T06:21:27.551 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:27.551 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:27.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=8856 2024-09-11T06:21:27.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 12 = 11 -a 1 -le 8856 2024-09-11T06:21:27.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:21:27.553 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:21:27.553 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:21:27.754 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:21:27.754 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:27.754 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:21:27.754 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:21:27.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:21:27.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:21:27.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:21:27.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:21:27.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:21:27.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:27.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:27.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:21:27.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:21:27.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:21:27.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:21:27.758 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:21:27.758 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:21:27.758 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:21:27.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:21:27.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:21:27.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:27.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:21:27.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:21:27.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:21:27.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:21:27.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:21:27.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:21:27.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:21:27.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:27.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:21:27.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:27.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:21:27.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:27.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:21:27.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:21:27.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:21:27.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:21:27.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:21:27.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:21:27.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:21:27.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:21:27.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:27.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:27.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:27.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:21:27.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:21:27.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:21:27.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:21:27.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:21:27.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:27.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:27.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:27.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:21:27.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:21:27.922 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:21:27.922 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:27.922 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:27.922 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:27.922 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:21:27.922 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:27.922 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:27.922 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:27.922 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:21:27.923 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:27.923 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:27.923 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:27.925 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:27.926 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:27.926 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:27.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 14 2024-09-11T06:21:27.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:27.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:27.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:27.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=56, tag_tid=11, entry_tid=8952], [object_number=59, tag_tid=11, entry_tid=8951], [object_number=58, tag_tid=11, entry_tid=8950], [object_number=57, tag_tid=11, entry_tid=8949]]], state=connected] 2024-09-11T06:21:27.981 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:27.983 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:27.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:21:27.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:27.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:27.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:27.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:21:27.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:27.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:27.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:27.985 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:21:27.985 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:27.985 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:27.985 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:27.985 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:27.985 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:27.986 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:28.032 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 14 2024-09-11T06:21:28.032 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:28.032 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:28.032 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:28.032 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=56, tag_tid=11, entry_tid=8952], [object_number=59, tag_tid=11, entry_tid=8951], [object_number=58, tag_tid=11, entry_tid=8950], [object_number=57, tag_tid=11, entry_tid=8949]]], state=connected] 2024-09-11T06:21:28.035 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:28.037 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=56, tag_tid=11, entry_tid=8952' 2024-09-11T06:21:28.037 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=56, tag_tid=11, entry_tid=8952' 2024-09-11T06:21:28.037 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=56, tag_tid=11, entry_tid=8952' '!=' 'object_number=56, tag_tid=11, entry_tid=8856' 2024-09-11T06:21:28.037 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:21:28.038 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=56, tag_tid=11, entry_tid=8952' = 'object_number=56, tag_tid=11, entry_tid=8856' 2024-09-11T06:21:28.038 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=56, tag_tid=11, entry_tid=8952' 2024-09-11T06:21:28.038 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:28.038 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:28.038 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:28.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=12 2024-09-11T06:21:28.040 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=56, tag_tid=11, entry_tid=8952' 2024-09-11T06:21:28.040 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:28.040 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:28.041 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=11 2024-09-11T06:21:28.041 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:28.041 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:28.041 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:28.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:21:28.043 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=56, tag_tid=11, entry_tid=8952' 2024-09-11T06:21:28.043 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:28.043 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:28.044 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=8952 2024-09-11T06:21:28.045 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 12 = 11 -a 1 -le 8952 2024-09-11T06:21:28.045 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:21:28.045 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:21:28.045 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:21:28.246 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:21:28.246 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:28.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:21:28.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:21:28.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:21:28.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:21:28.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:21:28.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:21:28.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:21:28.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:28.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:28.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:21:28.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:21:28.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:21:28.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:21:28.248 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:21:28.248 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:21:28.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:21:28.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:21:28.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:21:28.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:28.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:21:28.252 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:21:28.252 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:21:28.252 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:21:28.252 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:21:28.252 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:21:28.252 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:21:28.252 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:28.252 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:21:28.252 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:28.252 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:21:28.252 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:28.252 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:21:28.252 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:21:28.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:21:28.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:21:28.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:21:28.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:21:28.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:21:28.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:21:28.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:28.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:28.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:28.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:21:28.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:21:28.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:21:28.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:21:28.254 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:21:28.277 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:28.277 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:28.278 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:28.278 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:21:28.408 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:21:28.409 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:21:28.409 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:28.409 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:28.409 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:28.409 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:21:28.409 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:28.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:28.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:28.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:21:28.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:28.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:28.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:28.413 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:28.413 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:28.413 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:28.461 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 14 2024-09-11T06:21:28.461 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:28.461 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:28.461 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:28.461 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=59, tag_tid=11, entry_tid=9059], [object_number=58, tag_tid=11, entry_tid=9058], [object_number=57, tag_tid=11, entry_tid=9057], [object_number=56, tag_tid=11, entry_tid=9056]]], state=connected] 2024-09-11T06:21:28.464 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:28.465 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:28.466 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:21:28.466 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:28.466 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:28.467 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:28.467 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:21:28.467 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:28.467 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:28.467 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:28.467 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:21:28.467 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:28.467 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:28.467 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:28.468 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:28.468 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:28.468 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:28.513 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 14 2024-09-11T06:21:28.513 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:28.513 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:28.513 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:28.514 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=59, tag_tid=11, entry_tid=9059], [object_number=58, tag_tid=11, entry_tid=9058], [object_number=57, tag_tid=11, entry_tid=9057], [object_number=56, tag_tid=11, entry_tid=9056]]], state=connected] 2024-09-11T06:21:28.517 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:28.519 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=59, tag_tid=11, entry_tid=9059' 2024-09-11T06:21:28.519 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=59, tag_tid=11, entry_tid=9059' 2024-09-11T06:21:28.519 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=59, tag_tid=11, entry_tid=9059' '!=' 'object_number=56, tag_tid=11, entry_tid=8952' 2024-09-11T06:21:28.519 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:21:28.519 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=59, tag_tid=11, entry_tid=9059' = 'object_number=56, tag_tid=11, entry_tid=8952' 2024-09-11T06:21:28.519 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=59, tag_tid=11, entry_tid=9059' 2024-09-11T06:21:28.520 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:28.520 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:28.520 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:28.520 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=12 2024-09-11T06:21:28.521 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=59, tag_tid=11, entry_tid=9059' 2024-09-11T06:21:28.521 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:28.521 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:28.522 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=11 2024-09-11T06:21:28.522 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:28.523 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:28.523 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:28.523 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:21:28.524 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=59, tag_tid=11, entry_tid=9059' 2024-09-11T06:21:28.524 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:28.524 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:28.526 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=9059 2024-09-11T06:21:28.526 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 12 = 11 -a 1 -le 9059 2024-09-11T06:21:28.526 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:21:28.526 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:21:28.526 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:21:28.727 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:21:28.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:28.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:21:28.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:21:28.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:21:28.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:21:28.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:21:28.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:21:28.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:21:28.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:28.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:28.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:21:28.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:21:28.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:21:28.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:21:28.729 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:21:28.729 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:21:28.732 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:21:28.732 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:21:28.732 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:21:28.732 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:28.732 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:21:28.732 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:21:28.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:21:28.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:21:28.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:21:28.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:21:28.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:21:28.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:28.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:21:28.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:28.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:21:28.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:28.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:21:28.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:21:28.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:21:28.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:21:28.734 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:21:28.734 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:21:28.734 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:21:28.734 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:21:28.734 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:28.734 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:28.734 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:28.734 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:21:28.734 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:21:28.734 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:21:28.734 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:21:28.735 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:21:28.758 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:28.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:28.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:28.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:21:28.891 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:21:28.892 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:21:28.892 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:28.892 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:28.892 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:28.892 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:21:28.892 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:28.892 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:28.892 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:28.892 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:21:28.893 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:28.893 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:28.893 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:28.894 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:28.895 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:28.895 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:28.940 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 14 2024-09-11T06:21:28.941 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:28.941 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:28.941 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:28.941 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=58, tag_tid=11, entry_tid=9162], [object_number=57, tag_tid=11, entry_tid=9161], [object_number=56, tag_tid=11, entry_tid=9160], [object_number=59, tag_tid=11, entry_tid=9159]]], state=connected] 2024-09-11T06:21:28.944 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:28.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:28.947 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:21:28.947 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:28.947 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:28.947 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:28.947 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:21:28.947 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:28.947 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:28.947 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:28.947 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:21:28.947 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:28.947 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:28.947 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:28.948 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:28.948 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:28.948 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:28.994 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 14 2024-09-11T06:21:28.994 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:28.994 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:28.994 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:28.994 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=58, tag_tid=11, entry_tid=9162], [object_number=57, tag_tid=11, entry_tid=9161], [object_number=56, tag_tid=11, entry_tid=9160], [object_number=59, tag_tid=11, entry_tid=9159]]], state=connected] 2024-09-11T06:21:28.997 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:29.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=58, tag_tid=11, entry_tid=9162' 2024-09-11T06:21:29.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=58, tag_tid=11, entry_tid=9162' 2024-09-11T06:21:29.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=58, tag_tid=11, entry_tid=9162' '!=' 'object_number=59, tag_tid=11, entry_tid=9059' 2024-09-11T06:21:29.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:21:29.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=58, tag_tid=11, entry_tid=9162' = 'object_number=59, tag_tid=11, entry_tid=9059' 2024-09-11T06:21:29.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=58, tag_tid=11, entry_tid=9162' 2024-09-11T06:21:29.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:29.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:29.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:29.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=12 2024-09-11T06:21:29.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=58, tag_tid=11, entry_tid=9162' 2024-09-11T06:21:29.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:29.002 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:29.002 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=11 2024-09-11T06:21:29.003 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:29.003 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:29.003 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:29.004 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:21:29.005 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=58, tag_tid=11, entry_tid=9162' 2024-09-11T06:21:29.005 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:29.005 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:29.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=9162 2024-09-11T06:21:29.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 12 = 11 -a 1 -le 9162 2024-09-11T06:21:29.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:21:29.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:21:29.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:21:29.208 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:21:29.208 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:29.208 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:21:29.208 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:21:29.208 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:21:29.208 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:21:29.209 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:21:29.209 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:21:29.209 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:21:29.209 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:29.209 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:29.209 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:21:29.209 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:21:29.209 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:21:29.209 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:21:29.210 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:21:29.210 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:21:29.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:21:29.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:21:29.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:21:29.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:29.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:21:29.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:21:29.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:21:29.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:21:29.213 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:21:29.213 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:21:29.213 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:21:29.213 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:29.213 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:21:29.213 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:29.213 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:21:29.213 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:29.214 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:21:29.215 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:21:29.215 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:21:29.215 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:21:29.215 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:21:29.215 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:21:29.215 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:21:29.215 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:21:29.215 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:29.215 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:29.215 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:29.215 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:21:29.215 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:21:29.215 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:21:29.215 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:21:29.216 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:21:29.239 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:29.239 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:29.239 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:29.239 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:21:29.368 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:21:29.369 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:21:29.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:29.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:29.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:29.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:21:29.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:29.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:29.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:29.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:21:29.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:29.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:29.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:29.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:29.372 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:29.372 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:29.428 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 14 2024-09-11T06:21:29.428 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:29.428 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:29.428 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:29.429 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=56, tag_tid=11, entry_tid=9256], [object_number=59, tag_tid=11, entry_tid=9255], [object_number=58, tag_tid=11, entry_tid=9254], [object_number=57, tag_tid=11, entry_tid=9253]]], state=connected] 2024-09-11T06:21:29.432 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:29.433 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:29.434 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:21:29.435 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:29.435 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:29.435 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:29.435 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:21:29.435 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:29.435 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:29.435 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:29.435 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:21:29.435 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:29.435 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:29.435 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:29.436 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:29.436 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:29.436 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:29.483 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 14 2024-09-11T06:21:29.484 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:29.484 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:29.484 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:29.484 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=56, tag_tid=11, entry_tid=9256], [object_number=59, tag_tid=11, entry_tid=9255], [object_number=58, tag_tid=11, entry_tid=9254], [object_number=57, tag_tid=11, entry_tid=9253]]], state=connected] 2024-09-11T06:21:29.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:29.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=56, tag_tid=11, entry_tid=9256' 2024-09-11T06:21:29.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=56, tag_tid=11, entry_tid=9256' 2024-09-11T06:21:29.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=56, tag_tid=11, entry_tid=9256' '!=' 'object_number=58, tag_tid=11, entry_tid=9162' 2024-09-11T06:21:29.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:21:29.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=56, tag_tid=11, entry_tid=9256' = 'object_number=58, tag_tid=11, entry_tid=9162' 2024-09-11T06:21:29.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=56, tag_tid=11, entry_tid=9256' 2024-09-11T06:21:29.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:29.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:29.490 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:29.490 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=12 2024-09-11T06:21:29.491 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=56, tag_tid=11, entry_tid=9256' 2024-09-11T06:21:29.491 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:29.491 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:29.492 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=11 2024-09-11T06:21:29.492 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:29.493 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:29.493 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:29.493 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:21:29.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=56, tag_tid=11, entry_tid=9256' 2024-09-11T06:21:29.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:29.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:29.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=9256 2024-09-11T06:21:29.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 12 = 11 -a 1 -le 9256 2024-09-11T06:21:29.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:21:29.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:21:29.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:21:29.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:21:29.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:29.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:21:29.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:21:29.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:21:29.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:21:29.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:21:29.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:21:29.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:21:29.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:29.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:29.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:21:29.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:21:29.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:21:29.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:21:29.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:21:29.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:21:29.702 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:21:29.702 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:21:29.702 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:21:29.702 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:29.702 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:21:29.702 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:21:29.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:21:29.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:21:29.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:21:29.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:21:29.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:21:29.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:29.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:21:29.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:29.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:21:29.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:29.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:21:29.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:21:29.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:21:29.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:21:29.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:21:29.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:21:29.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:21:29.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:21:29.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:29.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:29.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:29.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:21:29.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:21:29.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:21:29.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:21:29.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:21:29.727 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:29.727 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:29.727 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:29.727 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:21:29.862 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:21:29.862 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:21:29.862 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:29.862 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:29.862 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:29.862 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:21:29.862 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:29.862 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:29.863 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:29.863 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:21:29.863 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:29.863 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:29.863 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:29.863 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:29.864 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:29.864 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:30.113 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 14 2024-09-11T06:21:30.113 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:30.113 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:30.113 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:30.113 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=56, tag_tid=11, entry_tid=9352], [object_number=59, tag_tid=11, entry_tid=9351], [object_number=58, tag_tid=11, entry_tid=9350], [object_number=57, tag_tid=11, entry_tid=9349]]], state=connected] 2024-09-11T06:21:30.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:30.120 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:30.121 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:21:30.121 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:30.121 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:30.121 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:30.122 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:21:30.122 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:30.122 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:30.122 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:30.122 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:21:30.122 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:30.122 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:30.123 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:30.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:30.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:30.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:30.181 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 14 2024-09-11T06:21:30.181 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:30.181 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:30.182 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:30.182 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=56, tag_tid=11, entry_tid=9352], [object_number=59, tag_tid=11, entry_tid=9351], [object_number=58, tag_tid=11, entry_tid=9350], [object_number=57, tag_tid=11, entry_tid=9349]]], state=connected] 2024-09-11T06:21:30.185 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:30.187 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=56, tag_tid=11, entry_tid=9352' 2024-09-11T06:21:30.187 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=56, tag_tid=11, entry_tid=9352' 2024-09-11T06:21:30.187 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=56, tag_tid=11, entry_tid=9352' '!=' 'object_number=56, tag_tid=11, entry_tid=9256' 2024-09-11T06:21:30.188 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:21:30.188 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=56, tag_tid=11, entry_tid=9352' = 'object_number=56, tag_tid=11, entry_tid=9256' 2024-09-11T06:21:30.188 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=56, tag_tid=11, entry_tid=9352' 2024-09-11T06:21:30.188 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:30.188 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:30.188 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:30.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=12 2024-09-11T06:21:30.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=56, tag_tid=11, entry_tid=9352' 2024-09-11T06:21:30.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:30.190 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:30.190 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=11 2024-09-11T06:21:30.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:30.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:30.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:30.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:21:30.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=56, tag_tid=11, entry_tid=9352' 2024-09-11T06:21:30.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:30.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:30.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=9352 2024-09-11T06:21:30.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 12 = 11 -a 1 -le 9352 2024-09-11T06:21:30.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:21:30.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:21:30.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:21:30.396 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:21:30.397 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:30.397 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:21:30.397 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:21:30.397 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:21:30.397 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:21:30.397 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:21:30.397 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:21:30.397 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:21:30.397 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:30.397 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:30.397 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:21:30.398 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:21:30.398 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:21:30.398 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:21:30.398 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:21:30.398 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:21:30.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:21:30.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:21:30.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:21:30.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:30.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:21:30.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:21:30.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:21:30.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:21:30.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:21:30.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:21:30.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:21:30.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:30.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:21:30.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:30.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:21:30.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:30.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:21:30.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:21:30.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:21:30.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:21:30.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:21:30.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:21:30.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:21:30.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:21:30.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:30.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:30.403 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:30.403 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:21:30.403 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:21:30.403 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:21:30.403 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:21:30.403 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:21:30.430 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:30.430 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:30.430 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:30.430 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:21:30.562 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:21:30.563 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:21:30.563 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:30.563 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:30.563 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:30.563 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:21:30.563 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:30.563 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:30.563 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:30.563 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:21:30.564 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:30.564 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:30.564 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:30.566 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:30.566 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:30.566 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:30.618 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 14 2024-09-11T06:21:30.618 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:30.618 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:30.618 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:30.618 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=59, tag_tid=11, entry_tid=9483], [object_number=58, tag_tid=11, entry_tid=9482], [object_number=57, tag_tid=11, entry_tid=9481], [object_number=56, tag_tid=11, entry_tid=9480]]], state=connected] 2024-09-11T06:21:30.621 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:30.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:30.623 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:21:30.624 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:30.624 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:30.624 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:30.624 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:21:30.624 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:30.624 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:30.624 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:30.624 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:21:30.624 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:30.624 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:30.624 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:30.625 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:30.625 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:30.625 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:30.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 14 2024-09-11T06:21:30.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:30.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:30.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:30.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=59, tag_tid=11, entry_tid=9483], [object_number=58, tag_tid=11, entry_tid=9482], [object_number=57, tag_tid=11, entry_tid=9481], [object_number=56, tag_tid=11, entry_tid=9480]]], state=connected] 2024-09-11T06:21:30.676 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:30.678 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=59, tag_tid=11, entry_tid=9483' 2024-09-11T06:21:30.678 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=59, tag_tid=11, entry_tid=9483' 2024-09-11T06:21:30.678 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=59, tag_tid=11, entry_tid=9483' '!=' 'object_number=56, tag_tid=11, entry_tid=9352' 2024-09-11T06:21:30.678 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:21:30.678 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=59, tag_tid=11, entry_tid=9483' = 'object_number=56, tag_tid=11, entry_tid=9352' 2024-09-11T06:21:30.678 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=59, tag_tid=11, entry_tid=9483' 2024-09-11T06:21:30.679 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:30.679 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:30.679 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:30.680 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=12 2024-09-11T06:21:30.680 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=59, tag_tid=11, entry_tid=9483' 2024-09-11T06:21:30.680 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:30.680 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:30.681 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=11 2024-09-11T06:21:30.682 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:30.682 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:30.682 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:30.683 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:21:30.684 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=59, tag_tid=11, entry_tid=9483' 2024-09-11T06:21:30.684 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:30.684 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:30.685 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=9483 2024-09-11T06:21:30.685 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 12 = 11 -a 1 -le 9483 2024-09-11T06:21:30.685 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:21:30.685 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:21:30.685 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:21:30.887 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:21:30.887 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:30.887 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:21:30.887 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:21:30.887 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:21:30.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:21:30.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:21:30.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:21:30.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:21:30.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:30.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:30.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:21:30.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:21:30.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:21:30.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:21:30.889 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:21:30.889 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:21:30.891 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:21:30.891 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:21:30.891 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:21:30.891 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:30.891 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:21:30.891 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:21:30.891 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:21:30.891 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:21:30.891 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:21:30.892 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:21:30.892 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:21:30.892 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:30.892 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:21:30.892 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:30.892 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:21:30.892 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:30.892 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:21:30.892 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:21:30.892 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:21:30.892 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:21:30.892 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:21:30.893 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:21:30.893 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:21:30.893 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:21:30.893 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:30.893 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:30.893 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:30.893 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:21:30.893 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:21:30.893 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:21:30.893 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:21:30.894 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:21:30.917 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:30.918 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:30.918 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:30.918 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:21:31.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:21:31.050 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:21:31.050 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:31.051 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:31.051 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:31.051 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:21:31.051 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:31.051 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:31.051 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:31.051 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:21:31.051 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:31.052 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:31.052 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:31.054 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:31.054 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:31.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:31.100 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 14 2024-09-11T06:21:31.100 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:31.100 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:31.100 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:31.100 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=57, tag_tid=11, entry_tid=9577], [object_number=56, tag_tid=11, entry_tid=9576], [object_number=59, tag_tid=11, entry_tid=9575], [object_number=58, tag_tid=11, entry_tid=9574]]], state=connected] 2024-09-11T06:21:31.103 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:31.104 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:31.106 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:21:31.106 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:31.106 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:31.106 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:31.106 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:21:31.106 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:31.106 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:31.106 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:31.106 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:21:31.107 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:31.107 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:31.107 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:31.107 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:31.108 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:31.108 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:31.153 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 14 2024-09-11T06:21:31.153 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:31.153 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:31.153 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:31.153 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=57, tag_tid=11, entry_tid=9577], [object_number=56, tag_tid=11, entry_tid=9576], [object_number=59, tag_tid=11, entry_tid=9575], [object_number=58, tag_tid=11, entry_tid=9574]]], state=connected] 2024-09-11T06:21:31.156 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:31.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=57, tag_tid=11, entry_tid=9577' 2024-09-11T06:21:31.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=57, tag_tid=11, entry_tid=9577' 2024-09-11T06:21:31.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=57, tag_tid=11, entry_tid=9577' '!=' 'object_number=59, tag_tid=11, entry_tid=9483' 2024-09-11T06:21:31.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:21:31.159 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=57, tag_tid=11, entry_tid=9577' = 'object_number=59, tag_tid=11, entry_tid=9483' 2024-09-11T06:21:31.159 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=57, tag_tid=11, entry_tid=9577' 2024-09-11T06:21:31.159 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:31.159 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:31.159 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:31.159 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=12 2024-09-11T06:21:31.160 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=57, tag_tid=11, entry_tid=9577' 2024-09-11T06:21:31.160 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:31.160 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:31.161 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=11 2024-09-11T06:21:31.162 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:31.162 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:31.162 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:31.163 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:21:31.163 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=57, tag_tid=11, entry_tid=9577' 2024-09-11T06:21:31.163 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:31.164 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:31.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=9577 2024-09-11T06:21:31.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 12 = 11 -a 1 -le 9577 2024-09-11T06:21:31.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:21:31.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:21:31.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:21:31.366 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:21:31.366 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:31.367 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:21:31.367 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:21:31.367 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:21:31.367 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:21:31.367 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:21:31.367 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:21:31.367 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:21:31.367 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:31.367 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:31.367 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:21:31.367 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:21:31.367 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:21:31.368 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:21:31.368 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:21:31.368 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:21:31.372 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:21:31.372 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:21:31.372 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:21:31.372 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:31.372 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:21:31.372 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:21:31.372 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:21:31.372 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:21:31.372 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:21:31.372 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:21:31.372 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:21:31.373 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:31.373 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:21:31.373 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:31.373 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:21:31.373 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:31.373 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:21:31.373 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:21:31.373 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:21:31.373 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:21:31.373 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:21:31.373 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:21:31.373 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:21:31.373 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:21:31.373 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:31.374 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:31.374 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:31.374 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:21:31.374 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:21:31.374 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:21:31.374 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:21:31.374 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:21:31.399 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:31.399 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:31.399 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:31.399 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:21:31.532 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:21:31.533 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:21:31.533 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:31.534 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:31.534 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:31.534 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:21:31.534 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:31.534 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:31.534 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:31.534 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:21:31.535 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:31.535 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:31.535 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:31.537 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:31.538 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:31.538 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:31.589 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 14 2024-09-11T06:21:31.589 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:31.589 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:31.589 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:31.589 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=59, tag_tid=11, entry_tid=9675], [object_number=58, tag_tid=11, entry_tid=9674], [object_number=57, tag_tid=11, entry_tid=9673], [object_number=56, tag_tid=11, entry_tid=9672]]], state=connected] 2024-09-11T06:21:31.592 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:31.594 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:31.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:21:31.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:31.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:31.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:31.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:21:31.596 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:31.596 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:31.596 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:31.596 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:21:31.596 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:31.596 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:31.596 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:31.597 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:31.597 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:31.597 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:31.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 14 2024-09-11T06:21:31.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:31.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:31.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:31.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=59, tag_tid=11, entry_tid=9675], [object_number=58, tag_tid=11, entry_tid=9674], [object_number=57, tag_tid=11, entry_tid=9673], [object_number=56, tag_tid=11, entry_tid=9672]]], state=connected] 2024-09-11T06:21:31.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:31.649 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=59, tag_tid=11, entry_tid=9675' 2024-09-11T06:21:31.649 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=59, tag_tid=11, entry_tid=9675' 2024-09-11T06:21:31.649 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=59, tag_tid=11, entry_tid=9675' '!=' 'object_number=57, tag_tid=11, entry_tid=9577' 2024-09-11T06:21:31.649 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:21:31.649 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=59, tag_tid=11, entry_tid=9675' = 'object_number=57, tag_tid=11, entry_tid=9577' 2024-09-11T06:21:31.649 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=59, tag_tid=11, entry_tid=9675' 2024-09-11T06:21:31.649 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:31.649 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:31.650 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:31.650 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=12 2024-09-11T06:21:31.651 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=59, tag_tid=11, entry_tid=9675' 2024-09-11T06:21:31.651 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:31.651 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:31.652 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=11 2024-09-11T06:21:31.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:31.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:31.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:31.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:21:31.654 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=59, tag_tid=11, entry_tid=9675' 2024-09-11T06:21:31.654 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:31.654 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:31.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=9675 2024-09-11T06:21:31.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 12 = 11 -a 1 -le 9675 2024-09-11T06:21:31.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:21:31.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:21:31.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:21:31.857 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:21:31.857 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:31.858 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:21:31.858 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:21:31.858 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:21:31.858 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:21:31.858 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:21:31.858 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:21:31.858 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:21:31.858 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:31.858 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:31.858 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:21:31.858 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:21:31.858 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:21:31.858 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:21:31.859 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:21:31.859 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:21:31.862 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:21:31.862 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:21:31.862 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:21:31.862 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:31.862 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:21:31.862 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:21:31.862 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:21:31.862 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:21:31.862 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:21:31.862 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:21:31.863 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:21:31.863 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:31.863 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:21:31.863 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:31.863 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:21:31.863 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:31.863 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:21:31.863 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:21:31.863 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:21:31.863 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:21:31.863 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:21:31.863 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:21:31.863 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:21:31.863 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:21:31.864 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:31.864 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:31.864 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:31.864 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:21:31.864 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:21:31.864 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:21:31.864 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:21:31.864 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:21:31.887 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:31.887 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:31.887 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:31.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:21:32.017 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:21:32.018 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:21:32.018 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:32.018 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:32.018 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:32.018 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:21:32.018 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:32.019 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:32.019 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:32.019 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:21:32.019 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:32.019 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:32.020 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:32.021 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:32.021 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:32.021 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:32.073 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 14 2024-09-11T06:21:32.074 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:32.074 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:32.074 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:32.074 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=57, tag_tid=11, entry_tid=9765], [object_number=56, tag_tid=11, entry_tid=9764], [object_number=59, tag_tid=11, entry_tid=9763], [object_number=58, tag_tid=11, entry_tid=9762]]], state=connected] 2024-09-11T06:21:32.077 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:32.078 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:32.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:21:32.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:32.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:32.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:32.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:21:32.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:32.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:32.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:32.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:21:32.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:32.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:32.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:32.081 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:32.081 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:32.081 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:32.128 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 14 2024-09-11T06:21:32.128 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:32.128 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:32.128 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:32.128 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=57, tag_tid=11, entry_tid=9765], [object_number=56, tag_tid=11, entry_tid=9764], [object_number=59, tag_tid=11, entry_tid=9763], [object_number=58, tag_tid=11, entry_tid=9762]]], state=connected] 2024-09-11T06:21:32.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:32.133 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=57, tag_tid=11, entry_tid=9765' 2024-09-11T06:21:32.133 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=57, tag_tid=11, entry_tid=9765' 2024-09-11T06:21:32.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=57, tag_tid=11, entry_tid=9765' '!=' 'object_number=59, tag_tid=11, entry_tid=9675' 2024-09-11T06:21:32.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:21:32.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=57, tag_tid=11, entry_tid=9765' = 'object_number=59, tag_tid=11, entry_tid=9675' 2024-09-11T06:21:32.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=57, tag_tid=11, entry_tid=9765' 2024-09-11T06:21:32.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:32.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:32.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:32.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=12 2024-09-11T06:21:32.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=57, tag_tid=11, entry_tid=9765' 2024-09-11T06:21:32.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:32.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:32.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=11 2024-09-11T06:21:32.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:32.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:32.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:32.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:21:32.139 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=57, tag_tid=11, entry_tid=9765' 2024-09-11T06:21:32.139 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:32.139 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:32.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=9765 2024-09-11T06:21:32.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 12 = 11 -a 1 -le 9765 2024-09-11T06:21:32.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:21:32.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:21:32.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:21:32.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:21:32.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:32.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:21:32.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:21:32.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:21:32.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:21:32.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:21:32.343 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:21:32.343 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:21:32.343 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:32.343 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:32.343 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:21:32.343 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:21:32.343 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:21:32.343 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:21:32.344 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:21:32.344 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:21:32.346 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:21:32.347 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:21:32.347 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:21:32.347 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:32.347 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:21:32.347 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:21:32.347 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:21:32.347 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:21:32.347 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:21:32.347 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:21:32.347 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:21:32.347 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:32.347 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:21:32.347 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:32.348 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:21:32.348 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:32.348 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:21:32.348 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:21:32.348 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:21:32.348 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:21:32.348 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:21:32.348 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:21:32.348 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:21:32.348 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:21:32.348 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:32.348 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:32.348 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:32.348 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:21:32.349 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:21:32.349 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:21:32.349 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:21:32.349 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:21:32.373 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:32.373 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:32.373 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:32.373 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:21:32.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:21:32.507 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:21:32.507 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:32.507 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:32.507 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:32.507 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:21:32.507 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:32.507 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:32.507 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:32.507 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:21:32.508 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:32.508 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:32.508 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:32.510 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:32.511 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:32.511 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:32.562 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 14 2024-09-11T06:21:32.562 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:32.562 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:32.562 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:32.562 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=58, tag_tid=11, entry_tid=9854], [object_number=57, tag_tid=11, entry_tid=9853], [object_number=56, tag_tid=11, entry_tid=9852], [object_number=59, tag_tid=11, entry_tid=9851]]], state=connected] 2024-09-11T06:21:32.565 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:32.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:32.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:21:32.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:32.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:32.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:32.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:21:32.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:32.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:32.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:32.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:21:32.569 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:32.569 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:32.569 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:32.569 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:32.569 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:32.569 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:32.616 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 14 2024-09-11T06:21:32.616 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:32.616 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:32.616 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:32.616 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=58, tag_tid=11, entry_tid=9854], [object_number=57, tag_tid=11, entry_tid=9853], [object_number=56, tag_tid=11, entry_tid=9852], [object_number=59, tag_tid=11, entry_tid=9851]]], state=connected] 2024-09-11T06:21:32.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:32.621 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=58, tag_tid=11, entry_tid=9854' 2024-09-11T06:21:32.621 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=58, tag_tid=11, entry_tid=9854' 2024-09-11T06:21:32.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=58, tag_tid=11, entry_tid=9854' '!=' 'object_number=57, tag_tid=11, entry_tid=9765' 2024-09-11T06:21:32.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:21:32.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=58, tag_tid=11, entry_tid=9854' = 'object_number=57, tag_tid=11, entry_tid=9765' 2024-09-11T06:21:32.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=58, tag_tid=11, entry_tid=9854' 2024-09-11T06:21:32.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:32.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:32.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:32.624 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=12 2024-09-11T06:21:32.624 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=58, tag_tid=11, entry_tid=9854' 2024-09-11T06:21:32.624 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:32.625 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:32.625 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=11 2024-09-11T06:21:32.625 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:32.625 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:32.625 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:32.626 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:21:32.627 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=58, tag_tid=11, entry_tid=9854' 2024-09-11T06:21:32.627 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:32.627 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:32.628 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=9854 2024-09-11T06:21:32.628 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 12 = 11 -a 1 -le 9854 2024-09-11T06:21:32.628 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:21:32.628 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:21:32.628 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:21:32.830 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:21:32.830 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:32.830 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:21:32.830 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:21:32.830 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:21:32.830 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:21:32.830 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:21:32.830 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:21:32.830 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:21:32.830 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:32.830 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:32.831 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:21:32.831 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:21:32.831 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:21:32.831 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:21:32.831 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:21:32.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:21:32.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:21:32.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:21:32.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:21:32.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:32.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:21:32.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:21:32.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:21:32.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:21:32.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:21:32.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:21:32.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:21:32.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:32.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:21:32.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:32.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:21:32.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:32.836 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:21:32.836 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:21:32.836 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:21:32.836 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:21:32.836 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:21:32.836 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:21:32.836 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:21:32.836 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:21:32.837 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:32.837 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:32.837 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:32.837 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:21:32.837 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:21:32.837 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:21:32.837 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:21:32.837 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:21:32.860 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:32.861 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:32.861 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:32.861 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:21:32.986 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:21:32.988 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:21:32.988 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:32.988 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:32.988 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:32.988 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:21:32.988 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:32.988 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:32.988 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:32.989 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:21:32.989 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:32.989 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:32.989 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:32.991 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:32.991 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:32.992 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:33.037 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 14 2024-09-11T06:21:33.037 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:33.037 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:33.037 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:33.038 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=59, tag_tid=11, entry_tid=9943], [object_number=58, tag_tid=11, entry_tid=9942], [object_number=57, tag_tid=11, entry_tid=9941], [object_number=56, tag_tid=11, entry_tid=9940]]], state=connected] 2024-09-11T06:21:33.041 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:33.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:33.043 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:21:33.043 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:33.043 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:33.043 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:33.044 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:21:33.044 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:33.044 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:33.044 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:33.044 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:21:33.044 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:33.044 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:33.044 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:33.045 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:33.045 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:33.045 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:33.090 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 14 2024-09-11T06:21:33.090 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:33.090 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:33.090 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:33.090 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=59, tag_tid=11, entry_tid=9943], [object_number=58, tag_tid=11, entry_tid=9942], [object_number=57, tag_tid=11, entry_tid=9941], [object_number=56, tag_tid=11, entry_tid=9940]]], state=connected] 2024-09-11T06:21:33.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:33.096 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=59, tag_tid=11, entry_tid=9943' 2024-09-11T06:21:33.096 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=59, tag_tid=11, entry_tid=9943' 2024-09-11T06:21:33.096 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=59, tag_tid=11, entry_tid=9943' '!=' 'object_number=58, tag_tid=11, entry_tid=9854' 2024-09-11T06:21:33.096 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:21:33.096 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=59, tag_tid=11, entry_tid=9943' = 'object_number=58, tag_tid=11, entry_tid=9854' 2024-09-11T06:21:33.096 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=59, tag_tid=11, entry_tid=9943' 2024-09-11T06:21:33.096 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:33.096 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:33.096 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:33.097 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=12 2024-09-11T06:21:33.097 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=59, tag_tid=11, entry_tid=9943' 2024-09-11T06:21:33.098 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:33.098 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:33.099 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=11 2024-09-11T06:21:33.099 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:33.100 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:33.100 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:33.100 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:21:33.101 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=59, tag_tid=11, entry_tid=9943' 2024-09-11T06:21:33.101 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:33.101 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:33.102 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=9943 2024-09-11T06:21:33.102 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 12 = 11 -a 1 -le 9943 2024-09-11T06:21:33.103 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:21:33.103 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:21:33.103 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:21:33.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:21:33.305 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:33.305 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:21:33.305 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:21:33.305 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:21:33.305 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:21:33.305 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:21:33.305 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:21:33.305 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:21:33.305 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:33.305 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:33.305 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:21:33.305 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:21:33.305 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:21:33.305 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:21:33.306 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:21:33.306 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:21:33.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:21:33.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:21:33.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:21:33.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:33.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:21:33.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:21:33.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:21:33.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:21:33.310 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:21:33.310 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:21:33.310 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:21:33.310 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:33.310 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:21:33.310 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:33.310 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:21:33.311 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:33.312 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:21:33.312 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:21:33.312 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:21:33.312 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:21:33.312 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:21:33.313 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:21:33.313 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:21:33.313 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:21:33.313 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:33.313 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:33.313 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:33.313 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:21:33.313 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:21:33.313 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:21:33.313 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:21:33.313 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:21:33.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:33.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:33.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:33.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:21:33.467 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:21:33.468 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:21:33.468 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:33.468 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:33.468 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:33.468 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:21:33.469 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:33.469 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:33.469 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:33.469 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:21:33.470 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:33.470 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:33.470 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:33.471 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:33.471 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:33.471 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:33.516 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 14 2024-09-11T06:21:33.516 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:33.516 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:33.516 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:33.516 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=58, tag_tid=11, entry_tid=10026], [object_number=57, tag_tid=11, entry_tid=10025], [object_number=56, tag_tid=11, entry_tid=10024], [object_number=59, tag_tid=11, entry_tid=10023]]], state=connected] 2024-09-11T06:21:33.519 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:33.522 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:33.522 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:21:33.522 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:33.522 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:33.522 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:33.522 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:21:33.523 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:33.523 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:33.523 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:33.523 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:21:33.523 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:33.523 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:33.523 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:33.524 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:33.524 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:33.524 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:33.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 14 2024-09-11T06:21:33.569 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:33.569 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:33.569 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:33.569 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=58, tag_tid=11, entry_tid=10026], [object_number=57, tag_tid=11, entry_tid=10025], [object_number=56, tag_tid=11, entry_tid=10024], [object_number=59, tag_tid=11, entry_tid=10023]]], state=connected] 2024-09-11T06:21:33.572 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:33.574 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=58, tag_tid=11, entry_tid=10026' 2024-09-11T06:21:33.574 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=58, tag_tid=11, entry_tid=10026' 2024-09-11T06:21:33.574 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=58, tag_tid=11, entry_tid=10026' '!=' 'object_number=59, tag_tid=11, entry_tid=9943' 2024-09-11T06:21:33.574 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:21:33.574 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=58, tag_tid=11, entry_tid=10026' = 'object_number=59, tag_tid=11, entry_tid=9943' 2024-09-11T06:21:33.574 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=58, tag_tid=11, entry_tid=10026' 2024-09-11T06:21:33.575 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:33.575 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:33.575 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:33.575 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=12 2024-09-11T06:21:33.576 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=58, tag_tid=11, entry_tid=10026' 2024-09-11T06:21:33.576 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:33.576 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:33.577 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=11 2024-09-11T06:21:33.578 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:33.578 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:33.578 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:33.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:21:33.580 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=58, tag_tid=11, entry_tid=10026' 2024-09-11T06:21:33.580 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:33.580 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:33.581 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=10026 2024-09-11T06:21:33.581 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 12 = 11 -a 1 -le 10026 2024-09-11T06:21:33.581 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:21:33.581 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:21:33.582 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:21:33.783 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:21:33.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:33.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:21:33.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:21:33.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:21:33.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:21:33.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:21:33.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:21:33.785 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:21:33.785 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:33.785 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:33.785 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:21:33.785 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:21:33.785 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:21:33.785 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:21:33.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:21:33.788 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:21:33.788 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:21:33.788 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:21:33.789 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:21:33.789 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:33.789 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:21:33.789 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:21:33.789 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:21:33.789 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:21:33.789 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:21:33.789 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:21:33.789 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:21:33.789 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:33.789 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:21:33.789 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:33.789 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:21:33.789 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:33.790 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:21:33.790 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:21:33.790 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:21:33.790 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:21:33.790 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:21:33.790 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:21:33.790 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:21:33.790 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:21:33.790 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:33.790 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:33.790 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:33.790 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:21:33.790 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:21:33.790 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:21:33.791 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:21:33.791 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:21:33.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:33.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:33.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:33.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:21:33.946 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:21:33.947 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:21:33.947 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:33.948 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:33.948 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:33.948 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:21:33.948 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:33.948 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:33.948 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:33.948 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:21:33.948 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:33.949 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:33.949 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:33.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:33.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:33.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:34.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 14 2024-09-11T06:21:34.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:34.002 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:34.002 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:34.002 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=59, tag_tid=11, entry_tid=10107], [object_number=58, tag_tid=11, entry_tid=10106], [object_number=57, tag_tid=11, entry_tid=10105], [object_number=56, tag_tid=11, entry_tid=10104]]], state=connected] 2024-09-11T06:21:34.005 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:34.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:34.007 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:21:34.007 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:34.007 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:34.008 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:34.008 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:21:34.008 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:34.008 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:34.008 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:34.008 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:21:34.008 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:34.008 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:34.008 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:34.009 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:34.009 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:34.009 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:34.056 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 14 2024-09-11T06:21:34.056 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:34.056 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:34.056 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:34.057 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=59, tag_tid=11, entry_tid=10107], [object_number=58, tag_tid=11, entry_tid=10106], [object_number=57, tag_tid=11, entry_tid=10105], [object_number=56, tag_tid=11, entry_tid=10104]]], state=connected] 2024-09-11T06:21:34.059 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:34.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=59, tag_tid=11, entry_tid=10107' 2024-09-11T06:21:34.063 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=59, tag_tid=11, entry_tid=10107' 2024-09-11T06:21:34.064 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=59, tag_tid=11, entry_tid=10107' '!=' 'object_number=58, tag_tid=11, entry_tid=10026' 2024-09-11T06:21:34.064 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:21:34.064 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=59, tag_tid=11, entry_tid=10107' = 'object_number=58, tag_tid=11, entry_tid=10026' 2024-09-11T06:21:34.064 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=59, tag_tid=11, entry_tid=10107' 2024-09-11T06:21:34.064 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:34.064 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:34.064 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:34.067 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=12 2024-09-11T06:21:34.067 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=59, tag_tid=11, entry_tid=10107' 2024-09-11T06:21:34.067 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:34.067 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:34.068 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=11 2024-09-11T06:21:34.068 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:34.068 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:34.068 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:34.068 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:21:34.069 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=59, tag_tid=11, entry_tid=10107' 2024-09-11T06:21:34.069 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:34.069 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:34.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=10107 2024-09-11T06:21:34.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 12 = 11 -a 1 -le 10107 2024-09-11T06:21:34.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:21:34.071 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:21:34.071 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:21:34.270 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:21:34.270 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:34.270 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:21:34.270 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:21:34.271 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:21:34.271 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:21:34.271 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:21:34.271 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:21:34.271 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:21:34.271 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:34.271 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:34.271 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:21:34.271 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:21:34.271 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:21:34.271 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:21:34.272 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:21:34.272 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:21:34.275 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:21:34.275 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:21:34.275 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:21:34.275 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:34.276 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:21:34.276 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:21:34.276 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:21:34.276 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:21:34.276 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:21:34.276 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:21:34.276 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:21:34.276 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:34.276 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:21:34.276 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:34.276 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:21:34.276 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:34.276 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:21:34.276 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:21:34.277 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:21:34.277 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:21:34.277 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:21:34.277 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:21:34.277 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:21:34.277 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:21:34.277 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:34.277 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:34.277 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:34.277 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:21:34.277 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:21:34.277 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:21:34.277 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:21:34.277 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:21:34.300 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:34.301 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:34.301 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:34.301 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:21:34.436 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:21:34.437 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:21:34.437 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:34.437 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:34.437 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:34.437 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:21:34.438 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:34.438 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:34.438 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:34.438 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:21:34.438 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:34.438 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:34.438 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:34.439 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:34.440 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:34.440 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:34.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 14 2024-09-11T06:21:34.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:34.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:34.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:34.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=58, tag_tid=11, entry_tid=10202], [object_number=57, tag_tid=11, entry_tid=10201], [object_number=56, tag_tid=11, entry_tid=10200], [object_number=59, tag_tid=11, entry_tid=10199]]], state=connected] 2024-09-11T06:21:34.498 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:34.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:34.501 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:21:34.501 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:34.501 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:34.501 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:34.501 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:21:34.501 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:34.501 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:34.501 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:34.501 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:21:34.501 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:34.502 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:34.502 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:34.502 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:34.502 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:34.502 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:34.549 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 14 2024-09-11T06:21:34.549 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:34.549 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:34.549 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:34.549 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=58, tag_tid=11, entry_tid=10202], [object_number=57, tag_tid=11, entry_tid=10201], [object_number=56, tag_tid=11, entry_tid=10200], [object_number=59, tag_tid=11, entry_tid=10199]]], state=connected] 2024-09-11T06:21:34.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:34.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=58, tag_tid=11, entry_tid=10202' 2024-09-11T06:21:34.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=58, tag_tid=11, entry_tid=10202' 2024-09-11T06:21:34.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=58, tag_tid=11, entry_tid=10202' '!=' 'object_number=59, tag_tid=11, entry_tid=10107' 2024-09-11T06:21:34.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:21:34.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=58, tag_tid=11, entry_tid=10202' = 'object_number=59, tag_tid=11, entry_tid=10107' 2024-09-11T06:21:34.555 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=58, tag_tid=11, entry_tid=10202' 2024-09-11T06:21:34.555 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:34.555 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:34.555 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:34.556 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=12 2024-09-11T06:21:34.556 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=58, tag_tid=11, entry_tid=10202' 2024-09-11T06:21:34.556 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:34.556 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:34.557 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=11 2024-09-11T06:21:34.558 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:34.558 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:34.558 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:34.559 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:21:34.560 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=58, tag_tid=11, entry_tid=10202' 2024-09-11T06:21:34.560 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:34.560 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:34.561 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=10202 2024-09-11T06:21:34.561 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 12 = 11 -a 1 -le 10202 2024-09-11T06:21:34.561 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:21:34.561 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:21:34.561 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:21:34.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:21:34.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:34.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:21:34.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:21:34.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:21:34.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:21:34.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:21:34.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:21:34.764 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:21:34.764 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:34.764 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:34.764 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:21:34.764 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:21:34.764 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:21:34.764 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:21:34.764 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:21:34.765 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:21:34.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:21:34.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:21:34.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:21:34.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:34.768 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:21:34.768 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:21:34.768 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:21:34.768 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:21:34.768 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:21:34.768 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:21:34.768 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:21:34.769 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:34.769 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:21:34.769 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:34.769 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:21:34.769 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:34.769 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:21:34.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:21:34.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:21:34.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:21:34.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:21:34.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:21:34.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:21:34.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:21:34.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:34.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:34.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:34.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:21:34.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:21:34.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:21:34.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:21:34.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:21:34.794 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:34.794 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:34.794 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:34.795 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:21:34.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:21:34.928 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:21:34.929 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:34.929 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:34.929 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:34.929 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:21:34.929 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:34.929 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:34.929 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:34.929 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:21:34.929 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:34.930 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:34.930 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:34.931 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:34.931 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:34.932 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:34.986 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 14 2024-09-11T06:21:34.987 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:34.987 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:34.987 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:34.987 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=56, tag_tid=11, entry_tid=10300], [object_number=59, tag_tid=11, entry_tid=10299], [object_number=58, tag_tid=11, entry_tid=10298], [object_number=57, tag_tid=11, entry_tid=10297]]], state=connected] 2024-09-11T06:21:34.990 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:34.991 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:34.992 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:21:34.992 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:34.992 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:34.992 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:34.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:21:34.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:34.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:34.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:34.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:21:34.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:34.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:34.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:34.994 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:34.994 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:34.994 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:35.041 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 14 2024-09-11T06:21:35.041 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:35.041 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:35.041 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:35.041 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=56, tag_tid=11, entry_tid=10300], [object_number=59, tag_tid=11, entry_tid=10299], [object_number=58, tag_tid=11, entry_tid=10298], [object_number=57, tag_tid=11, entry_tid=10297]]], state=connected] 2024-09-11T06:21:35.044 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:35.046 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=56, tag_tid=11, entry_tid=10300' 2024-09-11T06:21:35.046 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=56, tag_tid=11, entry_tid=10300' 2024-09-11T06:21:35.046 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=56, tag_tid=11, entry_tid=10300' '!=' 'object_number=58, tag_tid=11, entry_tid=10202' 2024-09-11T06:21:35.046 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:21:35.046 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=56, tag_tid=11, entry_tid=10300' = 'object_number=58, tag_tid=11, entry_tid=10202' 2024-09-11T06:21:35.046 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=56, tag_tid=11, entry_tid=10300' 2024-09-11T06:21:35.046 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:35.047 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:35.047 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:35.047 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=12 2024-09-11T06:21:35.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=56, tag_tid=11, entry_tid=10300' 2024-09-11T06:21:35.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:35.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:35.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=11 2024-09-11T06:21:35.050 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:35.050 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:35.050 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:35.051 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:21:35.051 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=56, tag_tid=11, entry_tid=10300' 2024-09-11T06:21:35.051 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:35.052 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:35.053 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=10300 2024-09-11T06:21:35.053 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 12 = 11 -a 1 -le 10300 2024-09-11T06:21:35.053 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:21:35.053 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:21:35.053 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:21:35.255 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:21:35.255 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:35.255 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:21:35.255 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:21:35.255 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:21:35.255 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:21:35.255 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:21:35.255 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:21:35.255 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:21:35.255 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:35.255 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:35.256 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:21:35.256 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:21:35.256 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:21:35.256 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:21:35.256 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:21:35.256 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:21:35.259 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:21:35.259 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:21:35.259 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:21:35.259 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:35.259 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:21:35.259 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:21:35.260 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:21:35.260 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:21:35.260 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:21:35.260 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:21:35.260 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:21:35.260 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:35.260 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:21:35.260 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:35.260 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:21:35.260 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:35.261 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:21:35.261 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:21:35.261 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:21:35.262 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:21:35.262 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:21:35.262 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:21:35.262 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:21:35.262 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:21:35.262 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:35.262 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:35.262 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:35.262 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:21:35.262 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:21:35.262 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:21:35.262 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:21:35.262 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:21:35.285 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:35.286 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:35.286 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:35.286 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:21:35.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:21:35.416 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:21:35.416 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:35.416 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:35.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:35.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:21:35.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:35.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:35.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:35.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:21:35.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:35.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:35.418 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:35.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:35.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:35.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:35.465 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 14 2024-09-11T06:21:35.466 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:35.466 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:35.466 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:35.466 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=57, tag_tid=11, entry_tid=10409], [object_number=56, tag_tid=11, entry_tid=10408], [object_number=59, tag_tid=11, entry_tid=10407], [object_number=58, tag_tid=11, entry_tid=10406]]], state=connected] 2024-09-11T06:21:35.469 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:35.470 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:35.471 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:21:35.471 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:35.472 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:35.472 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:35.472 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:21:35.472 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:35.472 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:35.472 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:35.472 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:21:35.472 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:35.472 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:35.472 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:35.473 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:35.473 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:35.473 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:35.519 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 14 2024-09-11T06:21:35.519 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:35.519 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:35.519 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:35.519 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=57, tag_tid=11, entry_tid=10409], [object_number=56, tag_tid=11, entry_tid=10408], [object_number=59, tag_tid=11, entry_tid=10407], [object_number=58, tag_tid=11, entry_tid=10406]]], state=connected] 2024-09-11T06:21:35.522 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:35.524 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=57, tag_tid=11, entry_tid=10409' 2024-09-11T06:21:35.524 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=57, tag_tid=11, entry_tid=10409' 2024-09-11T06:21:35.524 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=57, tag_tid=11, entry_tid=10409' '!=' 'object_number=56, tag_tid=11, entry_tid=10300' 2024-09-11T06:21:35.525 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:21:35.525 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=57, tag_tid=11, entry_tid=10409' = 'object_number=56, tag_tid=11, entry_tid=10300' 2024-09-11T06:21:35.525 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=57, tag_tid=11, entry_tid=10409' 2024-09-11T06:21:35.525 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:35.525 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:35.525 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:35.525 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=12 2024-09-11T06:21:35.526 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=57, tag_tid=11, entry_tid=10409' 2024-09-11T06:21:35.526 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:35.526 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:35.527 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=11 2024-09-11T06:21:35.528 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:35.528 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:35.528 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:35.529 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:21:35.529 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=57, tag_tid=11, entry_tid=10409' 2024-09-11T06:21:35.529 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:35.529 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:35.530 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=10409 2024-09-11T06:21:35.531 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 12 = 11 -a 1 -le 10409 2024-09-11T06:21:35.531 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:21:35.531 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:21:35.531 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:21:35.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:21:35.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:35.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:21:35.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:21:35.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:21:35.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:21:35.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:21:35.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:21:35.734 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:21:35.734 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:35.734 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:35.734 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:21:35.734 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:21:35.734 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:21:35.734 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:21:35.734 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:21:35.734 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:21:35.737 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:21:35.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:21:35.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:21:35.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:35.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:21:35.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:21:35.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:21:35.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:21:35.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:21:35.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:21:35.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:21:35.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:35.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:21:35.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:35.739 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:21:35.739 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:35.739 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:21:35.739 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:21:35.739 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:21:35.739 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:21:35.739 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:21:35.739 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:21:35.740 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:21:35.740 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:21:35.740 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:35.740 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:35.740 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:35.741 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:21:35.741 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:21:35.741 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:21:35.741 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:21:35.741 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:21:35.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:35.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:35.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:35.764 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:21:35.891 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:21:35.893 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:21:35.893 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:35.893 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:35.893 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:35.893 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:21:35.893 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:35.893 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:35.893 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:35.893 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:21:35.894 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:35.894 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:35.894 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:35.895 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:35.895 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:35.895 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:35.947 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 14 2024-09-11T06:21:35.948 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:35.948 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:35.948 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:35.948 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=57, tag_tid=11, entry_tid=10529], [object_number=56, tag_tid=11, entry_tid=10528], [object_number=59, tag_tid=11, entry_tid=10527], [object_number=58, tag_tid=11, entry_tid=10526]]], state=connected] 2024-09-11T06:21:35.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:35.952 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:35.953 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:21:35.953 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:35.954 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:35.954 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:35.954 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:21:35.954 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:35.954 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:35.954 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:35.954 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:21:35.954 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:35.954 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:35.954 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:35.955 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:35.955 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:35.955 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:36.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 14 2024-09-11T06:21:36.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:36.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:36.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:36.002 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=57, tag_tid=11, entry_tid=10529], [object_number=56, tag_tid=11, entry_tid=10528], [object_number=59, tag_tid=11, entry_tid=10527], [object_number=58, tag_tid=11, entry_tid=10526]]], state=connected] 2024-09-11T06:21:36.004 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:36.007 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=57, tag_tid=11, entry_tid=10529' 2024-09-11T06:21:36.007 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=57, tag_tid=11, entry_tid=10529' 2024-09-11T06:21:36.007 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=57, tag_tid=11, entry_tid=10529' '!=' 'object_number=57, tag_tid=11, entry_tid=10409' 2024-09-11T06:21:36.007 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:21:36.008 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=57, tag_tid=11, entry_tid=10529' = 'object_number=57, tag_tid=11, entry_tid=10409' 2024-09-11T06:21:36.008 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=57, tag_tid=11, entry_tid=10529' 2024-09-11T06:21:36.008 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:36.008 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:36.008 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:36.010 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=12 2024-09-11T06:21:36.010 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=57, tag_tid=11, entry_tid=10529' 2024-09-11T06:21:36.011 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:36.011 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:36.011 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=11 2024-09-11T06:21:36.011 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:36.011 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:36.012 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:36.013 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:21:36.014 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=57, tag_tid=11, entry_tid=10529' 2024-09-11T06:21:36.014 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:36.014 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:36.014 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=10529 2024-09-11T06:21:36.014 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 12 = 11 -a 1 -le 10529 2024-09-11T06:21:36.014 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:21:36.014 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:21:36.014 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:21:36.214 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:21:36.214 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:36.215 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:21:36.215 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:21:36.215 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:21:36.215 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:21:36.215 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:21:36.215 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:21:36.215 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:21:36.215 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:36.215 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:36.215 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:21:36.215 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:21:36.215 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:21:36.216 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:21:36.216 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:21:36.216 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:21:36.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:21:36.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:21:36.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:21:36.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:36.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:21:36.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:21:36.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:21:36.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:21:36.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:21:36.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:21:36.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:21:36.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:36.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:21:36.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:36.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:21:36.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:36.221 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:21:36.221 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:21:36.221 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:21:36.221 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:21:36.221 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:21:36.221 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:21:36.221 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:21:36.221 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:21:36.221 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:36.221 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:36.221 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:36.221 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:21:36.221 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:21:36.221 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:21:36.222 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:21:36.222 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:21:36.245 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:36.246 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:36.246 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:36.246 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:21:36.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:21:36.379 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:21:36.379 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:36.379 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:36.379 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:36.379 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:21:36.379 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:36.379 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:36.379 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:36.379 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:21:36.379 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:36.380 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:36.380 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:36.382 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:36.382 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:36.383 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:36.434 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 14 2024-09-11T06:21:36.434 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:36.434 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:36.434 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:36.434 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=56, tag_tid=11, entry_tid=10636], [object_number=59, tag_tid=11, entry_tid=10635], [object_number=58, tag_tid=11, entry_tid=10634], [object_number=57, tag_tid=11, entry_tid=10633]]], state=connected] 2024-09-11T06:21:36.437 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:36.440 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:36.440 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:21:36.440 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:36.440 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:36.440 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:36.440 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:21:36.440 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:36.440 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:36.441 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:36.441 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:21:36.441 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:36.441 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:36.441 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:36.441 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:36.442 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:36.442 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:36.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 14 2024-09-11T06:21:36.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:36.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:36.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:36.490 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=56, tag_tid=11, entry_tid=10636], [object_number=59, tag_tid=11, entry_tid=10635], [object_number=58, tag_tid=11, entry_tid=10634], [object_number=57, tag_tid=11, entry_tid=10633]]], state=connected] 2024-09-11T06:21:36.492 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:36.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=56, tag_tid=11, entry_tid=10636' 2024-09-11T06:21:36.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=56, tag_tid=11, entry_tid=10636' 2024-09-11T06:21:36.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=56, tag_tid=11, entry_tid=10636' '!=' 'object_number=57, tag_tid=11, entry_tid=10529' 2024-09-11T06:21:36.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:21:36.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=56, tag_tid=11, entry_tid=10636' = 'object_number=57, tag_tid=11, entry_tid=10529' 2024-09-11T06:21:36.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=56, tag_tid=11, entry_tid=10636' 2024-09-11T06:21:36.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:36.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:36.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:36.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=12 2024-09-11T06:21:36.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=56, tag_tid=11, entry_tid=10636' 2024-09-11T06:21:36.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:36.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:36.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=11 2024-09-11T06:21:36.498 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:36.498 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:36.498 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:36.499 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:21:36.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=56, tag_tid=11, entry_tid=10636' 2024-09-11T06:21:36.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:36.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:36.501 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=10636 2024-09-11T06:21:36.501 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 12 = 11 -a 1 -le 10636 2024-09-11T06:21:36.501 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:21:36.501 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:21:36.501 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:21:36.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:21:36.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:36.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:21:36.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:21:36.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:21:36.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:21:36.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:21:36.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:21:36.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:21:36.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:36.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:36.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:21:36.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:21:36.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:21:36.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:21:36.711 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:21:36.711 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:21:36.711 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:21:36.711 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:21:36.712 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:21:36.712 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:36.712 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:21:36.712 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:21:36.712 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:21:36.712 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:21:36.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:21:36.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:21:36.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:21:36.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:36.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:21:36.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:36.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:21:36.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:36.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:21:36.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:21:36.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:21:36.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:21:36.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:21:36.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:21:36.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:21:36.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:21:36.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:36.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:36.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:36.716 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:21:36.716 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:21:36.716 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:21:36.716 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:21:36.716 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:21:36.739 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:36.740 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:36.740 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:36.740 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:21:36.865 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:21:36.866 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:21:36.866 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:36.866 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:36.866 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:36.866 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:21:36.866 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:36.866 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:36.866 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:36.866 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:21:36.867 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:36.867 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:36.867 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:36.868 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:36.868 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:36.868 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:36.921 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 14 2024-09-11T06:21:36.922 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:36.922 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:36.922 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:36.922 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=58, tag_tid=11, entry_tid=10706], [object_number=61, tag_tid=11, entry_tid=10705], [object_number=56, tag_tid=11, entry_tid=10704], [object_number=59, tag_tid=11, entry_tid=10703]]], state=connected] 2024-09-11T06:21:36.926 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:36.928 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:36.929 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:21:36.929 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:36.929 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:36.929 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:36.930 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:21:36.930 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:36.930 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:36.930 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:36.930 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:21:36.930 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:36.930 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:36.930 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:36.931 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:36.932 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:36.932 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:36.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 14 2024-09-11T06:21:36.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:36.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:36.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:36.979 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=58, tag_tid=11, entry_tid=10706], [object_number=61, tag_tid=11, entry_tid=10705], [object_number=56, tag_tid=11, entry_tid=10704], [object_number=59, tag_tid=11, entry_tid=10703]]], state=connected] 2024-09-11T06:21:36.981 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:36.983 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=58, tag_tid=11, entry_tid=10706' 2024-09-11T06:21:36.983 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=58, tag_tid=11, entry_tid=10706' 2024-09-11T06:21:36.983 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=58, tag_tid=11, entry_tid=10706' '!=' 'object_number=56, tag_tid=11, entry_tid=10636' 2024-09-11T06:21:36.983 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:21:36.983 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=58, tag_tid=11, entry_tid=10706' = 'object_number=56, tag_tid=11, entry_tid=10636' 2024-09-11T06:21:36.983 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=58, tag_tid=11, entry_tid=10706' 2024-09-11T06:21:36.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:36.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:36.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:36.985 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=12 2024-09-11T06:21:36.985 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=58, tag_tid=11, entry_tid=10706' 2024-09-11T06:21:36.985 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:36.985 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:36.986 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=11 2024-09-11T06:21:36.987 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:36.987 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:36.987 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:36.988 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:21:36.989 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=58, tag_tid=11, entry_tid=10706' 2024-09-11T06:21:36.989 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:36.989 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:36.990 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=10706 2024-09-11T06:21:36.991 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 12 = 11 -a 1 -le 10706 2024-09-11T06:21:36.991 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:21:36.991 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:21:36.991 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:21:37.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:21:37.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:37.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:21:37.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:21:37.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:21:37.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:21:37.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:21:37.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:21:37.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:21:37.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:37.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:37.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:21:37.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:21:37.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:21:37.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:21:37.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:21:37.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:21:37.196 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:21:37.196 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:21:37.196 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:21:37.196 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:37.196 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:21:37.196 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:21:37.196 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:21:37.196 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:21:37.196 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:21:37.196 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:21:37.196 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:21:37.197 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:37.197 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:21:37.197 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:37.197 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:21:37.197 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:37.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:21:37.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:21:37.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:21:37.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:21:37.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:21:37.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:21:37.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:21:37.199 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:21:37.199 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:37.199 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:37.199 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:37.199 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:21:37.199 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:21:37.199 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:21:37.199 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:21:37.199 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:21:37.225 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:37.225 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:37.225 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:37.225 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:21:37.359 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:21:37.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:21:37.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:37.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:37.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:37.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:21:37.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:37.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:37.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:37.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:21:37.362 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:37.362 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:37.362 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:37.364 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:37.364 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:37.364 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:37.413 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 14 2024-09-11T06:21:37.413 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:37.413 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:37.413 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:37.413 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=60, tag_tid=11, entry_tid=10760], [object_number=59, tag_tid=11, entry_tid=10759], [object_number=62, tag_tid=11, entry_tid=10758], [object_number=61, tag_tid=11, entry_tid=10757]]], state=connected] 2024-09-11T06:21:37.416 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:37.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:37.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:21:37.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:37.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:37.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:37.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:21:37.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:37.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:37.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:37.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:21:37.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:37.420 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:37.420 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:37.420 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:37.420 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:37.420 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:37.468 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 14 2024-09-11T06:21:37.468 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:37.468 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:37.469 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:37.469 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=60, tag_tid=11, entry_tid=10760], [object_number=59, tag_tid=11, entry_tid=10759], [object_number=62, tag_tid=11, entry_tid=10758], [object_number=61, tag_tid=11, entry_tid=10757]]], state=connected] 2024-09-11T06:21:37.471 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:37.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=60, tag_tid=11, entry_tid=10760' 2024-09-11T06:21:37.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=60, tag_tid=11, entry_tid=10760' 2024-09-11T06:21:37.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=60, tag_tid=11, entry_tid=10760' '!=' 'object_number=58, tag_tid=11, entry_tid=10706' 2024-09-11T06:21:37.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:21:37.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=60, tag_tid=11, entry_tid=10760' = 'object_number=58, tag_tid=11, entry_tid=10706' 2024-09-11T06:21:37.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=60, tag_tid=11, entry_tid=10760' 2024-09-11T06:21:37.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:37.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:37.476 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:37.478 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=12 2024-09-11T06:21:37.478 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=60, tag_tid=11, entry_tid=10760' 2024-09-11T06:21:37.478 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:37.478 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:37.478 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=11 2024-09-11T06:21:37.479 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:37.479 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:37.479 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:37.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:21:37.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=60, tag_tid=11, entry_tid=10760' 2024-09-11T06:21:37.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:37.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:37.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=10760 2024-09-11T06:21:37.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 12 = 11 -a 1 -le 10760 2024-09-11T06:21:37.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:21:37.483 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:21:37.483 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:21:37.682 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:21:37.682 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:37.682 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:21:37.682 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:21:37.682 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:21:37.682 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:21:37.682 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:21:37.682 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:21:37.683 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:21:37.683 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:37.683 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:37.683 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:21:37.683 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:21:37.683 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:21:37.683 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:21:37.683 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:21:37.684 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:21:37.686 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:21:37.686 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:21:37.686 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:21:37.686 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:37.686 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:21:37.686 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:21:37.686 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:21:37.686 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:21:37.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:21:37.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:21:37.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:21:37.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:37.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:21:37.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:37.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:21:37.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:37.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:21:37.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:21:37.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:21:37.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:21:37.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:21:37.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:21:37.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:21:37.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:21:37.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:37.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:37.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:37.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:21:37.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:21:37.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:21:37.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:21:37.689 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:21:37.712 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:37.712 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:37.712 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:37.712 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:21:37.841 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:21:37.842 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:21:37.842 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:37.842 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:37.843 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:37.843 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:21:37.843 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:37.843 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:37.843 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:37.843 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:21:37.843 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:37.844 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:37.844 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:37.844 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:37.845 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:37.845 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:37.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 14 2024-09-11T06:21:37.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:37.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:37.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:37.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=62, tag_tid=11, entry_tid=10866], [object_number=61, tag_tid=11, entry_tid=10865], [object_number=60, tag_tid=11, entry_tid=10864], [object_number=59, tag_tid=11, entry_tid=10863]]], state=connected] 2024-09-11T06:21:37.906 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:37.908 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:37.909 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:21:37.909 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:37.909 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:37.909 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:37.909 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:21:37.909 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:37.909 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:37.909 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:37.909 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:21:37.909 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:37.910 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:37.910 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:37.910 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:37.910 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:37.910 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:37.958 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 14 2024-09-11T06:21:37.958 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:37.958 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:37.958 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:37.958 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=62, tag_tid=11, entry_tid=10866], [object_number=61, tag_tid=11, entry_tid=10865], [object_number=60, tag_tid=11, entry_tid=10864], [object_number=59, tag_tid=11, entry_tid=10863]]], state=connected] 2024-09-11T06:21:37.962 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:37.964 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=62, tag_tid=11, entry_tid=10866' 2024-09-11T06:21:37.964 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=62, tag_tid=11, entry_tid=10866' 2024-09-11T06:21:37.964 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=62, tag_tid=11, entry_tid=10866' '!=' 'object_number=60, tag_tid=11, entry_tid=10760' 2024-09-11T06:21:37.964 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:21:37.964 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=62, tag_tid=11, entry_tid=10866' = 'object_number=60, tag_tid=11, entry_tid=10760' 2024-09-11T06:21:37.964 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=62, tag_tid=11, entry_tid=10866' 2024-09-11T06:21:37.964 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:37.964 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:37.964 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:37.965 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=12 2024-09-11T06:21:37.966 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=62, tag_tid=11, entry_tid=10866' 2024-09-11T06:21:37.966 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:37.966 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:37.966 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=11 2024-09-11T06:21:37.967 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:37.967 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:37.967 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:37.968 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:21:37.969 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=62, tag_tid=11, entry_tid=10866' 2024-09-11T06:21:37.969 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:37.969 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:37.970 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=10866 2024-09-11T06:21:37.970 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 12 = 11 -a 1 -le 10866 2024-09-11T06:21:37.970 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:21:37.970 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:21:37.970 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:21:38.172 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:21:38.172 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:38.172 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:21:38.172 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:21:38.172 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:21:38.172 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:21:38.172 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:21:38.172 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:21:38.173 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:21:38.173 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:38.173 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:38.173 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:21:38.173 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:21:38.173 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:21:38.173 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:21:38.173 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:21:38.174 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:21:38.177 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:21:38.177 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:21:38.177 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:21:38.177 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:38.177 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:21:38.177 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:21:38.177 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:21:38.177 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:21:38.178 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:21:38.178 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:21:38.178 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:21:38.178 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:38.178 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:21:38.178 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:38.179 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:21:38.179 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:38.179 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:21:38.179 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:21:38.179 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:21:38.179 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:21:38.179 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:21:38.179 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:21:38.179 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:21:38.179 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:21:38.179 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:38.180 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:38.180 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:38.180 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:21:38.180 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:21:38.180 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:21:38.180 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:21:38.180 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:21:38.203 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:38.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:38.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:38.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:21:38.349 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:21:38.349 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:21:38.349 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:38.349 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:38.349 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:38.350 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:21:38.350 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:38.350 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:38.350 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:38.350 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:21:38.350 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:38.350 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:38.351 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:38.352 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:38.352 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:38.352 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:38.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 15 2024-09-11T06:21:38.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:38.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:38.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:38.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=61, tag_tid=11, entry_tid=10953], [object_number=60, tag_tid=11, entry_tid=10952], [object_number=63, tag_tid=11, entry_tid=10951], [object_number=62, tag_tid=11, entry_tid=10950]]], state=connected] 2024-09-11T06:21:38.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:38.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:38.416 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:21:38.416 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:38.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:38.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:38.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:21:38.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:38.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:38.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:38.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:21:38.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:38.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:38.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:38.418 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:38.418 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:38.418 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:38.465 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 15 2024-09-11T06:21:38.465 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:38.465 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:38.465 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:38.465 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=61, tag_tid=11, entry_tid=10953], [object_number=60, tag_tid=11, entry_tid=10952], [object_number=63, tag_tid=11, entry_tid=10951], [object_number=62, tag_tid=11, entry_tid=10950]]], state=connected] 2024-09-11T06:21:38.468 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:38.470 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=61, tag_tid=11, entry_tid=10953' 2024-09-11T06:21:38.471 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=61, tag_tid=11, entry_tid=10953' 2024-09-11T06:21:38.471 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=61, tag_tid=11, entry_tid=10953' '!=' 'object_number=62, tag_tid=11, entry_tid=10866' 2024-09-11T06:21:38.471 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:21:38.471 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=61, tag_tid=11, entry_tid=10953' = 'object_number=62, tag_tid=11, entry_tid=10866' 2024-09-11T06:21:38.471 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=61, tag_tid=11, entry_tid=10953' 2024-09-11T06:21:38.471 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:38.471 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:38.471 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:38.472 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=12 2024-09-11T06:21:38.472 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=61, tag_tid=11, entry_tid=10953' 2024-09-11T06:21:38.472 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:38.472 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:38.473 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=11 2024-09-11T06:21:38.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:38.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:38.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:38.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:21:38.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=61, tag_tid=11, entry_tid=10953' 2024-09-11T06:21:38.476 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:38.476 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:38.477 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=10953 2024-09-11T06:21:38.477 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 12 = 11 -a 1 -le 10953 2024-09-11T06:21:38.477 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:21:38.477 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:21:38.477 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:21:38.679 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:21:38.679 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:38.679 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:21:38.680 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:21:38.680 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:21:38.680 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:21:38.680 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:21:38.680 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:21:38.680 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:21:38.680 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:38.680 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:38.680 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:21:38.680 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:21:38.680 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:21:38.680 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:21:38.680 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:21:38.681 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:21:38.684 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:21:38.684 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:21:38.684 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:21:38.684 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:38.685 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:21:38.685 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:21:38.685 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:21:38.685 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:21:38.685 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:21:38.685 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:21:38.685 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:21:38.685 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:38.685 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:21:38.685 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:38.685 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:21:38.685 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:38.685 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:21:38.685 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:21:38.686 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:21:38.686 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:21:38.686 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:21:38.686 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:21:38.686 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:21:38.686 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:21:38.686 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:38.686 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:38.686 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:38.686 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:21:38.686 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:21:38.686 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:21:38.686 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:21:38.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:21:38.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:38.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:38.711 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:38.711 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:21:38.840 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:21:38.841 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:21:38.841 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:38.841 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:38.841 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:38.841 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:21:38.841 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:38.842 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:38.842 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:38.842 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:21:38.842 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:38.842 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:38.842 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:38.843 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:38.844 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:38.844 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:38.889 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 15 2024-09-11T06:21:38.890 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:38.890 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:38.890 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:38.890 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=62, tag_tid=11, entry_tid=11066], [object_number=61, tag_tid=11, entry_tid=11065], [object_number=60, tag_tid=11, entry_tid=11064], [object_number=63, tag_tid=11, entry_tid=11063]]], state=connected] 2024-09-11T06:21:38.893 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:38.894 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:38.895 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:21:38.895 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:38.895 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:38.895 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:38.896 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:21:38.896 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:38.896 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:38.896 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:38.896 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:21:38.896 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:38.896 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:38.896 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:38.897 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:38.897 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:38.897 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:38.942 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 15 2024-09-11T06:21:38.942 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:38.942 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:38.942 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:38.942 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=62, tag_tid=11, entry_tid=11066], [object_number=61, tag_tid=11, entry_tid=11065], [object_number=60, tag_tid=11, entry_tid=11064], [object_number=63, tag_tid=11, entry_tid=11063]]], state=connected] 2024-09-11T06:21:38.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:38.947 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=62, tag_tid=11, entry_tid=11066' 2024-09-11T06:21:38.947 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=62, tag_tid=11, entry_tid=11066' 2024-09-11T06:21:38.947 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=62, tag_tid=11, entry_tid=11066' '!=' 'object_number=61, tag_tid=11, entry_tid=10953' 2024-09-11T06:21:38.947 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:21:38.947 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=62, tag_tid=11, entry_tid=11066' = 'object_number=61, tag_tid=11, entry_tid=10953' 2024-09-11T06:21:38.948 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=62, tag_tid=11, entry_tid=11066' 2024-09-11T06:21:38.948 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:38.948 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:38.948 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:38.949 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=12 2024-09-11T06:21:38.949 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=62, tag_tid=11, entry_tid=11066' 2024-09-11T06:21:38.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:38.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:38.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=11 2024-09-11T06:21:38.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:38.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:38.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:38.952 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:21:38.953 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=62, tag_tid=11, entry_tid=11066' 2024-09-11T06:21:38.953 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:38.953 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:38.954 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=11066 2024-09-11T06:21:38.955 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 12 = 11 -a 1 -le 11066 2024-09-11T06:21:38.955 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:21:38.955 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:21:38.955 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:21:39.156 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:21:39.156 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:39.156 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:21:39.156 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:21:39.156 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:21:39.156 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:21:39.157 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:21:39.157 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:21:39.157 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:21:39.157 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:39.157 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:39.157 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:21:39.157 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:21:39.157 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:21:39.157 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:21:39.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:21:39.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:21:39.160 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:21:39.160 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:21:39.160 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:21:39.160 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:39.160 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:21:39.160 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:21:39.161 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:21:39.161 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:21:39.161 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:21:39.161 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:21:39.161 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:21:39.161 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:39.161 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:21:39.161 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:39.161 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:21:39.161 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:39.162 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:21:39.162 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:21:39.162 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:21:39.162 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:21:39.162 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:21:39.163 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:21:39.163 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:21:39.163 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:21:39.163 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:39.163 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:39.163 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:39.163 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:21:39.163 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:21:39.163 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:21:39.163 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:21:39.163 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:21:39.186 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:39.187 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:39.187 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:39.187 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:21:39.319 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:21:39.320 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:21:39.320 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:39.320 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:39.321 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:39.321 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:21:39.321 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:39.321 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:39.321 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:39.321 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:21:39.321 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:39.322 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:39.322 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:39.323 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:39.323 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:39.323 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:39.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 15 2024-09-11T06:21:39.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:39.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:39.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:39.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=61, tag_tid=11, entry_tid=11177], [object_number=60, tag_tid=11, entry_tid=11176], [object_number=63, tag_tid=11, entry_tid=11175], [object_number=62, tag_tid=11, entry_tid=11174]]], state=connected] 2024-09-11T06:21:39.374 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:39.376 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:39.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:21:39.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:39.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:39.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:39.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:21:39.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:39.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:39.378 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:39.378 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:21:39.378 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:39.378 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:39.378 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:39.378 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:39.378 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:39.379 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:39.424 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 15 2024-09-11T06:21:39.424 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:39.424 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:39.425 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:39.425 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=61, tag_tid=11, entry_tid=11177], [object_number=60, tag_tid=11, entry_tid=11176], [object_number=63, tag_tid=11, entry_tid=11175], [object_number=62, tag_tid=11, entry_tid=11174]]], state=connected] 2024-09-11T06:21:39.427 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:39.429 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=61, tag_tid=11, entry_tid=11177' 2024-09-11T06:21:39.429 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=61, tag_tid=11, entry_tid=11177' 2024-09-11T06:21:39.429 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=61, tag_tid=11, entry_tid=11177' '!=' 'object_number=62, tag_tid=11, entry_tid=11066' 2024-09-11T06:21:39.430 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:21:39.430 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=61, tag_tid=11, entry_tid=11177' = 'object_number=62, tag_tid=11, entry_tid=11066' 2024-09-11T06:21:39.430 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=61, tag_tid=11, entry_tid=11177' 2024-09-11T06:21:39.430 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:39.430 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:39.430 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:39.430 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=12 2024-09-11T06:21:39.431 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=61, tag_tid=11, entry_tid=11177' 2024-09-11T06:21:39.431 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:39.431 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:39.432 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=11 2024-09-11T06:21:39.433 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:39.433 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:39.433 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:39.434 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:21:39.435 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=61, tag_tid=11, entry_tid=11177' 2024-09-11T06:21:39.435 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:39.435 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:39.436 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=11177 2024-09-11T06:21:39.436 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 12 = 11 -a 1 -le 11177 2024-09-11T06:21:39.436 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:21:39.436 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:21:39.436 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:21:39.637 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:21:39.638 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:39.638 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:21:39.638 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:21:39.638 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:21:39.638 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:21:39.638 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:21:39.638 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:21:39.638 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:21:39.638 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:39.638 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:39.638 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:21:39.638 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:21:39.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:21:39.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:21:39.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:21:39.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:21:39.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:21:39.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:21:39.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:21:39.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:39.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:21:39.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:21:39.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:21:39.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:21:39.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:21:39.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:21:39.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:21:39.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:39.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:21:39.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:39.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:21:39.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:39.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:21:39.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:21:39.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:21:39.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:21:39.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:21:39.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:21:39.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:21:39.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:21:39.645 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:39.645 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:39.645 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:39.645 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:21:39.645 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:21:39.645 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:21:39.645 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:21:39.645 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:21:39.668 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:39.668 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:39.668 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:39.668 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:21:39.796 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:21:39.797 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:21:39.797 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:39.797 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:39.797 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:39.797 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:21:39.797 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:39.797 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:39.797 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:39.797 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:21:39.798 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:39.798 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:39.798 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:39.800 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:39.800 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:39.800 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:39.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 15 2024-09-11T06:21:39.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:39.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:39.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:39.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=61, tag_tid=11, entry_tid=11289], [object_number=60, tag_tid=11, entry_tid=11288], [object_number=63, tag_tid=11, entry_tid=11287], [object_number=62, tag_tid=11, entry_tid=11286]]], state=connected] 2024-09-11T06:21:39.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:39.858 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:39.859 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:21:39.859 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:39.859 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:39.859 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:39.859 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:21:39.859 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:39.859 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:39.859 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:39.859 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:21:39.860 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:39.860 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:39.860 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:39.860 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:39.860 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:39.860 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:39.908 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 15 2024-09-11T06:21:39.908 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:39.908 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:39.908 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:39.908 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=61, tag_tid=11, entry_tid=11289], [object_number=60, tag_tid=11, entry_tid=11288], [object_number=63, tag_tid=11, entry_tid=11287], [object_number=62, tag_tid=11, entry_tid=11286]]], state=connected] 2024-09-11T06:21:39.912 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:39.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=61, tag_tid=11, entry_tid=11289' 2024-09-11T06:21:39.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=61, tag_tid=11, entry_tid=11289' 2024-09-11T06:21:39.915 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=61, tag_tid=11, entry_tid=11289' '!=' 'object_number=61, tag_tid=11, entry_tid=11177' 2024-09-11T06:21:39.915 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:21:39.915 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=61, tag_tid=11, entry_tid=11289' = 'object_number=61, tag_tid=11, entry_tid=11177' 2024-09-11T06:21:39.915 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=61, tag_tid=11, entry_tid=11289' 2024-09-11T06:21:39.915 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:39.915 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:39.915 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:39.917 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=12 2024-09-11T06:21:39.917 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=61, tag_tid=11, entry_tid=11289' 2024-09-11T06:21:39.917 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:39.917 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:39.918 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=11 2024-09-11T06:21:39.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:39.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:39.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:39.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:21:39.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=61, tag_tid=11, entry_tid=11289' 2024-09-11T06:21:39.921 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:39.921 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:39.922 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=11289 2024-09-11T06:21:39.922 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 12 = 11 -a 1 -le 11289 2024-09-11T06:21:39.922 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:21:39.922 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:21:39.923 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:21:40.122 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:21:40.122 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:40.122 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:21:40.122 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:21:40.123 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:21:40.123 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:21:40.123 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:21:40.123 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:21:40.123 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:21:40.123 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:40.123 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:40.123 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:21:40.123 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:21:40.123 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:21:40.123 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:21:40.124 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:21:40.124 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:21:40.130 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:21:40.130 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:21:40.130 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:21:40.130 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:40.130 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:21:40.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:21:40.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:21:40.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:21:40.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:21:40.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:21:40.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:21:40.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:40.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:21:40.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:40.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:21:40.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:40.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:21:40.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:21:40.133 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:21:40.133 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:21:40.133 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:21:40.133 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:21:40.133 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:21:40.133 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:21:40.133 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:40.133 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:40.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:40.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:21:40.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:21:40.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:21:40.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:21:40.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:21:40.153 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:40.153 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:40.153 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:40.153 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:21:40.279 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:21:40.280 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:21:40.281 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:40.281 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:40.281 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:40.281 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:21:40.281 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:40.281 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:40.281 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:40.281 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:21:40.282 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:40.282 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:40.282 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:40.283 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:40.283 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:40.283 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:40.329 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 15 2024-09-11T06:21:40.329 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:40.329 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:40.329 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:40.329 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=60, tag_tid=11, entry_tid=11392], [object_number=63, tag_tid=11, entry_tid=11391], [object_number=62, tag_tid=11, entry_tid=11390], [object_number=61, tag_tid=11, entry_tid=11389]]], state=connected] 2024-09-11T06:21:40.332 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:40.334 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:40.335 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:21:40.335 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:40.335 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:40.335 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:40.335 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:21:40.335 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:40.335 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:40.335 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:40.335 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:21:40.335 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:40.335 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:40.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:40.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:40.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:40.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:40.382 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 15 2024-09-11T06:21:40.382 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:40.382 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:40.382 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:40.382 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=60, tag_tid=11, entry_tid=11392], [object_number=63, tag_tid=11, entry_tid=11391], [object_number=62, tag_tid=11, entry_tid=11390], [object_number=61, tag_tid=11, entry_tid=11389]]], state=connected] 2024-09-11T06:21:40.385 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:40.387 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=60, tag_tid=11, entry_tid=11392' 2024-09-11T06:21:40.387 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=60, tag_tid=11, entry_tid=11392' 2024-09-11T06:21:40.387 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=60, tag_tid=11, entry_tid=11392' '!=' 'object_number=61, tag_tid=11, entry_tid=11289' 2024-09-11T06:21:40.387 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:21:40.387 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=60, tag_tid=11, entry_tid=11392' = 'object_number=61, tag_tid=11, entry_tid=11289' 2024-09-11T06:21:40.387 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=60, tag_tid=11, entry_tid=11392' 2024-09-11T06:21:40.387 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:40.387 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:40.387 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:40.388 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=12 2024-09-11T06:21:40.389 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=60, tag_tid=11, entry_tid=11392' 2024-09-11T06:21:40.389 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:40.389 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:40.390 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=11 2024-09-11T06:21:40.390 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:40.391 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:40.391 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:40.393 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:21:40.393 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=60, tag_tid=11, entry_tid=11392' 2024-09-11T06:21:40.393 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:40.393 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:40.394 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=11392 2024-09-11T06:21:40.394 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 12 = 11 -a 1 -le 11392 2024-09-11T06:21:40.394 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:21:40.394 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:21:40.394 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:21:40.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:21:40.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:40.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:21:40.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:21:40.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:21:40.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:21:40.596 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:21:40.596 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:21:40.596 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:21:40.596 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:40.596 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:40.596 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:21:40.597 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:21:40.597 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:21:40.597 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:21:40.598 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:21:40.598 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:21:40.600 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:21:40.600 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:21:40.600 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:21:40.601 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:40.601 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:21:40.601 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:21:40.601 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:21:40.601 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:21:40.601 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:21:40.601 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:21:40.601 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:21:40.601 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:40.601 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:21:40.601 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:40.601 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:21:40.602 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:40.602 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:21:40.602 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:21:40.602 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:21:40.602 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:21:40.602 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:21:40.603 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:21:40.603 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:21:40.603 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:21:40.603 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:40.603 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:40.603 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:40.603 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:21:40.604 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:21:40.604 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:21:40.604 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:21:40.605 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:21:40.626 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:40.626 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:40.627 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:40.627 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:21:40.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:21:40.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:21:40.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:40.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:40.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:40.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:21:40.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:40.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:40.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:40.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:21:40.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:40.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:40.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:40.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:40.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:40.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:40.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 15 2024-09-11T06:21:40.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:40.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:40.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:40.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=61, tag_tid=11, entry_tid=11493], [object_number=60, tag_tid=11, entry_tid=11492], [object_number=63, tag_tid=11, entry_tid=11491], [object_number=62, tag_tid=11, entry_tid=11490]]], state=connected] 2024-09-11T06:21:40.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:40.815 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:40.816 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:21:40.816 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:40.817 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:40.817 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:40.817 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:21:40.817 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:40.817 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:40.817 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:40.817 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:21:40.817 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:40.817 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:40.817 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:40.818 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:40.818 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:40.818 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:40.864 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 15 2024-09-11T06:21:40.864 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:40.864 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:40.865 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:40.865 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=61, tag_tid=11, entry_tid=11493], [object_number=60, tag_tid=11, entry_tid=11492], [object_number=63, tag_tid=11, entry_tid=11491], [object_number=62, tag_tid=11, entry_tid=11490]]], state=connected] 2024-09-11T06:21:40.867 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:40.869 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=61, tag_tid=11, entry_tid=11493' 2024-09-11T06:21:40.869 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=61, tag_tid=11, entry_tid=11493' 2024-09-11T06:21:40.869 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=61, tag_tid=11, entry_tid=11493' '!=' 'object_number=60, tag_tid=11, entry_tid=11392' 2024-09-11T06:21:40.869 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:21:40.869 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=61, tag_tid=11, entry_tid=11493' = 'object_number=60, tag_tid=11, entry_tid=11392' 2024-09-11T06:21:40.869 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=61, tag_tid=11, entry_tid=11493' 2024-09-11T06:21:40.870 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:40.870 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:40.870 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:40.871 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=12 2024-09-11T06:21:40.871 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=61, tag_tid=11, entry_tid=11493' 2024-09-11T06:21:40.871 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:40.871 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:40.872 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=11 2024-09-11T06:21:40.873 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:40.873 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:40.873 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:40.874 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:21:40.875 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=61, tag_tid=11, entry_tid=11493' 2024-09-11T06:21:40.875 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:40.875 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:40.876 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=11493 2024-09-11T06:21:40.876 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 12 = 11 -a 1 -le 11493 2024-09-11T06:21:40.876 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:21:40.876 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:21:40.876 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:21:41.078 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:21:41.078 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:41.078 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:21:41.078 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:21:41.078 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:21:41.078 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:21:41.078 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:21:41.078 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:21:41.078 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:21:41.079 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:41.079 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:41.079 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:21:41.079 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:21:41.079 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:21:41.079 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:21:41.079 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:21:41.079 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:21:41.083 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:21:41.083 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:21:41.083 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:21:41.083 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:41.083 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:21:41.083 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:21:41.084 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:21:41.084 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:21:41.084 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:21:41.084 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:21:41.084 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:21:41.084 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:41.084 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:21:41.084 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:41.084 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:21:41.084 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:41.085 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:21:41.085 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:21:41.085 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:21:41.085 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:21:41.085 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:21:41.085 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:21:41.085 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:21:41.085 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:21:41.085 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:41.085 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:41.085 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:41.085 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:21:41.085 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:21:41.086 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:21:41.086 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:21:41.086 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:21:41.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:41.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:41.110 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:41.110 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:21:41.235 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:21:41.237 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:21:41.237 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:41.237 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:41.237 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:41.237 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:21:41.237 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:41.237 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:41.237 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:41.237 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:21:41.238 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:41.238 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:41.238 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:41.239 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:41.240 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:41.240 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:41.286 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 15 2024-09-11T06:21:41.287 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:41.287 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:41.287 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:41.287 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=62, tag_tid=11, entry_tid=11586], [object_number=61, tag_tid=11, entry_tid=11585], [object_number=60, tag_tid=11, entry_tid=11584], [object_number=63, tag_tid=11, entry_tid=11583]]], state=connected] 2024-09-11T06:21:41.290 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:41.291 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:41.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:21:41.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:41.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:41.293 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:41.293 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:21:41.293 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:41.293 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:41.293 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:41.293 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:21:41.293 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:41.293 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:41.293 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:41.294 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:41.294 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:41.294 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:41.551 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 15 2024-09-11T06:21:41.551 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:41.551 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:41.551 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:41.551 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=62, tag_tid=11, entry_tid=11586], [object_number=61, tag_tid=11, entry_tid=11585], [object_number=60, tag_tid=11, entry_tid=11584], [object_number=63, tag_tid=11, entry_tid=11583]]], state=connected] 2024-09-11T06:21:41.555 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:41.557 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=62, tag_tid=11, entry_tid=11586' 2024-09-11T06:21:41.557 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=62, tag_tid=11, entry_tid=11586' 2024-09-11T06:21:41.558 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=62, tag_tid=11, entry_tid=11586' '!=' 'object_number=61, tag_tid=11, entry_tid=11493' 2024-09-11T06:21:41.558 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:21:41.558 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=62, tag_tid=11, entry_tid=11586' = 'object_number=61, tag_tid=11, entry_tid=11493' 2024-09-11T06:21:41.558 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=62, tag_tid=11, entry_tid=11586' 2024-09-11T06:21:41.558 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:41.558 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:41.558 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:41.559 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=12 2024-09-11T06:21:41.560 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=62, tag_tid=11, entry_tid=11586' 2024-09-11T06:21:41.560 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:41.560 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:41.562 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=11 2024-09-11T06:21:41.563 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:41.563 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:41.563 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:41.564 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:21:41.565 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=62, tag_tid=11, entry_tid=11586' 2024-09-11T06:21:41.565 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:41.565 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:41.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=11586 2024-09-11T06:21:41.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 12 = 11 -a 1 -le 11586 2024-09-11T06:21:41.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:21:41.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:21:41.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:21:41.769 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:21:41.769 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:41.769 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:21:41.769 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:21:41.769 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:21:41.769 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:21:41.769 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:21:41.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:21:41.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:21:41.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:41.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:41.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:21:41.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:21:41.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:21:41.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:21:41.774 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:21:41.774 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:21:41.774 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:21:41.774 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:21:41.774 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:21:41.774 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:41.774 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:21:41.774 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:21:41.774 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:21:41.774 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:21:41.775 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:21:41.775 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:21:41.775 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:21:41.775 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:41.775 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:21:41.775 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:41.775 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:21:41.775 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:41.775 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:21:41.775 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:21:41.775 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:21:41.775 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:21:41.776 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:21:41.776 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:21:41.776 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:21:41.776 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:21:41.776 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:41.776 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:41.776 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:41.776 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:21:41.776 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:21:41.776 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:21:41.777 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:21:41.777 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:21:41.799 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:41.799 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:41.799 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:41.799 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:21:41.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:21:41.928 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:21:41.928 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:41.928 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:41.928 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:41.928 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:21:41.928 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:41.928 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:41.928 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:41.928 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:21:41.929 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:41.929 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:41.929 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:41.931 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:41.931 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:41.931 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:41.979 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 15 2024-09-11T06:21:41.979 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:41.979 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:41.979 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:41.979 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=62, tag_tid=11, entry_tid=11734], [object_number=61, tag_tid=11, entry_tid=11733], [object_number=60, tag_tid=11, entry_tid=11732], [object_number=63, tag_tid=11, entry_tid=11731]]], state=connected] 2024-09-11T06:21:41.982 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:41.983 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:41.985 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:21:41.985 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:41.985 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:41.985 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:41.985 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:21:41.985 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:41.985 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:41.985 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:41.985 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:21:41.985 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:41.985 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:41.985 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:41.986 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:41.986 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:41.986 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:42.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 15 2024-09-11T06:21:42.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:42.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:42.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:42.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=62, tag_tid=11, entry_tid=11734], [object_number=61, tag_tid=11, entry_tid=11733], [object_number=60, tag_tid=11, entry_tid=11732], [object_number=63, tag_tid=11, entry_tid=11731]]], state=connected] 2024-09-11T06:21:42.037 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:42.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=62, tag_tid=11, entry_tid=11734' 2024-09-11T06:21:42.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=62, tag_tid=11, entry_tid=11734' 2024-09-11T06:21:42.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=62, tag_tid=11, entry_tid=11734' '!=' 'object_number=62, tag_tid=11, entry_tid=11586' 2024-09-11T06:21:42.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:21:42.040 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=62, tag_tid=11, entry_tid=11734' = 'object_number=62, tag_tid=11, entry_tid=11586' 2024-09-11T06:21:42.040 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=62, tag_tid=11, entry_tid=11734' 2024-09-11T06:21:42.040 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:42.040 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:42.040 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:42.040 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=12 2024-09-11T06:21:42.041 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=62, tag_tid=11, entry_tid=11734' 2024-09-11T06:21:42.041 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:42.041 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:42.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=11 2024-09-11T06:21:42.043 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:42.043 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:42.043 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:42.044 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:21:42.044 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=62, tag_tid=11, entry_tid=11734' 2024-09-11T06:21:42.045 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:42.045 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:42.046 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=11734 2024-09-11T06:21:42.046 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 12 = 11 -a 1 -le 11734 2024-09-11T06:21:42.046 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:21:42.046 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:21:42.046 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:21:42.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:21:42.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:42.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:21:42.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:21:42.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:21:42.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:21:42.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:21:42.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:21:42.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:21:42.250 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:42.250 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:42.250 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:21:42.250 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:21:42.250 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:21:42.250 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:21:42.250 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:21:42.252 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:21:42.252 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:21:42.252 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:21:42.252 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:21:42.252 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:42.252 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:21:42.252 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:21:42.252 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:21:42.252 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:21:42.252 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:21:42.252 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:21:42.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:21:42.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:42.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:21:42.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:42.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:21:42.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:42.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:21:42.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:21:42.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:21:42.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:21:42.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:21:42.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:21:42.254 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:21:42.254 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:21:42.254 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:42.254 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:42.254 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:42.254 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:21:42.254 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:21:42.254 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:21:42.254 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:21:42.254 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:21:42.279 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:42.279 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:42.279 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:42.279 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:21:42.412 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:21:42.413 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:21:42.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:42.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:42.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:42.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:21:42.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:42.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:42.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:42.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:21:42.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:42.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:42.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:42.416 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:42.416 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:42.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:42.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 15 2024-09-11T06:21:42.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:42.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:42.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:42.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=60, tag_tid=11, entry_tid=11840], [object_number=63, tag_tid=11, entry_tid=11839], [object_number=62, tag_tid=11, entry_tid=11838], [object_number=61, tag_tid=11, entry_tid=11837]]], state=connected] 2024-09-11T06:21:42.477 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:42.479 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:42.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:21:42.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:42.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:42.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:42.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:21:42.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:42.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:42.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:42.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:21:42.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:42.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:42.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:42.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:42.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:42.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:42.530 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 15 2024-09-11T06:21:42.530 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:42.530 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:42.531 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:42.531 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=60, tag_tid=11, entry_tid=11868], [object_number=63, tag_tid=11, entry_tid=11867], [object_number=62, tag_tid=11, entry_tid=11866], [object_number=61, tag_tid=11, entry_tid=11865]]], state=connected] 2024-09-11T06:21:42.533 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:42.536 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=60, tag_tid=11, entry_tid=11868' 2024-09-11T06:21:42.536 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=60, tag_tid=11, entry_tid=11868' 2024-09-11T06:21:42.536 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=60, tag_tid=11, entry_tid=11868' '!=' 'object_number=62, tag_tid=11, entry_tid=11734' 2024-09-11T06:21:42.536 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:21:42.536 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=60, tag_tid=11, entry_tid=11868' = 'object_number=62, tag_tid=11, entry_tid=11734' 2024-09-11T06:21:42.536 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=60, tag_tid=11, entry_tid=11868' 2024-09-11T06:21:42.536 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:42.536 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:42.536 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:42.537 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=12 2024-09-11T06:21:42.537 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=60, tag_tid=11, entry_tid=11868' 2024-09-11T06:21:42.538 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:42.538 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:42.538 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=11 2024-09-11T06:21:42.539 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:42.539 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:42.539 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:42.540 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:21:42.541 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=60, tag_tid=11, entry_tid=11868' 2024-09-11T06:21:42.541 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:42.541 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:42.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=11868 2024-09-11T06:21:42.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 12 = 11 -a 1 -le 11868 2024-09-11T06:21:42.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:21:42.543 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:21:42.543 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:21:42.745 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:21:42.745 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:42.745 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:21:42.745 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:21:42.746 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:21:42.746 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:21:42.746 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:21:42.746 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:21:42.746 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:21:42.746 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:42.746 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:42.746 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:21:42.747 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:21:42.747 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:21:42.747 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:21:42.750 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:21:42.750 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:21:42.750 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:21:42.751 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:21:42.751 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:21:42.751 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:42.751 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:21:42.751 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:21:42.751 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:21:42.751 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:21:42.751 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:21:42.751 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:21:42.751 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:21:42.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:42.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:21:42.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:42.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:21:42.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:42.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:21:42.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:21:42.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:21:42.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:21:42.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:21:42.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:21:42.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:21:42.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:21:42.753 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:42.753 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:42.753 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:42.753 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:21:42.753 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:21:42.753 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:21:42.753 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:21:42.753 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:21:42.775 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:42.776 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:42.776 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:42.776 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:21:42.912 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:21:42.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:21:42.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:42.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:42.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:42.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:21:42.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:42.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:42.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:42.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:21:42.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:42.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:42.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:42.916 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:42.916 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:42.917 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:42.968 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 15 2024-09-11T06:21:42.968 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:42.968 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:42.968 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:42.968 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=62, tag_tid=11, entry_tid=11950], [object_number=61, tag_tid=11, entry_tid=11949], [object_number=60, tag_tid=11, entry_tid=11948], [object_number=63, tag_tid=11, entry_tid=11947]]], state=connected] 2024-09-11T06:21:42.971 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:42.972 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:42.974 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:21:42.974 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:42.974 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:42.974 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:42.974 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:21:42.974 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:42.974 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:42.974 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:42.974 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:21:42.974 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:42.974 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:42.974 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:42.975 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:42.975 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:42.975 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:43.021 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 15 2024-09-11T06:21:43.021 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:43.021 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:43.021 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:43.022 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=62, tag_tid=11, entry_tid=11950], [object_number=61, tag_tid=11, entry_tid=11949], [object_number=60, tag_tid=11, entry_tid=11948], [object_number=63, tag_tid=11, entry_tid=11947]]], state=connected] 2024-09-11T06:21:43.024 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:43.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=62, tag_tid=11, entry_tid=11950' 2024-09-11T06:21:43.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=62, tag_tid=11, entry_tid=11950' 2024-09-11T06:21:43.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=62, tag_tid=11, entry_tid=11950' '!=' 'object_number=60, tag_tid=11, entry_tid=11868' 2024-09-11T06:21:43.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:21:43.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=62, tag_tid=11, entry_tid=11950' = 'object_number=60, tag_tid=11, entry_tid=11868' 2024-09-11T06:21:43.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=62, tag_tid=11, entry_tid=11950' 2024-09-11T06:21:43.028 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:43.028 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:43.028 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:43.028 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=12 2024-09-11T06:21:43.028 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=62, tag_tid=11, entry_tid=11950' 2024-09-11T06:21:43.029 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:43.029 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:43.029 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=11 2024-09-11T06:21:43.030 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:43.030 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:43.030 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:43.031 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:21:43.032 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=62, tag_tid=11, entry_tid=11950' 2024-09-11T06:21:43.032 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:43.032 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:43.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=11950 2024-09-11T06:21:43.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 12 = 11 -a 1 -le 11950 2024-09-11T06:21:43.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:21:43.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:21:43.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:21:43.235 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:21:43.235 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:43.235 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:21:43.235 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:21:43.235 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:21:43.235 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:21:43.235 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:21:43.235 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:21:43.235 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:21:43.236 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:43.236 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:43.236 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:21:43.236 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:21:43.236 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:21:43.236 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:21:43.236 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:21:43.237 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:21:43.239 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:21:43.239 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:21:43.239 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:21:43.239 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:43.239 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:21:43.239 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:21:43.239 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:21:43.239 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:21:43.239 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:21:43.239 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:21:43.239 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:21:43.240 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:43.240 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:21:43.240 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:43.240 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:21:43.240 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:43.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:21:43.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:21:43.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:21:43.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:21:43.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:21:43.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:21:43.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:21:43.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:21:43.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:43.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:43.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:43.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:21:43.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:21:43.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:21:43.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:21:43.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:21:43.266 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:43.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:43.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:43.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:21:43.398 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:21:43.399 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:21:43.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:43.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:43.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:43.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:21:43.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:43.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:43.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:43.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:21:43.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:43.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:43.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:43.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:43.403 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:43.403 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:43.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 15 2024-09-11T06:21:43.454 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:43.454 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:43.454 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:43.454 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=63, tag_tid=11, entry_tid=12051], [object_number=62, tag_tid=11, entry_tid=12050], [object_number=61, tag_tid=11, entry_tid=12049], [object_number=60, tag_tid=11, entry_tid=12048]]], state=connected] 2024-09-11T06:21:43.457 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:43.458 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:43.459 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:21:43.460 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:43.460 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:43.460 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:43.460 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:21:43.460 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:43.460 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:43.460 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:43.460 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:21:43.460 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:43.460 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:43.460 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:43.461 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:43.461 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:43.461 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:43.509 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 15 2024-09-11T06:21:43.509 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:43.509 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:43.509 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:43.509 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=63, tag_tid=11, entry_tid=12051], [object_number=62, tag_tid=11, entry_tid=12050], [object_number=61, tag_tid=11, entry_tid=12049], [object_number=60, tag_tid=11, entry_tid=12048]]], state=connected] 2024-09-11T06:21:43.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:43.514 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=63, tag_tid=11, entry_tid=12051' 2024-09-11T06:21:43.514 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=63, tag_tid=11, entry_tid=12051' 2024-09-11T06:21:43.515 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=63, tag_tid=11, entry_tid=12051' '!=' 'object_number=62, tag_tid=11, entry_tid=11950' 2024-09-11T06:21:43.515 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:21:43.515 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=63, tag_tid=11, entry_tid=12051' = 'object_number=62, tag_tid=11, entry_tid=11950' 2024-09-11T06:21:43.515 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=63, tag_tid=11, entry_tid=12051' 2024-09-11T06:21:43.515 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:43.515 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:43.515 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:43.515 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=12 2024-09-11T06:21:43.516 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=63, tag_tid=11, entry_tid=12051' 2024-09-11T06:21:43.516 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:43.516 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:43.517 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=11 2024-09-11T06:21:43.518 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:43.518 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:43.518 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:43.519 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:21:43.520 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=63, tag_tid=11, entry_tid=12051' 2024-09-11T06:21:43.520 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:43.520 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:43.521 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=12051 2024-09-11T06:21:43.521 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 12 = 11 -a 1 -le 12051 2024-09-11T06:21:43.521 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:21:43.521 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:21:43.521 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:21:43.723 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:21:43.723 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:43.723 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:21:43.723 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:21:43.723 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:21:43.723 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:21:43.723 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:21:43.723 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:21:43.723 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:21:43.724 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:43.724 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:43.724 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:21:43.724 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:21:43.724 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:21:43.724 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:21:43.724 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:21:43.725 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:21:43.727 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:21:43.727 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:21:43.727 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:21:43.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:43.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:21:43.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:21:43.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:21:43.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:21:43.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:21:43.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:21:43.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:21:43.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:43.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:21:43.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:43.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:21:43.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:43.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:21:43.729 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:21:43.729 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:21:43.729 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:21:43.729 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:21:43.729 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:21:43.729 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:21:43.729 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:21:43.729 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:43.729 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:43.730 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:43.730 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:21:43.730 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:21:43.730 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:21:43.730 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:21:43.730 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:21:43.753 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:43.753 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:43.753 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:43.754 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:21:43.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:21:43.889 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:21:43.889 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:43.889 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:43.889 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:43.889 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:21:43.890 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:43.890 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:43.890 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:43.890 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:21:43.890 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:43.890 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:43.890 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:43.892 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:43.892 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:43.892 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:43.943 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 15 2024-09-11T06:21:43.943 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:43.943 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:43.943 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:43.943 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=62, tag_tid=11, entry_tid=12158], [object_number=61, tag_tid=11, entry_tid=12157], [object_number=60, tag_tid=11, entry_tid=12156], [object_number=63, tag_tid=11, entry_tid=12155]]], state=connected] 2024-09-11T06:21:43.946 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:43.949 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:43.949 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:21:43.949 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:43.949 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:43.949 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:43.949 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:21:43.949 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:43.949 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:43.949 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:43.949 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:21:43.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:43.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:43.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:43.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:43.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:43.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:43.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 15 2024-09-11T06:21:43.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:43.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:43.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:43.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=62, tag_tid=11, entry_tid=12158], [object_number=61, tag_tid=11, entry_tid=12157], [object_number=60, tag_tid=11, entry_tid=12156], [object_number=63, tag_tid=11, entry_tid=12155]]], state=connected] 2024-09-11T06:21:44.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:44.003 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=62, tag_tid=11, entry_tid=12158' 2024-09-11T06:21:44.004 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=62, tag_tid=11, entry_tid=12158' 2024-09-11T06:21:44.004 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=62, tag_tid=11, entry_tid=12158' '!=' 'object_number=63, tag_tid=11, entry_tid=12051' 2024-09-11T06:21:44.004 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:21:44.004 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=62, tag_tid=11, entry_tid=12158' = 'object_number=63, tag_tid=11, entry_tid=12051' 2024-09-11T06:21:44.004 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=62, tag_tid=11, entry_tid=12158' 2024-09-11T06:21:44.004 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:44.004 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:44.004 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:44.005 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=12 2024-09-11T06:21:44.005 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=62, tag_tid=11, entry_tid=12158' 2024-09-11T06:21:44.005 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:44.005 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:44.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=11 2024-09-11T06:21:44.007 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:44.007 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:44.007 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:44.008 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:21:44.009 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=62, tag_tid=11, entry_tid=12158' 2024-09-11T06:21:44.009 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:44.009 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:44.010 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=12158 2024-09-11T06:21:44.010 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 12 = 11 -a 1 -le 12158 2024-09-11T06:21:44.010 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:21:44.011 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:21:44.011 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:21:44.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:21:44.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:44.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:21:44.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:21:44.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:21:44.213 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:21:44.213 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:21:44.213 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:21:44.213 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:21:44.213 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:44.213 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:44.213 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:21:44.213 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:21:44.213 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:21:44.213 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:21:44.214 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:21:44.214 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:21:44.216 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:21:44.216 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:21:44.216 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:21:44.216 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:44.216 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:21:44.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:21:44.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:21:44.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:21:44.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:21:44.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:21:44.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:21:44.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:44.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:21:44.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:44.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:21:44.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:44.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:21:44.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:21:44.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:21:44.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:21:44.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:21:44.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:21:44.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:21:44.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:21:44.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:44.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:44.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:44.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:21:44.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:21:44.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:21:44.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:21:44.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:21:44.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:44.244 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:44.244 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:44.244 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:21:44.376 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:21:44.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:21:44.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:44.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:44.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:44.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:21:44.378 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:44.378 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:44.378 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:44.378 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:21:44.378 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:44.378 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:44.378 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:44.381 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:44.382 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:44.382 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:44.635 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 15 2024-09-11T06:21:44.635 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:44.635 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:44.635 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:44.635 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=61, tag_tid=11, entry_tid=12269], [object_number=60, tag_tid=11, entry_tid=12268], [object_number=63, tag_tid=11, entry_tid=12267], [object_number=62, tag_tid=11, entry_tid=12266]]], state=connected] 2024-09-11T06:21:44.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:44.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:44.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:21:44.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:44.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:44.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:44.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:21:44.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:44.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:44.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:44.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:21:44.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:44.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:44.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:44.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:44.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:44.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:44.697 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 15 2024-09-11T06:21:44.697 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:44.697 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:44.697 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:44.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=61, tag_tid=11, entry_tid=12269], [object_number=60, tag_tid=11, entry_tid=12268], [object_number=63, tag_tid=11, entry_tid=12267], [object_number=62, tag_tid=11, entry_tid=12266]]], state=connected] 2024-09-11T06:21:44.701 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:44.702 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=61, tag_tid=11, entry_tid=12269' 2024-09-11T06:21:44.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=61, tag_tid=11, entry_tid=12269' 2024-09-11T06:21:44.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=61, tag_tid=11, entry_tid=12269' '!=' 'object_number=62, tag_tid=11, entry_tid=12158' 2024-09-11T06:21:44.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:21:44.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=61, tag_tid=11, entry_tid=12269' = 'object_number=62, tag_tid=11, entry_tid=12158' 2024-09-11T06:21:44.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=61, tag_tid=11, entry_tid=12269' 2024-09-11T06:21:44.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:44.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:44.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:44.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=12 2024-09-11T06:21:44.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=61, tag_tid=11, entry_tid=12269' 2024-09-11T06:21:44.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:44.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:44.706 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=11 2024-09-11T06:21:44.707 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:44.707 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:44.707 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:44.708 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:21:44.708 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=61, tag_tid=11, entry_tid=12269' 2024-09-11T06:21:44.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:44.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:44.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=12269 2024-09-11T06:21:44.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 12 = 11 -a 1 -le 12269 2024-09-11T06:21:44.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:21:44.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:21:44.711 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:21:44.912 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:21:44.912 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:44.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:21:44.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:21:44.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:21:44.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:21:44.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:21:44.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:21:44.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:21:44.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:44.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:44.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:21:44.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:21:44.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:21:44.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:21:44.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:21:44.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:21:44.917 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:21:44.917 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:21:44.917 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:21:44.917 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:44.917 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:21:44.917 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:21:44.917 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:21:44.917 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:21:44.918 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:21:44.918 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:21:44.918 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:21:44.918 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:44.918 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:21:44.918 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:44.918 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:21:44.918 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:44.918 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:21:44.918 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:21:44.918 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:21:44.918 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:21:44.918 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:21:44.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:21:44.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:21:44.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:21:44.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:44.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:44.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:44.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:21:44.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:21:44.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:21:44.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:21:44.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:21:44.942 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:44.942 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:44.942 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:44.942 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:21:45.073 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:21:45.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:21:45.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:45.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:45.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:45.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:21:45.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:45.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:45.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:45.076 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:21:45.076 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:45.076 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:45.076 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:45.077 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:45.077 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:45.077 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:45.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 15 2024-09-11T06:21:45.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:45.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:45.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:45.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=62, tag_tid=11, entry_tid=12426], [object_number=61, tag_tid=11, entry_tid=12425], [object_number=60, tag_tid=11, entry_tid=12424], [object_number=63, tag_tid=11, entry_tid=12423]]], state=connected] 2024-09-11T06:21:45.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:45.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:45.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:21:45.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:45.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:45.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:45.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:21:45.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:45.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:45.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:45.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:21:45.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:45.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:45.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:45.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:45.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:45.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:45.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 15 2024-09-11T06:21:45.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:45.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:45.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:45.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=62, tag_tid=11, entry_tid=12426], [object_number=61, tag_tid=11, entry_tid=12425], [object_number=60, tag_tid=11, entry_tid=12424], [object_number=63, tag_tid=11, entry_tid=12423]]], state=connected] 2024-09-11T06:21:45.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:45.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=62, tag_tid=11, entry_tid=12426' 2024-09-11T06:21:45.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=62, tag_tid=11, entry_tid=12426' 2024-09-11T06:21:45.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=62, tag_tid=11, entry_tid=12426' '!=' 'object_number=61, tag_tid=11, entry_tid=12269' 2024-09-11T06:21:45.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:21:45.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=62, tag_tid=11, entry_tid=12426' = 'object_number=61, tag_tid=11, entry_tid=12269' 2024-09-11T06:21:45.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=62, tag_tid=11, entry_tid=12426' 2024-09-11T06:21:45.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:45.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:45.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:45.196 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=12 2024-09-11T06:21:45.196 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=62, tag_tid=11, entry_tid=12426' 2024-09-11T06:21:45.196 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:45.197 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:45.197 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=11 2024-09-11T06:21:45.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:45.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:45.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:45.199 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:21:45.200 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=62, tag_tid=11, entry_tid=12426' 2024-09-11T06:21:45.200 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:45.200 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:45.201 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=12426 2024-09-11T06:21:45.201 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 12 = 11 -a 1 -le 12426 2024-09-11T06:21:45.201 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:21:45.201 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:21:45.202 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:21:45.403 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:21:45.403 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:45.403 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:21:45.403 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:21:45.403 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:21:45.403 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:21:45.403 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:21:45.404 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:21:45.404 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:21:45.404 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:45.404 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:45.404 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:21:45.404 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:21:45.404 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:21:45.404 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:21:45.404 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:21:45.405 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:21:45.408 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:21:45.408 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:21:45.408 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:21:45.408 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:45.408 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:21:45.408 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:21:45.408 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:21:45.408 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:21:45.408 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:21:45.408 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:21:45.409 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:21:45.409 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:45.409 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:21:45.409 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:45.409 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:21:45.409 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:45.409 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:21:45.409 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:21:45.409 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:21:45.409 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:21:45.409 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:21:45.409 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:21:45.409 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:21:45.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:21:45.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:45.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:45.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:45.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:21:45.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:21:45.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:21:45.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:21:45.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:21:45.436 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:45.436 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:45.436 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:45.436 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:21:45.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:21:45.569 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:21:45.569 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:45.569 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:45.569 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:45.569 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:21:45.569 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:45.569 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:45.569 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:45.569 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:21:45.570 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:45.570 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:45.570 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:45.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:45.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:45.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:45.616 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 15 2024-09-11T06:21:45.617 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:45.617 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:45.617 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:45.617 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=61, tag_tid=11, entry_tid=12525], [object_number=60, tag_tid=11, entry_tid=12524], [object_number=63, tag_tid=11, entry_tid=12523], [object_number=62, tag_tid=11, entry_tid=12522]]], state=connected] 2024-09-11T06:21:45.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:45.621 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:45.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:21:45.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:45.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:45.623 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:45.623 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:21:45.623 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:45.623 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:45.623 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:45.623 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:21:45.623 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:45.623 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:45.623 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:45.624 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:45.624 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:45.624 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:45.670 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 15 2024-09-11T06:21:45.670 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:45.671 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:45.671 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:45.671 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=61, tag_tid=11, entry_tid=12525], [object_number=60, tag_tid=11, entry_tid=12524], [object_number=63, tag_tid=11, entry_tid=12523], [object_number=62, tag_tid=11, entry_tid=12522]]], state=connected] 2024-09-11T06:21:45.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:45.675 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=61, tag_tid=11, entry_tid=12525' 2024-09-11T06:21:45.675 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=61, tag_tid=11, entry_tid=12525' 2024-09-11T06:21:45.675 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=61, tag_tid=11, entry_tid=12525' '!=' 'object_number=62, tag_tid=11, entry_tid=12426' 2024-09-11T06:21:45.675 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:21:45.675 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=61, tag_tid=11, entry_tid=12525' = 'object_number=62, tag_tid=11, entry_tid=12426' 2024-09-11T06:21:45.676 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=61, tag_tid=11, entry_tid=12525' 2024-09-11T06:21:45.676 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:45.676 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:45.677 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:45.677 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=12 2024-09-11T06:21:45.677 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=61, tag_tid=11, entry_tid=12525' 2024-09-11T06:21:45.677 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:45.678 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:45.678 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=11 2024-09-11T06:21:45.679 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:45.679 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:45.679 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:45.680 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:21:45.681 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=61, tag_tid=11, entry_tid=12525' 2024-09-11T06:21:45.681 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:45.681 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:45.682 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=12525 2024-09-11T06:21:45.682 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 12 = 11 -a 1 -le 12525 2024-09-11T06:21:45.683 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:21:45.683 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:21:45.683 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:21:45.884 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:21:45.884 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:45.884 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:21:45.884 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:21:45.884 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:21:45.884 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:21:45.884 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:21:45.884 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:21:45.885 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:21:45.885 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:45.885 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:45.885 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:21:45.885 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:21:45.885 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:21:45.885 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:21:45.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:21:45.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:21:45.887 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:21:45.887 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:21:45.887 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:21:45.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:45.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:21:45.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:21:45.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:21:45.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:21:45.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:21:45.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:21:45.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:21:45.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:45.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:21:45.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:45.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:21:45.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:45.890 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:21:45.890 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:21:45.890 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:21:45.890 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:21:45.890 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:21:45.890 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:21:45.890 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:21:45.890 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:21:45.891 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:45.891 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:45.891 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:45.891 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:21:45.891 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:21:45.891 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:21:45.891 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:21:45.891 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:21:45.915 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:45.915 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:45.915 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:45.915 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:21:46.046 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:21:46.047 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:21:46.047 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:46.047 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:46.047 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:46.047 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:21:46.047 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:46.047 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:46.047 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:46.047 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:21:46.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:46.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:46.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:46.051 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:46.051 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:46.051 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:46.102 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 15 2024-09-11T06:21:46.103 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:46.103 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:46.103 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:46.103 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=62, tag_tid=11, entry_tid=12622], [object_number=61, tag_tid=11, entry_tid=12621], [object_number=60, tag_tid=11, entry_tid=12620], [object_number=63, tag_tid=11, entry_tid=12619]]], state=connected] 2024-09-11T06:21:46.106 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:46.107 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:46.108 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:21:46.108 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:46.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:46.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:46.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:21:46.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:46.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:46.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:46.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:21:46.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:46.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:46.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:46.110 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:46.110 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:46.110 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:46.157 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 15 2024-09-11T06:21:46.157 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:46.157 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:46.157 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:46.157 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=62, tag_tid=11, entry_tid=12622], [object_number=61, tag_tid=11, entry_tid=12621], [object_number=60, tag_tid=11, entry_tid=12620], [object_number=63, tag_tid=11, entry_tid=12619]]], state=connected] 2024-09-11T06:21:46.160 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:46.162 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=62, tag_tid=11, entry_tid=12622' 2024-09-11T06:21:46.162 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=62, tag_tid=11, entry_tid=12622' 2024-09-11T06:21:46.162 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=62, tag_tid=11, entry_tid=12622' '!=' 'object_number=61, tag_tid=11, entry_tid=12525' 2024-09-11T06:21:46.162 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:21:46.162 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=62, tag_tid=11, entry_tid=12622' = 'object_number=61, tag_tid=11, entry_tid=12525' 2024-09-11T06:21:46.163 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=62, tag_tid=11, entry_tid=12622' 2024-09-11T06:21:46.163 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:46.163 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:46.163 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:46.164 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=12 2024-09-11T06:21:46.164 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=62, tag_tid=11, entry_tid=12622' 2024-09-11T06:21:46.164 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:46.164 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:46.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=11 2024-09-11T06:21:46.166 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:46.166 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:46.166 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:46.167 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:21:46.168 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=62, tag_tid=11, entry_tid=12622' 2024-09-11T06:21:46.168 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:46.168 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:46.169 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=12622 2024-09-11T06:21:46.169 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 12 = 11 -a 1 -le 12622 2024-09-11T06:21:46.170 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:21:46.170 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:21:46.170 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:21:46.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:21:46.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:46.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:21:46.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:21:46.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:21:46.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:21:46.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:21:46.372 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:21:46.372 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:21:46.372 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:46.372 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:46.372 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:21:46.372 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:21:46.372 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:21:46.372 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:21:46.373 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:21:46.373 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:21:46.376 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:21:46.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:21:46.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:21:46.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:46.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:21:46.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:21:46.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:21:46.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:21:46.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:21:46.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:21:46.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:21:46.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:46.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:21:46.378 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:46.378 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:21:46.378 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:46.378 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:21:46.378 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:21:46.378 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:21:46.378 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:21:46.378 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:21:46.378 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:21:46.378 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:21:46.378 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:21:46.378 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:46.378 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:46.378 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:46.379 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:21:46.379 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:21:46.379 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:21:46.379 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:21:46.379 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:21:46.397 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:46.398 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:46.398 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:46.398 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:21:46.527 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:21:46.528 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:21:46.528 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:46.528 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:46.528 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:46.528 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:21:46.529 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:46.529 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:46.529 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:46.529 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:21:46.529 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:46.529 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:46.530 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:46.531 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:46.532 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:46.532 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:46.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 15 2024-09-11T06:21:46.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:46.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:46.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:46.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=61, tag_tid=11, entry_tid=12721], [object_number=60, tag_tid=11, entry_tid=12720], [object_number=63, tag_tid=11, entry_tid=12719], [object_number=62, tag_tid=11, entry_tid=12718]]], state=connected] 2024-09-11T06:21:46.582 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:46.584 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:46.585 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:21:46.585 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:46.585 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:46.585 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:46.585 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:21:46.585 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:46.585 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:46.585 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:46.586 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:21:46.586 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:46.586 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:46.586 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:46.586 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:46.586 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:46.587 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:46.632 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 15 2024-09-11T06:21:46.632 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:46.632 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:46.632 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:46.633 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=61, tag_tid=11, entry_tid=12721], [object_number=60, tag_tid=11, entry_tid=12720], [object_number=63, tag_tid=11, entry_tid=12719], [object_number=62, tag_tid=11, entry_tid=12718]]], state=connected] 2024-09-11T06:21:46.635 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:46.637 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=61, tag_tid=11, entry_tid=12721' 2024-09-11T06:21:46.637 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=61, tag_tid=11, entry_tid=12721' 2024-09-11T06:21:46.638 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=61, tag_tid=11, entry_tid=12721' '!=' 'object_number=62, tag_tid=11, entry_tid=12622' 2024-09-11T06:21:46.638 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:21:46.638 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=61, tag_tid=11, entry_tid=12721' = 'object_number=62, tag_tid=11, entry_tid=12622' 2024-09-11T06:21:46.638 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=61, tag_tid=11, entry_tid=12721' 2024-09-11T06:21:46.638 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:46.638 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:46.638 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:46.638 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=12 2024-09-11T06:21:46.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=61, tag_tid=11, entry_tid=12721' 2024-09-11T06:21:46.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:46.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:46.640 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=11 2024-09-11T06:21:46.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:46.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:46.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:46.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:21:46.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=61, tag_tid=11, entry_tid=12721' 2024-09-11T06:21:46.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:46.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:46.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=12721 2024-09-11T06:21:46.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 12 = 11 -a 1 -le 12721 2024-09-11T06:21:46.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:21:46.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:21:46.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:21:46.846 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:21:46.846 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:46.846 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:21:46.846 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:21:46.846 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:21:46.846 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:21:46.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:21:46.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:21:46.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:21:46.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:46.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:46.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:21:46.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:21:46.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:21:46.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:21:46.848 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:21:46.848 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:21:46.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:21:46.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:21:46.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:21:46.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:46.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:21:46.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:21:46.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:21:46.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:21:46.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:21:46.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:21:46.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:21:46.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:46.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:21:46.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:46.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:21:46.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:46.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:21:46.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:21:46.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:21:46.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:21:46.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:21:46.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:21:46.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:21:46.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:21:46.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:46.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:46.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:46.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:21:46.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:21:46.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:21:46.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:21:46.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:21:46.876 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:46.877 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:46.877 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:46.877 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:21:47.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:21:47.007 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:21:47.007 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:47.007 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:47.007 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:47.007 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:21:47.007 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:47.007 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:47.007 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:47.008 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:21:47.008 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:47.008 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:47.008 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:47.010 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:47.011 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:47.011 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:47.064 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 15 2024-09-11T06:21:47.064 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:47.064 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:47.064 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:47.064 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=62, tag_tid=11, entry_tid=12822], [object_number=61, tag_tid=11, entry_tid=12821], [object_number=60, tag_tid=11, entry_tid=12820], [object_number=63, tag_tid=11, entry_tid=12819]]], state=connected] 2024-09-11T06:21:47.067 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:47.069 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:47.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:21:47.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:47.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:47.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:47.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:21:47.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:47.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:47.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:47.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:21:47.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:47.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:47.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:47.071 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:47.071 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:47.071 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:47.118 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 15 2024-09-11T06:21:47.118 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:47.118 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:47.118 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:47.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=62, tag_tid=11, entry_tid=12822], [object_number=61, tag_tid=11, entry_tid=12821], [object_number=60, tag_tid=11, entry_tid=12820], [object_number=63, tag_tid=11, entry_tid=12819]]], state=connected] 2024-09-11T06:21:47.122 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:47.124 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=62, tag_tid=11, entry_tid=12822' 2024-09-11T06:21:47.124 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=62, tag_tid=11, entry_tid=12822' 2024-09-11T06:21:47.124 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=62, tag_tid=11, entry_tid=12822' '!=' 'object_number=61, tag_tid=11, entry_tid=12721' 2024-09-11T06:21:47.124 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:21:47.124 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=62, tag_tid=11, entry_tid=12822' = 'object_number=61, tag_tid=11, entry_tid=12721' 2024-09-11T06:21:47.124 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=62, tag_tid=11, entry_tid=12822' 2024-09-11T06:21:47.124 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:47.124 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:47.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:47.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=12 2024-09-11T06:21:47.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=62, tag_tid=11, entry_tid=12822' 2024-09-11T06:21:47.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:47.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:47.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=11 2024-09-11T06:21:47.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:47.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:47.128 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:47.128 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:21:47.129 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=62, tag_tid=11, entry_tid=12822' 2024-09-11T06:21:47.129 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:47.129 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:47.130 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=12822 2024-09-11T06:21:47.130 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 12 = 11 -a 1 -le 12822 2024-09-11T06:21:47.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:21:47.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:21:47.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:21:47.332 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:21:47.333 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:47.333 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:21:47.333 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:21:47.333 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:21:47.333 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:21:47.333 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:21:47.333 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:21:47.333 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:21:47.333 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:47.333 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:47.333 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:21:47.334 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:21:47.334 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:21:47.334 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:21:47.334 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:21:47.334 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:21:47.337 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:21:47.338 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:21:47.338 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:21:47.338 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:47.338 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:21:47.338 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:21:47.338 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:21:47.338 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:21:47.338 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:21:47.338 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:21:47.338 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:21:47.338 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:47.338 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:21:47.339 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:47.339 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:21:47.339 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:47.339 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:21:47.339 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:21:47.339 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:21:47.339 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:21:47.339 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:21:47.339 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:21:47.339 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:21:47.339 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:21:47.339 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:47.340 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:47.340 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:47.340 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:21:47.340 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:21:47.340 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:21:47.340 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:21:47.340 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:21:47.363 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:47.364 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:47.364 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:47.364 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:21:47.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:21:47.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:21:47.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:47.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:47.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:47.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:21:47.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:47.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:47.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:47.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:21:47.498 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:47.498 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:47.498 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:47.499 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:47.499 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:47.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:47.555 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 15 2024-09-11T06:21:47.555 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:47.555 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:47.556 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:47.556 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=62, tag_tid=11, entry_tid=12922], [object_number=61, tag_tid=11, entry_tid=12921], [object_number=60, tag_tid=11, entry_tid=12920], [object_number=63, tag_tid=11, entry_tid=12919]]], state=connected] 2024-09-11T06:21:47.559 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:47.560 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:47.562 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:21:47.562 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:47.562 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:47.562 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:47.562 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:21:47.562 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:47.562 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:47.562 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:47.562 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:21:47.562 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:47.562 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:47.562 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:47.563 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:47.563 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:47.563 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:47.611 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 15 2024-09-11T06:21:47.611 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:47.612 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:47.612 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:47.612 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=62, tag_tid=11, entry_tid=12922], [object_number=61, tag_tid=11, entry_tid=12921], [object_number=60, tag_tid=11, entry_tid=12920], [object_number=63, tag_tid=11, entry_tid=12919]]], state=connected] 2024-09-11T06:21:47.615 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:47.617 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=62, tag_tid=11, entry_tid=12922' 2024-09-11T06:21:47.617 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=62, tag_tid=11, entry_tid=12922' 2024-09-11T06:21:47.617 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=62, tag_tid=11, entry_tid=12922' '!=' 'object_number=62, tag_tid=11, entry_tid=12822' 2024-09-11T06:21:47.617 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:21:47.617 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=62, tag_tid=11, entry_tid=12922' = 'object_number=62, tag_tid=11, entry_tid=12822' 2024-09-11T06:21:47.617 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=62, tag_tid=11, entry_tid=12922' 2024-09-11T06:21:47.617 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:47.617 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:47.617 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:47.618 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=12 2024-09-11T06:21:47.618 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=62, tag_tid=11, entry_tid=12922' 2024-09-11T06:21:47.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:47.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:47.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=11 2024-09-11T06:21:47.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:47.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:47.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:47.621 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:21:47.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=62, tag_tid=11, entry_tid=12922' 2024-09-11T06:21:47.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:47.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:47.623 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=12922 2024-09-11T06:21:47.623 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 12 = 11 -a 1 -le 12922 2024-09-11T06:21:47.623 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:21:47.624 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:21:47.624 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:21:47.825 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:21:47.825 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:47.825 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:21:47.826 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:21:47.826 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:21:47.826 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:21:47.826 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:21:47.826 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:21:47.826 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:21:47.826 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:47.826 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:47.826 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:21:47.826 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:21:47.826 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:21:47.827 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:21:47.827 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:21:47.827 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:21:47.831 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:21:47.831 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:21:47.831 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:21:47.831 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:47.831 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:21:47.831 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:21:47.831 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:21:47.831 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:21:47.831 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:21:47.831 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:21:47.831 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:21:47.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:47.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:21:47.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:47.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:21:47.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:47.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:21:47.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:21:47.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:21:47.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:21:47.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:21:47.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:21:47.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:21:47.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:21:47.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:47.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:47.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:47.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:21:47.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:21:47.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:21:47.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:21:47.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:21:47.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:47.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:47.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:47.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:21:47.985 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:21:47.986 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:21:47.986 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:47.986 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:47.986 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:47.986 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:21:47.986 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:47.986 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:47.986 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:47.986 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:21:47.986 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:47.987 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:47.987 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:47.987 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:47.988 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:47.988 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:48.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 15 2024-09-11T06:21:48.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:48.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:48.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:48.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=61, tag_tid=11, entry_tid=13029], [object_number=60, tag_tid=11, entry_tid=13028], [object_number=63, tag_tid=11, entry_tid=13027], [object_number=62, tag_tid=11, entry_tid=13026]]], state=connected] 2024-09-11T06:21:48.045 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:48.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:48.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:21:48.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:48.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:48.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:48.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:21:48.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:48.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:48.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:48.050 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:21:48.050 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:48.050 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:48.050 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:48.051 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:48.051 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:48.051 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:48.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 15 2024-09-11T06:21:48.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:48.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:48.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:48.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=61, tag_tid=11, entry_tid=13029], [object_number=60, tag_tid=11, entry_tid=13028], [object_number=63, tag_tid=11, entry_tid=13027], [object_number=62, tag_tid=11, entry_tid=13026]]], state=connected] 2024-09-11T06:21:48.097 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:48.099 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=61, tag_tid=11, entry_tid=13029' 2024-09-11T06:21:48.099 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=61, tag_tid=11, entry_tid=13029' 2024-09-11T06:21:48.099 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=61, tag_tid=11, entry_tid=13029' '!=' 'object_number=62, tag_tid=11, entry_tid=12922' 2024-09-11T06:21:48.099 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:21:48.099 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=61, tag_tid=11, entry_tid=13029' = 'object_number=62, tag_tid=11, entry_tid=12922' 2024-09-11T06:21:48.099 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=61, tag_tid=11, entry_tid=13029' 2024-09-11T06:21:48.100 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:48.100 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:48.100 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:48.100 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=12 2024-09-11T06:21:48.101 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=61, tag_tid=11, entry_tid=13029' 2024-09-11T06:21:48.101 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:48.101 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:48.102 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=11 2024-09-11T06:21:48.102 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:48.103 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:48.103 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:48.103 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:21:48.104 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=61, tag_tid=11, entry_tid=13029' 2024-09-11T06:21:48.104 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:48.104 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:48.105 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=13029 2024-09-11T06:21:48.106 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 12 = 11 -a 1 -le 13029 2024-09-11T06:21:48.106 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:21:48.106 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:21:48.106 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:21:48.307 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:21:48.307 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:48.307 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:21:48.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:21:48.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:21:48.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:21:48.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:21:48.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:21:48.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:21:48.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:48.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:48.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:21:48.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:21:48.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:21:48.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:21:48.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:21:48.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:21:48.312 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:21:48.312 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:21:48.312 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:21:48.313 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:48.313 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:21:48.313 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:21:48.313 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:21:48.313 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:21:48.313 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:21:48.313 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:21:48.313 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:21:48.313 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:48.313 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:21:48.313 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:48.313 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:21:48.313 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:48.314 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:21:48.314 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:21:48.314 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:21:48.314 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:21:48.314 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:21:48.314 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:21:48.314 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:21:48.314 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:21:48.314 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:48.314 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:48.315 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:48.315 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:21:48.315 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:21:48.315 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:21:48.315 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:21:48.315 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:21:48.338 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:48.338 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:48.338 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:48.338 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:21:48.469 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:21:48.470 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:21:48.470 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:48.470 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:48.470 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:48.470 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:21:48.471 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:48.471 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:48.471 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:48.471 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:21:48.471 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:48.471 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:48.472 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:48.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:48.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:48.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:48.525 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 15 2024-09-11T06:21:48.525 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:48.525 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:48.526 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:48.526 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=61, tag_tid=11, entry_tid=13125], [object_number=60, tag_tid=11, entry_tid=13124], [object_number=63, tag_tid=11, entry_tid=13123], [object_number=62, tag_tid=11, entry_tid=13122]]], state=connected] 2024-09-11T06:21:48.529 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:48.531 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:48.531 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:21:48.531 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:48.531 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:48.531 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:48.531 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:21:48.531 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:48.532 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:48.532 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:48.532 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:21:48.532 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:48.532 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:48.532 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:48.533 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:48.533 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:48.533 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:48.577 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 15 2024-09-11T06:21:48.577 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:48.577 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:48.577 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:48.577 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=61, tag_tid=11, entry_tid=13125], [object_number=60, tag_tid=11, entry_tid=13124], [object_number=63, tag_tid=11, entry_tid=13123], [object_number=62, tag_tid=11, entry_tid=13122]]], state=connected] 2024-09-11T06:21:48.580 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:48.582 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=61, tag_tid=11, entry_tid=13125' 2024-09-11T06:21:48.582 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=61, tag_tid=11, entry_tid=13125' 2024-09-11T06:21:48.582 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=61, tag_tid=11, entry_tid=13125' '!=' 'object_number=61, tag_tid=11, entry_tid=13029' 2024-09-11T06:21:48.582 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:21:48.582 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=61, tag_tid=11, entry_tid=13125' = 'object_number=61, tag_tid=11, entry_tid=13029' 2024-09-11T06:21:48.582 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=61, tag_tid=11, entry_tid=13125' 2024-09-11T06:21:48.582 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:48.582 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:48.583 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:48.583 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=12 2024-09-11T06:21:48.584 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=61, tag_tid=11, entry_tid=13125' 2024-09-11T06:21:48.584 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:48.584 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:48.585 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=11 2024-09-11T06:21:48.585 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:48.586 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:48.586 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:48.587 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:21:48.587 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=61, tag_tid=11, entry_tid=13125' 2024-09-11T06:21:48.587 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:48.588 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:48.589 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=13125 2024-09-11T06:21:48.589 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 12 = 11 -a 1 -le 13125 2024-09-11T06:21:48.589 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:21:48.589 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:21:48.590 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:21:48.790 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:21:48.790 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:48.790 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:21:48.790 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:21:48.790 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:21:48.790 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:21:48.790 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:21:48.790 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:21:48.790 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:21:48.791 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:48.791 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:48.791 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:21:48.791 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:21:48.791 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:21:48.791 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:21:48.791 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:21:48.792 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:21:48.797 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:21:48.797 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:21:48.797 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:21:48.797 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:48.797 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:21:48.797 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:21:48.798 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:21:48.798 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:21:48.798 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:21:48.798 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:21:48.798 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:21:48.798 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:48.798 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:21:48.798 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:48.799 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:21:48.799 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:48.799 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:21:48.799 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:21:48.799 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:21:48.799 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:21:48.799 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:21:48.799 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:21:48.799 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:21:48.799 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:21:48.799 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:48.799 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:48.799 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:48.799 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:21:48.800 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:21:48.800 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:21:48.800 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:21:48.800 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:21:48.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:48.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:48.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:48.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:21:48.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:21:48.946 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:21:48.946 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:48.946 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:48.946 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:48.947 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:21:48.947 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:48.947 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:48.947 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:48.947 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:21:48.947 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:48.947 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:48.948 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:48.949 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:48.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:48.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:48.995 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 15 2024-09-11T06:21:48.996 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:48.996 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:48.996 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:48.996 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=63, tag_tid=11, entry_tid=13223], [object_number=62, tag_tid=11, entry_tid=13222], [object_number=61, tag_tid=11, entry_tid=13221], [object_number=60, tag_tid=11, entry_tid=13220]]], state=connected] 2024-09-11T06:21:49.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:49.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:49.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:21:49.002 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:49.002 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:49.002 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:49.002 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:21:49.002 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:49.002 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:49.002 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:49.002 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:21:49.002 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:49.002 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:49.002 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:49.003 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:49.003 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:49.003 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:49.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 15 2024-09-11T06:21:49.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:49.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:49.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:49.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=63, tag_tid=11, entry_tid=13223], [object_number=62, tag_tid=11, entry_tid=13222], [object_number=61, tag_tid=11, entry_tid=13221], [object_number=60, tag_tid=11, entry_tid=13220]]], state=connected] 2024-09-11T06:21:49.052 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:49.054 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=63, tag_tid=11, entry_tid=13223' 2024-09-11T06:21:49.054 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=63, tag_tid=11, entry_tid=13223' 2024-09-11T06:21:49.054 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=63, tag_tid=11, entry_tid=13223' '!=' 'object_number=61, tag_tid=11, entry_tid=13125' 2024-09-11T06:21:49.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:21:49.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=63, tag_tid=11, entry_tid=13223' = 'object_number=61, tag_tid=11, entry_tid=13125' 2024-09-11T06:21:49.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=63, tag_tid=11, entry_tid=13223' 2024-09-11T06:21:49.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:49.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:49.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:49.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=12 2024-09-11T06:21:49.056 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=63, tag_tid=11, entry_tid=13223' 2024-09-11T06:21:49.056 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:49.056 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:49.057 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=11 2024-09-11T06:21:49.058 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:49.058 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:49.058 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:49.059 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:21:49.059 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=63, tag_tid=11, entry_tid=13223' 2024-09-11T06:21:49.059 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:49.059 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:49.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=13223 2024-09-11T06:21:49.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 12 = 11 -a 1 -le 13223 2024-09-11T06:21:49.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:21:49.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:21:49.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:21:49.262 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:21:49.262 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:49.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:21:49.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:21:49.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:21:49.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:21:49.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:21:49.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:21:49.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:21:49.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:49.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:49.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:21:49.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:21:49.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:21:49.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:21:49.264 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:21:49.264 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:21:49.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:21:49.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:21:49.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:21:49.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:49.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:21:49.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:21:49.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:21:49.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:21:49.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:21:49.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:21:49.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:21:49.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:49.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:21:49.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:49.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:21:49.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:49.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:21:49.269 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:21:49.269 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:21:49.269 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:21:49.269 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:21:49.269 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:21:49.269 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:21:49.269 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:21:49.269 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:49.269 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:49.269 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:49.269 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:21:49.269 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:21:49.270 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:21:49.270 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:21:49.270 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:21:49.294 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:49.294 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:49.294 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:49.294 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:21:49.423 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:21:49.425 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:21:49.425 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:49.425 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:49.425 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:49.425 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:21:49.425 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:49.425 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:49.425 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:49.425 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:21:49.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:49.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:49.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:49.429 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:49.429 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:49.429 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:49.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 15 2024-09-11T06:21:49.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:49.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:49.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:49.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=61, tag_tid=11, entry_tid=13317], [object_number=60, tag_tid=11, entry_tid=13316], [object_number=63, tag_tid=11, entry_tid=13315], [object_number=62, tag_tid=11, entry_tid=13314]]], state=connected] 2024-09-11T06:21:49.485 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:49.486 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:49.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:21:49.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:49.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:49.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:49.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:21:49.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:49.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:49.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:49.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:21:49.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:49.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:49.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:49.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:49.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:49.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:49.537 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 15 2024-09-11T06:21:49.537 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:49.537 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:49.537 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:49.538 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=61, tag_tid=11, entry_tid=13317], [object_number=60, tag_tid=11, entry_tid=13316], [object_number=63, tag_tid=11, entry_tid=13315], [object_number=62, tag_tid=11, entry_tid=13314]]], state=connected] 2024-09-11T06:21:49.540 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:49.543 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=61, tag_tid=11, entry_tid=13317' 2024-09-11T06:21:49.543 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=61, tag_tid=11, entry_tid=13317' 2024-09-11T06:21:49.543 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=61, tag_tid=11, entry_tid=13317' '!=' 'object_number=63, tag_tid=11, entry_tid=13223' 2024-09-11T06:21:49.543 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:21:49.543 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=61, tag_tid=11, entry_tid=13317' = 'object_number=63, tag_tid=11, entry_tid=13223' 2024-09-11T06:21:49.543 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=61, tag_tid=11, entry_tid=13317' 2024-09-11T06:21:49.543 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:49.543 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:49.543 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:49.544 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=12 2024-09-11T06:21:49.544 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=61, tag_tid=11, entry_tid=13317' 2024-09-11T06:21:49.545 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:49.545 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:49.546 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=11 2024-09-11T06:21:49.546 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:49.546 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:49.546 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:49.547 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:21:49.548 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=61, tag_tid=11, entry_tid=13317' 2024-09-11T06:21:49.548 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:49.548 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:49.549 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=13317 2024-09-11T06:21:49.549 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 12 = 11 -a 1 -le 13317 2024-09-11T06:21:49.549 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:21:49.549 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:21:49.550 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:21:49.751 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:21:49.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:49.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:21:49.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:21:49.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:21:49.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:21:49.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:21:49.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:21:49.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:21:49.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:49.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:49.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:21:49.753 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:21:49.753 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:21:49.753 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:21:49.753 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:21:49.753 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:21:49.756 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:21:49.758 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:21:49.758 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:21:49.758 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:49.758 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:21:49.758 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:21:49.758 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:21:49.758 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:21:49.758 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:21:49.758 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:21:49.758 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:21:49.758 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:49.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:21:49.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:49.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:21:49.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:49.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:21:49.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:21:49.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:21:49.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:21:49.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:21:49.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:21:49.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:21:49.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:21:49.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:49.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:49.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:49.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:21:49.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:21:49.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:21:49.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:21:49.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:21:49.782 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:49.782 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:49.782 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:49.782 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:21:49.911 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:21:49.912 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:21:49.912 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:49.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:49.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:49.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:21:49.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:49.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:49.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:49.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:21:49.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:49.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:49.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:49.916 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:49.917 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:49.917 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:49.962 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 15 2024-09-11T06:21:49.962 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:49.962 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:49.962 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:49.963 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=60, tag_tid=11, entry_tid=13408], [object_number=63, tag_tid=11, entry_tid=13407], [object_number=62, tag_tid=11, entry_tid=13406], [object_number=61, tag_tid=11, entry_tid=13405]]], state=connected] 2024-09-11T06:21:49.966 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:49.967 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:49.968 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:21:49.968 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:49.969 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:49.969 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:49.969 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:21:49.969 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:49.969 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:49.969 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:49.969 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:21:49.969 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:49.969 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:49.969 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:49.970 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:49.970 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:49.970 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:50.017 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 15 2024-09-11T06:21:50.017 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:50.017 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:50.017 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:50.017 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=60, tag_tid=11, entry_tid=13408], [object_number=63, tag_tid=11, entry_tid=13407], [object_number=62, tag_tid=11, entry_tid=13406], [object_number=61, tag_tid=11, entry_tid=13405]]], state=connected] 2024-09-11T06:21:50.021 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:50.023 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=60, tag_tid=11, entry_tid=13408' 2024-09-11T06:21:50.023 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=60, tag_tid=11, entry_tid=13408' 2024-09-11T06:21:50.023 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=60, tag_tid=11, entry_tid=13408' '!=' 'object_number=61, tag_tid=11, entry_tid=13317' 2024-09-11T06:21:50.023 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:21:50.023 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=60, tag_tid=11, entry_tid=13408' = 'object_number=61, tag_tid=11, entry_tid=13317' 2024-09-11T06:21:50.023 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=60, tag_tid=11, entry_tid=13408' 2024-09-11T06:21:50.023 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:50.023 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:50.023 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:50.024 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=12 2024-09-11T06:21:50.025 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=60, tag_tid=11, entry_tid=13408' 2024-09-11T06:21:50.025 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:50.025 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:50.026 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=11 2024-09-11T06:21:50.026 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:50.026 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:50.026 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:50.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:21:50.028 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=60, tag_tid=11, entry_tid=13408' 2024-09-11T06:21:50.028 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:50.028 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:50.029 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=13408 2024-09-11T06:21:50.029 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 12 = 11 -a 1 -le 13408 2024-09-11T06:21:50.029 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:21:50.029 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:21:50.030 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:21:50.231 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:21:50.231 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:50.231 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:21:50.231 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:21:50.231 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:21:50.231 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:21:50.231 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:21:50.231 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:21:50.232 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:21:50.232 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:50.232 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:50.232 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:21:50.232 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:21:50.232 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:21:50.232 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:21:50.233 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:21:50.233 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:21:50.236 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:21:50.236 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:21:50.236 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:21:50.236 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:50.236 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:21:50.236 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:21:50.236 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:21:50.236 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:21:50.237 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:21:50.237 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:21:50.237 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:21:50.237 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:50.237 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:21:50.237 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:50.237 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:21:50.237 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:50.237 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:21:50.237 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:21:50.238 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:21:50.238 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:21:50.238 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:21:50.238 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:21:50.238 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:21:50.238 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:21:50.238 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:50.238 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:50.238 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:50.238 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:21:50.239 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:21:50.239 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:21:50.239 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:21:50.239 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:21:50.261 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:50.261 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:50.262 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:50.262 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:21:50.394 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:21:50.395 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:21:50.395 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:50.395 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:50.395 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:50.395 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:21:50.395 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:50.395 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:50.396 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:50.396 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:21:50.396 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:50.396 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:50.397 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:50.399 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:50.399 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:50.399 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:50.444 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 15 2024-09-11T06:21:50.444 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:50.445 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:50.445 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:50.445 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=60, tag_tid=11, entry_tid=13504], [object_number=63, tag_tid=11, entry_tid=13503], [object_number=62, tag_tid=11, entry_tid=13502], [object_number=61, tag_tid=11, entry_tid=13501]]], state=connected] 2024-09-11T06:21:50.448 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:50.450 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:50.450 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:21:50.451 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:50.451 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:50.451 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:50.451 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:21:50.451 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:50.451 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:50.451 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:50.451 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:21:50.451 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:50.451 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:50.451 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:50.452 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:50.452 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:50.452 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:50.498 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 15 2024-09-11T06:21:50.498 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:50.498 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:50.498 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:50.499 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=60, tag_tid=11, entry_tid=13504], [object_number=63, tag_tid=11, entry_tid=13503], [object_number=62, tag_tid=11, entry_tid=13502], [object_number=61, tag_tid=11, entry_tid=13501]]], state=connected] 2024-09-11T06:21:50.502 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:50.504 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=60, tag_tid=11, entry_tid=13504' 2024-09-11T06:21:50.504 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=60, tag_tid=11, entry_tid=13504' 2024-09-11T06:21:50.504 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=60, tag_tid=11, entry_tid=13504' '!=' 'object_number=60, tag_tid=11, entry_tid=13408' 2024-09-11T06:21:50.504 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:21:50.504 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=60, tag_tid=11, entry_tid=13504' = 'object_number=60, tag_tid=11, entry_tid=13408' 2024-09-11T06:21:50.504 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=60, tag_tid=11, entry_tid=13504' 2024-09-11T06:21:50.504 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:50.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:50.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:50.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=12 2024-09-11T06:21:50.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=60, tag_tid=11, entry_tid=13504' 2024-09-11T06:21:50.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:50.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:50.507 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=11 2024-09-11T06:21:50.507 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:50.508 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:50.508 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:50.508 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:21:50.509 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=60, tag_tid=11, entry_tid=13504' 2024-09-11T06:21:50.509 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:50.509 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:50.511 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=13504 2024-09-11T06:21:50.511 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 12 = 11 -a 1 -le 13504 2024-09-11T06:21:50.511 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:21:50.511 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:21:50.511 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:21:50.712 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:21:50.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:50.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:21:50.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:21:50.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:21:50.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:21:50.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:21:50.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:21:50.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:21:50.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:50.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:50.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:21:50.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:21:50.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:21:50.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:21:50.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:21:50.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:21:50.716 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:21:50.716 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:21:50.716 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:21:50.716 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:50.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:21:50.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:21:50.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:21:50.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:21:50.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:21:50.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:21:50.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:21:50.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:50.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:21:50.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:50.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:21:50.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:50.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:21:50.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:21:50.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:21:50.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:21:50.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:21:50.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:21:50.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:21:50.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:21:50.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:50.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:50.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:50.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:21:50.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:21:50.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:21:50.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:21:50.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:21:50.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:50.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:50.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:50.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:21:50.873 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:21:50.874 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:21:50.874 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:50.875 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:50.875 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:50.875 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:21:50.875 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:50.875 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:50.875 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:50.875 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:21:50.876 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:50.876 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:50.876 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:50.877 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:50.877 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:50.878 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:50.924 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 15 2024-09-11T06:21:50.924 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:50.924 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:50.924 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:50.925 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=60, tag_tid=11, entry_tid=13608], [object_number=63, tag_tid=11, entry_tid=13607], [object_number=62, tag_tid=11, entry_tid=13606], [object_number=61, tag_tid=11, entry_tid=13605]]], state=connected] 2024-09-11T06:21:50.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:50.928 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:50.930 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:21:50.930 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:50.930 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:50.930 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:50.930 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:21:50.930 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:50.930 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:50.931 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:50.931 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:21:50.931 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:50.931 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:50.931 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:50.931 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:50.932 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:50.932 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:50.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 15 2024-09-11T06:21:50.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:50.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:50.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:50.979 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=60, tag_tid=11, entry_tid=13608], [object_number=63, tag_tid=11, entry_tid=13607], [object_number=62, tag_tid=11, entry_tid=13606], [object_number=61, tag_tid=11, entry_tid=13605]]], state=connected] 2024-09-11T06:21:50.982 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:50.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=60, tag_tid=11, entry_tid=13608' 2024-09-11T06:21:50.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=60, tag_tid=11, entry_tid=13608' 2024-09-11T06:21:50.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=60, tag_tid=11, entry_tid=13608' '!=' 'object_number=60, tag_tid=11, entry_tid=13504' 2024-09-11T06:21:50.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:21:50.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=60, tag_tid=11, entry_tid=13608' = 'object_number=60, tag_tid=11, entry_tid=13504' 2024-09-11T06:21:50.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=60, tag_tid=11, entry_tid=13608' 2024-09-11T06:21:50.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:50.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:50.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:50.985 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=12 2024-09-11T06:21:50.986 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=60, tag_tid=11, entry_tid=13608' 2024-09-11T06:21:50.986 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:50.986 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:50.986 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=11 2024-09-11T06:21:50.987 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:50.987 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:50.987 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:50.988 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:21:50.989 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=60, tag_tid=11, entry_tid=13608' 2024-09-11T06:21:50.989 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:50.989 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:50.990 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=13608 2024-09-11T06:21:50.990 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 12 = 11 -a 1 -le 13608 2024-09-11T06:21:50.990 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:21:50.991 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:21:50.991 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:21:51.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:21:51.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:51.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:21:51.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:21:51.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:21:51.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:21:51.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:21:51.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:21:51.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:21:51.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:51.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:51.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:21:51.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:21:51.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:21:51.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:21:51.197 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:21:51.197 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:21:51.197 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:21:51.197 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:21:51.197 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:21:51.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:51.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:21:51.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:21:51.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:21:51.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:21:51.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:21:51.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:21:51.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:21:51.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:51.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:21:51.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:51.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:21:51.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:51.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:21:51.199 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:21:51.199 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:21:51.199 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:21:51.199 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:21:51.199 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:21:51.199 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:21:51.199 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:21:51.199 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:51.199 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:51.200 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:51.200 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:21:51.200 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:21:51.200 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:21:51.200 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:21:51.200 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:21:51.222 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:51.223 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:51.223 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:51.223 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:21:51.352 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:21:51.354 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:21:51.354 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:51.354 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:51.354 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:51.354 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:21:51.354 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:51.354 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:51.354 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:51.354 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:21:51.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:51.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:51.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:51.357 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:51.358 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:51.358 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:51.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 15 2024-09-11T06:21:51.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:51.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:51.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:51.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=61, tag_tid=11, entry_tid=13713], [object_number=60, tag_tid=11, entry_tid=13712], [object_number=63, tag_tid=11, entry_tid=13711], [object_number=62, tag_tid=11, entry_tid=13710]]], state=connected] 2024-09-11T06:21:51.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:51.416 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:51.416 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:21:51.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:51.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:51.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:51.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:21:51.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:51.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:51.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:51.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:21:51.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:51.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:51.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:51.418 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:51.418 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:51.418 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:51.466 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 15 2024-09-11T06:21:51.466 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:51.466 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:51.466 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:51.466 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=61, tag_tid=11, entry_tid=13713], [object_number=60, tag_tid=11, entry_tid=13712], [object_number=63, tag_tid=11, entry_tid=13711], [object_number=62, tag_tid=11, entry_tid=13710]]], state=connected] 2024-09-11T06:21:51.469 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:51.471 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=61, tag_tid=11, entry_tid=13713' 2024-09-11T06:21:51.471 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=61, tag_tid=11, entry_tid=13713' 2024-09-11T06:21:51.472 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=61, tag_tid=11, entry_tid=13713' '!=' 'object_number=60, tag_tid=11, entry_tid=13608' 2024-09-11T06:21:51.472 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:21:51.472 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=61, tag_tid=11, entry_tid=13713' = 'object_number=60, tag_tid=11, entry_tid=13608' 2024-09-11T06:21:51.472 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=61, tag_tid=11, entry_tid=13713' 2024-09-11T06:21:51.472 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:51.472 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:51.472 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:51.472 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=12 2024-09-11T06:21:51.473 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=61, tag_tid=11, entry_tid=13713' 2024-09-11T06:21:51.473 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:51.473 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:51.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=11 2024-09-11T06:21:51.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:51.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:51.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:51.476 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:21:51.477 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=61, tag_tid=11, entry_tid=13713' 2024-09-11T06:21:51.477 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:51.477 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:51.478 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=13713 2024-09-11T06:21:51.478 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 12 = 11 -a 1 -le 13713 2024-09-11T06:21:51.478 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:21:51.478 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:21:51.478 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:21:51.680 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:21:51.680 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:51.680 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:21:51.680 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:21:51.680 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:21:51.681 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:21:51.681 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:21:51.681 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:21:51.681 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:21:51.681 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:51.681 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:51.681 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:21:51.681 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:21:51.681 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:21:51.681 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:21:51.682 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:21:51.682 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:21:51.684 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:21:51.684 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:21:51.684 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:21:51.684 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:51.684 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:21:51.684 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:21:51.685 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:21:51.685 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:21:51.685 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:21:51.685 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:21:51.685 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:21:51.685 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:51.685 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:21:51.685 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:51.685 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:21:51.685 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:51.685 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:21:51.685 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:21:51.685 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:21:51.686 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:21:51.686 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:21:51.686 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:21:51.686 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:21:51.686 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:21:51.686 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:51.686 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:51.686 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:51.686 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:21:51.686 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:21:51.686 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:21:51.686 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:21:51.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:21:51.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:51.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:51.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:51.711 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:21:51.839 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:21:51.840 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:21:51.840 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:51.841 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:51.841 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:51.841 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:21:51.841 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:51.841 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:51.841 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:51.841 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:21:51.841 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:51.842 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:51.842 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:51.842 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:51.843 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:51.843 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:51.890 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 15 2024-09-11T06:21:51.890 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:51.890 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:51.890 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:51.890 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=61, tag_tid=11, entry_tid=13821], [object_number=60, tag_tid=11, entry_tid=13820], [object_number=63, tag_tid=11, entry_tid=13819], [object_number=62, tag_tid=11, entry_tid=13818]]], state=connected] 2024-09-11T06:21:51.893 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:51.894 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:51.896 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:21:51.896 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:51.896 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:51.896 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:51.896 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:21:51.896 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:51.896 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:51.896 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:51.896 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:21:51.897 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:51.897 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:51.897 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:51.897 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:51.897 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:51.898 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:51.943 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 15 2024-09-11T06:21:51.943 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:51.943 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:51.943 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:51.943 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=61, tag_tid=11, entry_tid=13821], [object_number=60, tag_tid=11, entry_tid=13820], [object_number=63, tag_tid=11, entry_tid=13819], [object_number=62, tag_tid=11, entry_tid=13818]]], state=connected] 2024-09-11T06:21:51.947 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:51.949 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=61, tag_tid=11, entry_tid=13821' 2024-09-11T06:21:51.949 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=61, tag_tid=11, entry_tid=13821' 2024-09-11T06:21:51.949 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=61, tag_tid=11, entry_tid=13821' '!=' 'object_number=61, tag_tid=11, entry_tid=13713' 2024-09-11T06:21:51.949 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:21:51.949 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=61, tag_tid=11, entry_tid=13821' = 'object_number=61, tag_tid=11, entry_tid=13713' 2024-09-11T06:21:51.949 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=61, tag_tid=11, entry_tid=13821' 2024-09-11T06:21:51.949 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:51.949 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:51.949 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:51.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=12 2024-09-11T06:21:51.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=61, tag_tid=11, entry_tid=13821' 2024-09-11T06:21:51.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:51.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:51.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=11 2024-09-11T06:21:51.952 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:51.953 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:51.953 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:51.954 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:21:51.954 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=61, tag_tid=11, entry_tid=13821' 2024-09-11T06:21:51.954 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:51.954 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:51.955 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=13821 2024-09-11T06:21:51.955 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 12 = 11 -a 1 -le 13821 2024-09-11T06:21:51.955 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:21:51.955 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:21:51.955 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:21:52.157 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:21:52.157 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:52.157 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:21:52.157 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:21:52.157 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:21:52.157 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:21:52.157 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:21:52.157 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:21:52.157 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:21:52.157 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:52.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:52.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:21:52.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:21:52.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:21:52.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:21:52.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:21:52.159 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:21:52.161 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:21:52.162 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:21:52.162 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:21:52.162 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:52.162 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:21:52.162 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:21:52.162 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:21:52.162 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:21:52.162 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:21:52.162 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:21:52.162 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:21:52.162 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:52.162 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:21:52.162 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:52.163 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:21:52.163 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:52.163 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:21:52.163 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:21:52.163 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:21:52.163 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:21:52.163 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:21:52.163 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:21:52.163 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:21:52.163 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:21:52.163 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:52.163 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:52.163 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:52.163 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:21:52.163 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:21:52.164 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:21:52.164 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:21:52.164 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:21:52.187 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:52.187 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:52.187 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:52.188 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:21:52.314 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:21:52.315 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:21:52.315 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:52.315 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:52.315 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:52.315 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:21:52.315 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:52.315 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:52.315 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:52.315 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:21:52.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:52.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:52.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:52.318 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:52.319 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:52.319 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:52.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 15 2024-09-11T06:21:52.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:52.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:52.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:52.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=63, tag_tid=11, entry_tid=13923], [object_number=62, tag_tid=11, entry_tid=13922], [object_number=61, tag_tid=11, entry_tid=13921], [object_number=60, tag_tid=11, entry_tid=13920]]], state=connected] 2024-09-11T06:21:52.374 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:52.375 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:52.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:21:52.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:52.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:52.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:52.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:21:52.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:52.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:52.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:52.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:21:52.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:52.378 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:52.378 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:52.378 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:52.378 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:52.378 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:52.424 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 15 2024-09-11T06:21:52.424 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:52.424 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:52.424 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:52.424 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=63, tag_tid=11, entry_tid=13923], [object_number=62, tag_tid=11, entry_tid=13922], [object_number=61, tag_tid=11, entry_tid=13921], [object_number=60, tag_tid=11, entry_tid=13920]]], state=connected] 2024-09-11T06:21:52.427 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:52.429 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=63, tag_tid=11, entry_tid=13923' 2024-09-11T06:21:52.429 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=63, tag_tid=11, entry_tid=13923' 2024-09-11T06:21:52.429 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=63, tag_tid=11, entry_tid=13923' '!=' 'object_number=61, tag_tid=11, entry_tid=13821' 2024-09-11T06:21:52.430 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:21:52.430 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=63, tag_tid=11, entry_tid=13923' = 'object_number=61, tag_tid=11, entry_tid=13821' 2024-09-11T06:21:52.430 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=63, tag_tid=11, entry_tid=13923' 2024-09-11T06:21:52.430 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:52.430 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:52.430 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:52.430 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=12 2024-09-11T06:21:52.431 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=63, tag_tid=11, entry_tid=13923' 2024-09-11T06:21:52.431 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:52.431 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:52.432 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=11 2024-09-11T06:21:52.433 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:52.433 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:52.433 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:52.434 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:21:52.434 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=63, tag_tid=11, entry_tid=13923' 2024-09-11T06:21:52.435 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:52.435 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:52.436 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=13923 2024-09-11T06:21:52.436 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 12 = 11 -a 1 -le 13923 2024-09-11T06:21:52.436 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:21:52.436 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:21:52.436 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:21:52.638 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:21:52.638 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:52.638 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:21:52.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:21:52.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:21:52.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:21:52.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:21:52.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:21:52.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:21:52.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:52.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:52.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:21:52.640 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:21:52.640 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:21:52.640 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:21:52.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:21:52.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:21:52.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:21:52.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:21:52.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:21:52.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:52.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:21:52.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:21:52.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:21:52.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:21:52.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:21:52.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:21:52.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:21:52.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:52.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:21:52.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:52.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:21:52.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:52.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:21:52.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:21:52.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:21:52.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:21:52.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:21:52.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:21:52.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:21:52.649 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:21:52.649 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:52.649 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:52.649 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:52.649 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:21:52.649 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:21:52.649 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:21:52.649 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:21:52.649 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:21:52.668 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:52.669 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:52.669 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:52.669 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:21:52.797 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:21:52.798 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:21:52.798 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:52.798 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:52.798 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:52.798 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:21:52.798 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:52.799 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:52.799 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:52.799 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:21:52.799 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:52.799 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:52.800 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:52.801 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:52.801 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:52.801 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:52.857 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 15 2024-09-11T06:21:52.857 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:52.857 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:52.857 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:52.857 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=63, tag_tid=11, entry_tid=14023], [object_number=62, tag_tid=11, entry_tid=14022], [object_number=61, tag_tid=11, entry_tid=14021], [object_number=60, tag_tid=11, entry_tid=14020]]], state=connected] 2024-09-11T06:21:52.861 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:52.862 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:52.863 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:21:52.863 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:52.863 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:52.863 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:52.863 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:21:52.863 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:52.863 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:52.863 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:52.863 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:21:52.864 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:52.864 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:52.864 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:52.864 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:52.864 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:52.865 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:52.910 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 15 2024-09-11T06:21:52.911 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:52.911 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:52.911 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:52.911 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=63, tag_tid=11, entry_tid=14023], [object_number=62, tag_tid=11, entry_tid=14022], [object_number=61, tag_tid=11, entry_tid=14021], [object_number=60, tag_tid=11, entry_tid=14020]]], state=connected] 2024-09-11T06:21:52.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:52.916 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=63, tag_tid=11, entry_tid=14023' 2024-09-11T06:21:52.916 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=63, tag_tid=11, entry_tid=14023' 2024-09-11T06:21:52.916 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=63, tag_tid=11, entry_tid=14023' '!=' 'object_number=63, tag_tid=11, entry_tid=13923' 2024-09-11T06:21:52.917 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:21:52.917 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=63, tag_tid=11, entry_tid=14023' = 'object_number=63, tag_tid=11, entry_tid=13923' 2024-09-11T06:21:52.917 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=63, tag_tid=11, entry_tid=14023' 2024-09-11T06:21:52.917 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:52.917 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:52.917 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:52.917 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=12 2024-09-11T06:21:52.918 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=63, tag_tid=11, entry_tid=14023' 2024-09-11T06:21:52.918 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:52.918 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:52.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=11 2024-09-11T06:21:52.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:52.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:52.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:52.921 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:21:52.921 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=63, tag_tid=11, entry_tid=14023' 2024-09-11T06:21:52.921 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:52.922 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:52.923 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=14023 2024-09-11T06:21:52.923 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 12 = 11 -a 1 -le 14023 2024-09-11T06:21:52.923 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:21:52.923 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:21:52.923 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:21:53.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:21:53.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:53.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:21:53.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:21:53.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:21:53.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:21:53.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:21:53.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:21:53.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:21:53.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:53.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:53.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:21:53.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:21:53.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:21:53.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:21:53.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:21:53.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:21:53.129 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:21:53.129 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:21:53.129 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:21:53.129 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:53.129 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:21:53.130 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:21:53.130 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:21:53.130 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:21:53.130 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:21:53.130 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:21:53.130 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:21:53.130 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:53.130 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:21:53.130 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:53.130 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:21:53.130 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:53.130 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:21:53.130 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:21:53.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:21:53.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:21:53.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:21:53.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:21:53.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:21:53.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:21:53.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:53.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:53.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:53.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:21:53.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:21:53.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:21:53.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:21:53.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:21:53.156 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:53.157 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:53.157 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:53.157 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:21:53.289 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:21:53.290 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:21:53.290 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:53.290 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:53.290 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:53.290 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:21:53.290 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:53.290 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:53.290 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:53.290 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:21:53.291 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:53.291 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:53.291 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:53.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:53.293 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:53.293 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:53.344 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 15 2024-09-11T06:21:53.344 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:53.344 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:53.345 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:53.345 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=60, tag_tid=11, entry_tid=14124], [object_number=63, tag_tid=11, entry_tid=14123], [object_number=62, tag_tid=11, entry_tid=14122], [object_number=61, tag_tid=11, entry_tid=14121]]], state=connected] 2024-09-11T06:21:53.348 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:53.349 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:53.350 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:21:53.350 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:53.350 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:53.350 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:53.350 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:21:53.351 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:53.351 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:53.351 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:53.351 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:21:53.351 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:53.351 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:53.351 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:53.352 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:53.352 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:53.352 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:53.398 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 15 2024-09-11T06:21:53.398 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:53.398 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:53.399 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:53.399 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=60, tag_tid=11, entry_tid=14124], [object_number=63, tag_tid=11, entry_tid=14123], [object_number=62, tag_tid=11, entry_tid=14122], [object_number=61, tag_tid=11, entry_tid=14121]]], state=connected] 2024-09-11T06:21:53.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:53.404 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=60, tag_tid=11, entry_tid=14124' 2024-09-11T06:21:53.404 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=60, tag_tid=11, entry_tid=14124' 2024-09-11T06:21:53.404 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=60, tag_tid=11, entry_tid=14124' '!=' 'object_number=63, tag_tid=11, entry_tid=14023' 2024-09-11T06:21:53.404 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:21:53.404 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=60, tag_tid=11, entry_tid=14124' = 'object_number=63, tag_tid=11, entry_tid=14023' 2024-09-11T06:21:53.404 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=60, tag_tid=11, entry_tid=14124' 2024-09-11T06:21:53.404 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:53.404 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:53.404 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:53.405 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=12 2024-09-11T06:21:53.405 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=60, tag_tid=11, entry_tid=14124' 2024-09-11T06:21:53.406 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:53.406 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:53.406 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=11 2024-09-11T06:21:53.407 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:53.407 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:53.407 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:53.408 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:21:53.409 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=60, tag_tid=11, entry_tid=14124' 2024-09-11T06:21:53.409 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:53.409 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:53.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=14124 2024-09-11T06:21:53.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 12 = 11 -a 1 -le 14124 2024-09-11T06:21:53.411 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:21:53.411 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:21:53.411 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:21:53.612 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:21:53.612 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:53.613 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:21:53.613 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:21:53.613 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:21:53.613 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:21:53.613 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:21:53.613 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:21:53.613 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:21:53.613 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:53.613 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:53.613 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:21:53.613 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:21:53.613 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:21:53.613 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:21:53.614 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:21:53.614 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:21:53.618 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:21:53.618 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:21:53.618 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:21:53.618 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:53.618 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:21:53.618 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:21:53.618 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:21:53.618 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:21:53.618 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:21:53.618 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:21:53.618 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:21:53.618 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:53.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:21:53.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:53.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:21:53.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:53.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:21:53.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:21:53.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:21:53.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:21:53.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:21:53.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:21:53.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:21:53.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:21:53.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:53.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:53.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:53.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:21:53.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:21:53.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:21:53.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:21:53.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:21:53.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:53.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:53.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:53.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:21:53.774 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:21:53.775 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:21:53.776 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:53.776 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:53.776 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:53.776 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:21:53.776 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:53.776 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:53.776 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:53.776 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:21:53.777 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:53.777 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:53.777 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:53.778 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:53.779 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:53.779 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:53.831 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 15 2024-09-11T06:21:53.831 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:53.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:53.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:53.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=63, tag_tid=11, entry_tid=14231], [object_number=62, tag_tid=11, entry_tid=14230], [object_number=61, tag_tid=11, entry_tid=14229], [object_number=60, tag_tid=11, entry_tid=14228]]], state=connected] 2024-09-11T06:21:53.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:53.837 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:53.837 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:21:53.838 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:53.838 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:53.838 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:53.838 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:21:53.838 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:53.838 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:53.838 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:53.838 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:21:53.838 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:53.839 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:53.839 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:53.839 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:53.839 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:53.839 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:53.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 15 2024-09-11T06:21:53.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:53.887 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:53.887 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:53.887 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=63, tag_tid=11, entry_tid=14231], [object_number=62, tag_tid=11, entry_tid=14230], [object_number=61, tag_tid=11, entry_tid=14229], [object_number=60, tag_tid=11, entry_tid=14228]]], state=connected] 2024-09-11T06:21:53.890 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:53.891 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=63, tag_tid=11, entry_tid=14231' 2024-09-11T06:21:53.892 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=63, tag_tid=11, entry_tid=14231' 2024-09-11T06:21:53.892 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=63, tag_tid=11, entry_tid=14231' '!=' 'object_number=60, tag_tid=11, entry_tid=14124' 2024-09-11T06:21:53.892 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:21:53.892 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=63, tag_tid=11, entry_tid=14231' = 'object_number=60, tag_tid=11, entry_tid=14124' 2024-09-11T06:21:53.892 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=63, tag_tid=11, entry_tid=14231' 2024-09-11T06:21:53.892 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:53.893 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:53.893 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:53.893 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=12 2024-09-11T06:21:53.894 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=63, tag_tid=11, entry_tid=14231' 2024-09-11T06:21:53.894 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:53.894 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:53.895 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=11 2024-09-11T06:21:53.895 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:53.895 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:53.896 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:53.896 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:21:53.897 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=63, tag_tid=11, entry_tid=14231' 2024-09-11T06:21:53.897 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:53.897 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:53.899 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=14231 2024-09-11T06:21:53.899 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 12 = 11 -a 1 -le 14231 2024-09-11T06:21:53.899 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:21:53.899 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:21:53.899 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:21:54.100 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:21:54.101 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:54.101 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:21:54.101 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:21:54.101 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:21:54.101 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:21:54.101 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:21:54.101 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:21:54.101 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:21:54.101 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:54.101 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:54.101 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:21:54.102 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:21:54.102 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:21:54.102 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:21:54.102 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:21:54.102 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:21:54.105 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:21:54.105 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:21:54.105 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:21:54.105 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:54.105 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:21:54.106 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:21:54.106 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:21:54.106 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:21:54.106 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:21:54.106 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:21:54.106 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:21:54.106 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:54.106 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:21:54.106 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:54.106 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:21:54.106 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:54.106 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:21:54.107 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:21:54.107 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:21:54.107 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:21:54.107 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:21:54.107 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:21:54.107 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:21:54.107 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:21:54.107 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:54.107 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:54.107 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:54.107 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:21:54.107 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:21:54.107 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:21:54.108 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:21:54.108 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:21:54.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:54.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:54.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:54.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:21:54.266 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:21:54.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:21:54.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:54.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:54.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:54.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:21:54.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:54.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:54.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:54.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:21:54.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:54.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:54.269 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:54.270 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:54.271 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:54.271 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:54.324 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 15 2024-09-11T06:21:54.324 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:54.324 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:54.324 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:54.324 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=63, tag_tid=11, entry_tid=14327], [object_number=62, tag_tid=11, entry_tid=14326], [object_number=65, tag_tid=11, entry_tid=14325], [object_number=64, tag_tid=11, entry_tid=14324]]], state=connected] 2024-09-11T06:21:54.327 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:54.329 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:54.330 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:21:54.330 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:54.330 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:54.330 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:54.330 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:21:54.330 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:54.330 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:54.330 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:54.330 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:21:54.331 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:54.331 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:54.331 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:54.331 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:54.331 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:54.331 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:54.378 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 15 2024-09-11T06:21:54.378 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:54.378 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:54.378 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:54.378 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=63, tag_tid=11, entry_tid=14327], [object_number=62, tag_tid=11, entry_tid=14326], [object_number=65, tag_tid=11, entry_tid=14325], [object_number=64, tag_tid=11, entry_tid=14324]]], state=connected] 2024-09-11T06:21:54.381 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:54.383 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=63, tag_tid=11, entry_tid=14327' 2024-09-11T06:21:54.383 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=63, tag_tid=11, entry_tid=14327' 2024-09-11T06:21:54.383 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=63, tag_tid=11, entry_tid=14327' '!=' 'object_number=63, tag_tid=11, entry_tid=14231' 2024-09-11T06:21:54.383 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:21:54.383 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=63, tag_tid=11, entry_tid=14327' = 'object_number=63, tag_tid=11, entry_tid=14231' 2024-09-11T06:21:54.383 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=63, tag_tid=11, entry_tid=14327' 2024-09-11T06:21:54.384 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:54.384 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:54.384 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:54.384 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=12 2024-09-11T06:21:54.385 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=63, tag_tid=11, entry_tid=14327' 2024-09-11T06:21:54.385 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:54.385 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:54.386 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=11 2024-09-11T06:21:54.387 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:54.387 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:54.387 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:54.388 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:21:54.388 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=63, tag_tid=11, entry_tid=14327' 2024-09-11T06:21:54.389 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:54.389 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:54.390 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=14327 2024-09-11T06:21:54.390 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 12 = 11 -a 1 -le 14327 2024-09-11T06:21:54.390 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:21:54.390 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:21:54.390 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:21:54.592 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:21:54.592 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:54.592 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:21:54.592 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:21:54.592 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:21:54.592 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:21:54.592 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:21:54.592 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:21:54.592 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:21:54.592 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:54.592 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:54.593 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:21:54.593 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:21:54.593 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:21:54.593 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:21:54.593 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:21:54.594 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:21:54.597 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:21:54.598 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:21:54.598 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:21:54.598 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:54.598 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:21:54.598 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:21:54.598 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:21:54.598 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:21:54.598 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:21:54.598 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:21:54.598 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:21:54.598 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:54.598 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:21:54.598 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:54.599 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:21:54.599 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:54.599 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:21:54.599 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:21:54.599 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:21:54.599 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:21:54.599 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:21:54.599 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:21:54.599 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:21:54.599 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:21:54.599 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:54.599 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:54.599 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:54.600 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:21:54.600 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:21:54.600 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:21:54.600 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:21:54.600 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:21:54.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:54.623 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:54.623 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:54.623 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:21:54.750 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:21:54.751 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:21:54.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:54.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:54.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:54.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:21:54.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:54.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:54.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:54.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:21:54.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:54.753 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:54.753 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:54.754 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:54.754 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:54.754 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:54.806 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 15 2024-09-11T06:21:54.807 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:54.807 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:54.807 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:54.807 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=64, tag_tid=11, entry_tid=14428], [object_number=63, tag_tid=11, entry_tid=14427], [object_number=66, tag_tid=11, entry_tid=14426], [object_number=65, tag_tid=11, entry_tid=14425]]], state=connected] 2024-09-11T06:21:54.810 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:54.813 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:54.813 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:21:54.813 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:54.813 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:54.813 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:54.813 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:21:54.813 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:54.813 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:54.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:54.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:21:54.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:54.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:54.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:54.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:54.815 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:54.815 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:54.860 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 15 2024-09-11T06:21:54.861 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:54.861 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:54.861 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:54.861 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=64, tag_tid=11, entry_tid=14428], [object_number=63, tag_tid=11, entry_tid=14427], [object_number=66, tag_tid=11, entry_tid=14426], [object_number=65, tag_tid=11, entry_tid=14425]]], state=connected] 2024-09-11T06:21:54.864 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:54.866 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=64, tag_tid=11, entry_tid=14428' 2024-09-11T06:21:54.866 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=64, tag_tid=11, entry_tid=14428' 2024-09-11T06:21:54.866 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=64, tag_tid=11, entry_tid=14428' '!=' 'object_number=63, tag_tid=11, entry_tid=14327' 2024-09-11T06:21:54.866 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:21:54.866 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=64, tag_tid=11, entry_tid=14428' = 'object_number=63, tag_tid=11, entry_tid=14327' 2024-09-11T06:21:54.866 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=64, tag_tid=11, entry_tid=14428' 2024-09-11T06:21:54.866 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:54.867 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:54.867 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:54.867 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=12 2024-09-11T06:21:54.868 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=64, tag_tid=11, entry_tid=14428' 2024-09-11T06:21:54.868 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:54.868 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:54.869 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=11 2024-09-11T06:21:54.869 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:54.870 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:54.870 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:54.870 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:21:54.871 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=64, tag_tid=11, entry_tid=14428' 2024-09-11T06:21:54.871 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:54.871 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:54.872 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=14428 2024-09-11T06:21:54.873 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 12 = 11 -a 1 -le 14428 2024-09-11T06:21:54.873 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:21:54.873 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:21:54.873 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:21:55.074 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:21:55.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:55.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:21:55.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:21:55.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:21:55.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:21:55.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:21:55.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:21:55.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:21:55.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:55.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:55.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:21:55.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:21:55.076 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:21:55.076 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:21:55.076 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:21:55.076 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:21:55.079 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:21:55.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:21:55.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:21:55.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:55.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:21:55.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:21:55.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:21:55.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:21:55.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:21:55.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:21:55.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:21:55.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:55.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:21:55.081 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:55.081 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:21:55.081 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:55.081 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:21:55.081 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:21:55.081 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:21:55.081 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:21:55.081 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:21:55.081 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:21:55.081 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:21:55.081 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:21:55.081 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:55.081 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:55.081 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:55.082 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:21:55.082 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:21:55.082 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:21:55.082 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:21:55.082 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:21:55.105 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:55.106 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:55.106 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:55.106 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:21:55.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:21:55.244 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:21:55.244 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:55.244 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:55.245 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:55.245 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:21:55.245 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:55.245 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:55.245 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:55.245 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:21:55.245 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:55.245 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:55.246 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:55.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:55.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:55.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:55.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 16 2024-09-11T06:21:55.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:55.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:55.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:55.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=67, tag_tid=11, entry_tid=14531], [object_number=66, tag_tid=11, entry_tid=14530], [object_number=65, tag_tid=11, entry_tid=14529], [object_number=64, tag_tid=11, entry_tid=14528]]], state=connected] 2024-09-11T06:21:55.307 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:55.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:55.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:21:55.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:55.310 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:55.310 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:55.310 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:21:55.310 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:55.310 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:55.310 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:55.310 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:21:55.310 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:55.310 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:55.310 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:55.311 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:55.311 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:55.311 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:55.356 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 16 2024-09-11T06:21:55.357 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:55.357 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:55.357 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:55.357 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=67, tag_tid=11, entry_tid=14531], [object_number=66, tag_tid=11, entry_tid=14530], [object_number=65, tag_tid=11, entry_tid=14529], [object_number=64, tag_tid=11, entry_tid=14528]]], state=connected] 2024-09-11T06:21:55.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:55.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=67, tag_tid=11, entry_tid=14531' 2024-09-11T06:21:55.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=67, tag_tid=11, entry_tid=14531' 2024-09-11T06:21:55.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=67, tag_tid=11, entry_tid=14531' '!=' 'object_number=64, tag_tid=11, entry_tid=14428' 2024-09-11T06:21:55.362 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:21:55.362 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=67, tag_tid=11, entry_tid=14531' = 'object_number=64, tag_tid=11, entry_tid=14428' 2024-09-11T06:21:55.362 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=67, tag_tid=11, entry_tid=14531' 2024-09-11T06:21:55.362 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:55.362 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:55.362 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:55.363 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=12 2024-09-11T06:21:55.364 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=67, tag_tid=11, entry_tid=14531' 2024-09-11T06:21:55.364 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:55.364 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:55.365 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=11 2024-09-11T06:21:55.365 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:55.365 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:55.365 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:55.366 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:21:55.367 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=67, tag_tid=11, entry_tid=14531' 2024-09-11T06:21:55.367 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:55.367 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:55.368 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=14531 2024-09-11T06:21:55.368 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 12 = 11 -a 1 -le 14531 2024-09-11T06:21:55.368 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:21:55.369 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:21:55.369 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:21:55.570 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:21:55.570 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:55.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:21:55.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:21:55.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:21:55.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:21:55.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:21:55.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:21:55.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:21:55.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:55.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:55.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:21:55.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:21:55.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:21:55.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:21:55.572 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:21:55.572 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:21:55.575 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:21:55.575 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:21:55.575 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:21:55.575 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:55.575 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:21:55.575 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:21:55.575 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:21:55.575 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:21:55.575 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:21:55.575 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:21:55.575 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:21:55.576 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:55.576 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:21:55.576 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:55.576 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:21:55.576 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:55.577 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:21:55.577 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:21:55.577 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:21:55.577 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:21:55.577 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:21:55.577 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:21:55.577 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:21:55.577 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:21:55.577 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:55.577 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:55.578 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:55.578 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:21:55.578 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:21:55.578 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:21:55.578 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:21:55.578 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:21:55.602 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:55.602 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:55.602 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:55.602 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:21:55.735 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:21:55.736 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:21:55.736 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:55.736 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:55.736 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:55.736 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:21:55.736 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:55.736 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:55.736 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:55.736 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:21:55.737 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:55.737 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:55.737 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:55.739 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:55.739 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:55.740 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:55.785 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 16 2024-09-11T06:21:55.785 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:55.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:55.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:55.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=66, tag_tid=11, entry_tid=14638], [object_number=65, tag_tid=11, entry_tid=14637], [object_number=64, tag_tid=11, entry_tid=14636], [object_number=67, tag_tid=11, entry_tid=14635]]], state=connected] 2024-09-11T06:21:55.789 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:55.790 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:55.791 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:21:55.791 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:55.791 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:55.791 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:55.791 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:21:55.791 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:55.791 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:55.791 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:55.792 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:21:55.792 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:55.792 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:55.792 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:55.792 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:55.792 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:55.793 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:55.839 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 16 2024-09-11T06:21:55.839 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:55.839 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:55.840 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:55.840 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=66, tag_tid=11, entry_tid=14638], [object_number=65, tag_tid=11, entry_tid=14637], [object_number=64, tag_tid=11, entry_tid=14636], [object_number=67, tag_tid=11, entry_tid=14635]]], state=connected] 2024-09-11T06:21:55.843 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:55.845 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=66, tag_tid=11, entry_tid=14638' 2024-09-11T06:21:55.845 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=66, tag_tid=11, entry_tid=14638' 2024-09-11T06:21:55.845 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=66, tag_tid=11, entry_tid=14638' '!=' 'object_number=67, tag_tid=11, entry_tid=14531' 2024-09-11T06:21:55.845 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:21:55.845 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=66, tag_tid=11, entry_tid=14638' = 'object_number=67, tag_tid=11, entry_tid=14531' 2024-09-11T06:21:55.845 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=66, tag_tid=11, entry_tid=14638' 2024-09-11T06:21:55.845 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:55.846 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:55.846 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:55.846 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=12 2024-09-11T06:21:55.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=66, tag_tid=11, entry_tid=14638' 2024-09-11T06:21:55.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:55.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:55.848 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=11 2024-09-11T06:21:55.848 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:55.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:55.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:55.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:21:55.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=66, tag_tid=11, entry_tid=14638' 2024-09-11T06:21:55.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:55.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:55.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=14638 2024-09-11T06:21:55.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 12 = 11 -a 1 -le 14638 2024-09-11T06:21:55.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:21:55.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:21:55.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:21:56.053 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:21:56.053 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:56.053 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:21:56.054 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:21:56.054 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:21:56.054 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:21:56.054 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:21:56.054 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:21:56.054 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:21:56.054 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:56.054 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:56.054 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:21:56.054 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:21:56.054 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:21:56.054 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:21:56.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:21:56.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:21:56.058 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:21:56.058 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:21:56.058 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:21:56.058 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:56.059 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:21:56.059 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:21:56.059 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:21:56.059 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:21:56.059 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:21:56.059 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:21:56.059 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:21:56.059 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:56.059 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:21:56.059 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:56.059 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:21:56.059 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:56.059 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:21:56.059 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:21:56.060 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:21:56.060 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:21:56.060 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:21:56.060 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:21:56.060 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:21:56.060 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:21:56.060 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:56.060 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:56.060 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:56.060 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:21:56.060 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:21:56.060 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:21:56.060 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:21:56.060 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:21:56.084 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:56.085 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:56.085 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:56.085 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:21:56.215 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:21:56.216 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:21:56.216 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:56.216 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:56.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:56.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:21:56.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:56.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:56.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:56.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:21:56.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:56.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:56.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:56.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:56.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:56.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:56.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 16 2024-09-11T06:21:56.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:56.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:56.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:56.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=67, tag_tid=11, entry_tid=14735], [object_number=66, tag_tid=11, entry_tid=14734], [object_number=65, tag_tid=11, entry_tid=14733], [object_number=64, tag_tid=11, entry_tid=14732]]], state=connected] 2024-09-11T06:21:56.486 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:56.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:56.490 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:21:56.490 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:56.490 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:56.490 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:56.490 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:21:56.490 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:56.490 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:56.490 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:56.490 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:21:56.490 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:56.491 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:56.491 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:56.492 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:56.493 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:56.493 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:56.548 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 16 2024-09-11T06:21:56.548 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:56.548 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:56.548 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:56.548 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=67, tag_tid=11, entry_tid=14735], [object_number=66, tag_tid=11, entry_tid=14734], [object_number=65, tag_tid=11, entry_tid=14733], [object_number=64, tag_tid=11, entry_tid=14732]]], state=connected] 2024-09-11T06:21:56.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:56.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=67, tag_tid=11, entry_tid=14735' 2024-09-11T06:21:56.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=67, tag_tid=11, entry_tid=14735' 2024-09-11T06:21:56.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=67, tag_tid=11, entry_tid=14735' '!=' 'object_number=66, tag_tid=11, entry_tid=14638' 2024-09-11T06:21:56.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:21:56.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=67, tag_tid=11, entry_tid=14735' = 'object_number=66, tag_tid=11, entry_tid=14638' 2024-09-11T06:21:56.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=67, tag_tid=11, entry_tid=14735' 2024-09-11T06:21:56.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:56.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:56.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:56.555 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=12 2024-09-11T06:21:56.556 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=67, tag_tid=11, entry_tid=14735' 2024-09-11T06:21:56.556 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:56.556 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:56.557 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=11 2024-09-11T06:21:56.558 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:56.558 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:56.558 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:56.559 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:21:56.560 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=67, tag_tid=11, entry_tid=14735' 2024-09-11T06:21:56.560 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:56.560 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:56.566 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=14735 2024-09-11T06:21:56.566 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 12 = 11 -a 1 -le 14735 2024-09-11T06:21:56.566 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:21:56.566 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:21:56.566 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:21:56.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:21:56.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:56.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:21:56.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:21:56.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:21:56.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:21:56.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:21:56.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:21:56.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:21:56.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:56.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:56.764 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:21:56.764 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:21:56.764 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:21:56.764 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:21:56.764 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:21:56.765 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:21:56.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:21:56.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:21:56.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:21:56.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:56.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:21:56.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:21:56.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:21:56.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:21:56.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:21:56.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:21:56.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:21:56.768 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:56.768 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:21:56.768 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:56.768 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:21:56.768 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:56.769 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:21:56.769 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:21:56.769 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:21:56.769 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:21:56.769 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:21:56.769 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:21:56.769 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:21:56.769 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:21:56.769 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:56.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:56.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:56.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:21:56.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:21:56.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:21:56.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:21:56.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:21:56.793 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:56.794 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:56.794 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:56.794 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:21:56.921 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:21:56.922 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:21:56.922 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:56.922 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:56.922 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:56.922 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:21:56.922 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:56.922 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:56.922 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:56.922 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:21:56.923 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:56.923 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:56.923 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:56.926 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:56.926 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:56.926 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:56.975 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 16 2024-09-11T06:21:56.975 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:56.975 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:56.975 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:56.975 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=64, tag_tid=11, entry_tid=14868], [object_number=67, tag_tid=11, entry_tid=14867], [object_number=66, tag_tid=11, entry_tid=14866], [object_number=65, tag_tid=11, entry_tid=14865]]], state=connected] 2024-09-11T06:21:56.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:56.980 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:56.981 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:21:56.981 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:56.981 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:56.981 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:56.981 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:21:56.981 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:56.982 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:56.982 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:56.982 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:21:56.982 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:56.982 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:56.982 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:56.983 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:56.983 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:56.983 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:57.028 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 16 2024-09-11T06:21:57.028 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:57.029 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:57.029 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:57.029 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=64, tag_tid=11, entry_tid=14868], [object_number=67, tag_tid=11, entry_tid=14867], [object_number=66, tag_tid=11, entry_tid=14866], [object_number=65, tag_tid=11, entry_tid=14865]]], state=connected] 2024-09-11T06:21:57.032 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:57.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=64, tag_tid=11, entry_tid=14868' 2024-09-11T06:21:57.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=64, tag_tid=11, entry_tid=14868' 2024-09-11T06:21:57.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=64, tag_tid=11, entry_tid=14868' '!=' 'object_number=67, tag_tid=11, entry_tid=14735' 2024-09-11T06:21:57.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:21:57.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=64, tag_tid=11, entry_tid=14868' = 'object_number=67, tag_tid=11, entry_tid=14735' 2024-09-11T06:21:57.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=64, tag_tid=11, entry_tid=14868' 2024-09-11T06:21:57.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:57.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:57.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:57.035 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=12 2024-09-11T06:21:57.035 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=64, tag_tid=11, entry_tid=14868' 2024-09-11T06:21:57.036 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:57.036 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:57.036 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=11 2024-09-11T06:21:57.037 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:57.037 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:57.037 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:57.038 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:21:57.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=64, tag_tid=11, entry_tid=14868' 2024-09-11T06:21:57.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:57.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:57.040 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=14868 2024-09-11T06:21:57.040 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 12 = 11 -a 1 -le 14868 2024-09-11T06:21:57.040 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:21:57.040 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:21:57.040 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:21:57.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:21:57.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:57.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:21:57.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:21:57.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:21:57.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:21:57.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:21:57.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:21:57.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:21:57.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:57.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:57.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:21:57.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:21:57.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:21:57.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:21:57.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:21:57.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:21:57.246 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:21:57.246 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:21:57.246 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:21:57.246 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:57.246 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:21:57.246 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:21:57.246 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:21:57.246 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:21:57.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:21:57.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:21:57.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:21:57.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:57.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:21:57.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:57.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:21:57.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:57.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:21:57.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:21:57.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:21:57.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:21:57.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:21:57.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:21:57.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:21:57.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:21:57.250 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:57.250 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:57.250 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:57.250 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:21:57.250 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:21:57.250 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:21:57.250 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:21:57.250 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:21:57.272 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:57.273 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:57.273 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:57.273 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:21:57.403 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:21:57.404 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:21:57.404 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:57.404 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:57.404 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:57.404 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:21:57.404 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:57.404 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:57.404 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:57.405 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:21:57.405 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:57.405 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:57.405 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:57.406 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:57.407 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:57.407 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:57.459 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 16 2024-09-11T06:21:57.459 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:57.459 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:57.459 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:57.459 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=64, tag_tid=11, entry_tid=14960], [object_number=67, tag_tid=11, entry_tid=14959], [object_number=66, tag_tid=11, entry_tid=14958], [object_number=65, tag_tid=11, entry_tid=14957]]], state=connected] 2024-09-11T06:21:57.463 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:57.464 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:57.465 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:21:57.465 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:57.466 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:57.466 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:57.466 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:21:57.466 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:57.466 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:57.466 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:57.466 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:21:57.466 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:57.466 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:57.466 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:57.466 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:57.467 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:57.467 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:57.514 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 16 2024-09-11T06:21:57.514 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:57.514 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:57.514 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:57.514 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=64, tag_tid=11, entry_tid=14960], [object_number=67, tag_tid=11, entry_tid=14959], [object_number=66, tag_tid=11, entry_tid=14958], [object_number=65, tag_tid=11, entry_tid=14957]]], state=connected] 2024-09-11T06:21:57.517 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:57.519 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=64, tag_tid=11, entry_tid=14960' 2024-09-11T06:21:57.519 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=64, tag_tid=11, entry_tid=14960' 2024-09-11T06:21:57.520 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=64, tag_tid=11, entry_tid=14960' '!=' 'object_number=64, tag_tid=11, entry_tid=14868' 2024-09-11T06:21:57.520 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:21:57.520 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=64, tag_tid=11, entry_tid=14960' = 'object_number=64, tag_tid=11, entry_tid=14868' 2024-09-11T06:21:57.520 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=64, tag_tid=11, entry_tid=14960' 2024-09-11T06:21:57.520 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:57.520 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:57.520 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:57.521 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=12 2024-09-11T06:21:57.521 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=64, tag_tid=11, entry_tid=14960' 2024-09-11T06:21:57.521 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:57.521 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:57.522 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=11 2024-09-11T06:21:57.523 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:57.523 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:57.523 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:57.524 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:21:57.525 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=64, tag_tid=11, entry_tid=14960' 2024-09-11T06:21:57.525 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:57.525 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:57.526 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=14960 2024-09-11T06:21:57.526 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 12 = 11 -a 1 -le 14960 2024-09-11T06:21:57.526 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:21:57.526 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:21:57.526 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:21:57.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:21:57.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:57.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:21:57.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:21:57.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:21:57.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:21:57.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:21:57.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:21:57.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:21:57.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:57.729 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:57.729 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:21:57.729 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:21:57.729 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:21:57.729 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:21:57.729 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:21:57.730 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:21:57.732 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:21:57.732 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:21:57.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:21:57.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:57.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:21:57.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:21:57.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:21:57.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:21:57.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:21:57.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:21:57.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:21:57.734 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:57.734 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:21:57.734 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:57.734 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:21:57.734 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:57.735 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:21:57.735 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:21:57.735 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:21:57.735 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:21:57.735 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:21:57.735 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:21:57.736 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:21:57.736 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:21:57.736 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:57.736 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:57.736 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:57.736 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:21:57.736 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:21:57.736 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:21:57.736 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:21:57.736 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:21:57.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:57.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:57.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:57.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:21:57.894 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:21:57.896 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:21:57.896 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:57.896 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:57.896 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:57.896 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:21:57.896 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:57.896 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:57.896 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:57.896 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:21:57.897 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:57.897 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:57.897 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:57.899 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:57.899 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:57.899 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:57.944 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 16 2024-09-11T06:21:57.944 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:57.944 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:57.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:57.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=67, tag_tid=11, entry_tid=15059], [object_number=66, tag_tid=11, entry_tid=15058], [object_number=65, tag_tid=11, entry_tid=15057], [object_number=64, tag_tid=11, entry_tid=15056]]], state=connected] 2024-09-11T06:21:57.948 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:57.949 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:57.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:21:57.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:57.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:57.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:57.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:21:57.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:57.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:57.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:57.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:21:57.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:57.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:57.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:57.952 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:57.952 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:57.952 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:57.997 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 16 2024-09-11T06:21:57.997 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:57.997 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:57.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:57.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=67, tag_tid=11, entry_tid=15059], [object_number=66, tag_tid=11, entry_tid=15058], [object_number=65, tag_tid=11, entry_tid=15057], [object_number=64, tag_tid=11, entry_tid=15056]]], state=connected] 2024-09-11T06:21:58.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:58.003 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=67, tag_tid=11, entry_tid=15059' 2024-09-11T06:21:58.003 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=67, tag_tid=11, entry_tid=15059' 2024-09-11T06:21:58.003 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=67, tag_tid=11, entry_tid=15059' '!=' 'object_number=64, tag_tid=11, entry_tid=14960' 2024-09-11T06:21:58.003 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:21:58.003 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=67, tag_tid=11, entry_tid=15059' = 'object_number=64, tag_tid=11, entry_tid=14960' 2024-09-11T06:21:58.003 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=67, tag_tid=11, entry_tid=15059' 2024-09-11T06:21:58.003 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:58.004 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:58.004 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:58.004 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=12 2024-09-11T06:21:58.005 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=67, tag_tid=11, entry_tid=15059' 2024-09-11T06:21:58.005 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:58.005 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:58.005 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=11 2024-09-11T06:21:58.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:58.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:58.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:58.007 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:21:58.008 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=67, tag_tid=11, entry_tid=15059' 2024-09-11T06:21:58.008 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:58.008 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:58.009 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=15059 2024-09-11T06:21:58.009 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 12 = 11 -a 1 -le 15059 2024-09-11T06:21:58.009 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:21:58.009 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:21:58.009 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:21:58.211 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:21:58.211 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:58.211 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:21:58.211 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:21:58.211 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:21:58.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:21:58.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:21:58.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:21:58.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:21:58.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:58.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:58.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:21:58.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:21:58.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:21:58.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:21:58.213 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:21:58.213 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:21:58.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:21:58.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:21:58.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:21:58.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:58.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:21:58.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:21:58.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:21:58.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:21:58.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:21:58.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:21:58.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:21:58.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:58.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:21:58.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:58.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:21:58.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:58.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:21:58.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:21:58.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:21:58.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:21:58.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:21:58.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:21:58.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:21:58.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:21:58.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:58.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:58.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:58.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:21:58.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:21:58.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:21:58.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:21:58.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:21:58.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:58.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:58.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:58.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:21:58.380 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:21:58.381 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:21:58.381 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:58.381 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:58.381 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:58.381 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:21:58.381 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:58.382 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:58.382 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:58.382 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:21:58.382 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:58.382 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:58.382 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:58.385 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:58.385 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:58.385 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:58.430 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 16 2024-09-11T06:21:58.430 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:58.431 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:58.431 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:58.431 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=64, tag_tid=11, entry_tid=15128], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126], [object_number=65, tag_tid=11, entry_tid=15125]]], state=connected] 2024-09-11T06:21:58.434 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:58.435 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:58.437 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:21:58.437 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:58.437 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:58.437 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:58.437 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:21:58.437 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:58.437 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:58.437 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:58.437 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:21:58.437 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:58.437 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:58.438 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:58.438 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:58.438 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:58.438 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:58.485 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 16 2024-09-11T06:21:58.485 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:58.485 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:58.485 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:58.485 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=64, tag_tid=11, entry_tid=15128], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126], [object_number=65, tag_tid=11, entry_tid=15125]]], state=connected] 2024-09-11T06:21:58.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:58.490 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=64, tag_tid=11, entry_tid=15128' 2024-09-11T06:21:58.491 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=64, tag_tid=11, entry_tid=15128' 2024-09-11T06:21:58.491 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=64, tag_tid=11, entry_tid=15128' '!=' 'object_number=67, tag_tid=11, entry_tid=15059' 2024-09-11T06:21:58.491 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:21:58.491 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=64, tag_tid=11, entry_tid=15128' = 'object_number=67, tag_tid=11, entry_tid=15059' 2024-09-11T06:21:58.491 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=64, tag_tid=11, entry_tid=15128' 2024-09-11T06:21:58.491 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:58.491 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:58.491 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:58.492 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=12 2024-09-11T06:21:58.492 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=64, tag_tid=11, entry_tid=15128' 2024-09-11T06:21:58.492 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:21:58.493 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:58.493 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=11 2024-09-11T06:21:58.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:58.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:58.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:58.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:21:58.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=64, tag_tid=11, entry_tid=15128' 2024-09-11T06:21:58.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:21:58.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:21:58.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=15128 2024-09-11T06:21:58.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 12 = 11 -a 1 -le 15128 2024-09-11T06:21:58.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:21:58.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:21:58.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:21:58.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:21:58.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:58.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:21:58.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:21:58.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:21:58.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:21:58.700 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:21:58.700 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:21:58.700 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:21:58.700 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:58.700 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:58.700 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:21:58.700 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:21:58.700 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:21:58.700 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:21:58.700 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:21:58.701 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:21:58.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:21:58.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:21:58.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:21:58.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:21:58.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:21:58.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:21:58.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:21:58.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:21:58.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:21:58.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:21:58.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:21:58.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:58.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:21:58.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:58.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:21:58.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:58.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:21:58.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:21:58.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:21:58.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:21:58.706 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:21:58.706 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:21:58.706 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:21:58.706 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:21:58.706 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:21:58.706 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:21:58.706 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:21:58.706 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:21:58.706 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:21:58.706 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:21:58.706 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:21:58.706 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:21:58.730 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:58.731 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:58.731 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:21:58.731 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:21:58.855 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:21:58.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:21:58.857 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:58.857 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:58.857 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:58.857 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:21:58.857 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:58.857 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:58.857 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:58.857 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:21:58.858 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:58.858 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:58.858 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:58.859 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:58.859 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:58.860 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:58.911 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 16 2024-09-11T06:21:58.911 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:58.911 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:58.911 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:58.911 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:58.915 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:58.916 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:58.918 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:21:58.918 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:58.918 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:58.918 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:58.918 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:21:58.918 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:21:58.918 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:21:58.918 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:21:58.918 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:21:58.918 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:21:58.918 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:21:58.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:21:58.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:58.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:21:58.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:58.965 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 16 2024-09-11T06:21:58.965 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 16 2024-09-11T06:21:58.965 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:21:58.965 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:58.965 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=65, tag_tid=12, entry_tid=1], [object_number=64, tag_tid=12, entry_tid=0], [object_number=67, tag_tid=11, entry_tid=15127], [object_number=66, tag_tid=11, entry_tid=15126]]], state=connected] 2024-09-11T06:21:58.968 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:21:58.970 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:58.970 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=65, tag_tid=12, entry_tid=1' -a 'object_number=65, tag_tid=12, entry_tid=1' = 'object_number=65, tag_tid=12, entry_tid=1' 2024-09-11T06:21:58.971 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:21:58.971 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_snap_present cluster1 mirror test snap4 2024-09-11T06:21:58.971 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:21:58.971 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:21:58.971 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:21:58.971 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap4 2024-09-11T06:21:58.971 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:21:58.971 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T06:21:58.971 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:21:59.971 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 info mirror/test@snap4 2024-09-11T06:22:00.044 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:rbd image 'test': 2024-09-11T06:22:00.044 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: size 512 MiB in 128 objects 2024-09-11T06:22:00.044 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: order 22 (4 MiB objects) 2024-09-11T06:22:00.044 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: snapshot_count: 7 2024-09-11T06:22:00.044 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: id: 10a618cfe337 2024-09-11T06:22:00.044 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: block_name_prefix: rbd_data.10a618cfe337 2024-09-11T06:22:00.044 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: format: 2 2024-09-11T06:22:00.044 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: features: layering, exclusive-lock, journaling 2024-09-11T06:22:00.045 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: op_features: 2024-09-11T06:22:00.045 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: flags: 2024-09-11T06:22:00.045 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: create_timestamp: Wed Sep 11 06:16:46 2024 2024-09-11T06:22:00.045 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: access_timestamp: Wed Sep 11 06:20:30 2024 2024-09-11T06:22:00.045 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: modify_timestamp: Wed Sep 11 06:21:50 2024 2024-09-11T06:22:00.045 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: protected: False 2024-09-11T06:22:00.045 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: journal: 10a618cfe337 2024-09-11T06:22:00.045 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring state: enabled 2024-09-11T06:22:00.045 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring mode: journal 2024-09-11T06:22:00.045 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring global id: a0f7d515-874e-4c67-a26b-4958c7ab3083 2024-09-11T06:22:00.045 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring primary: false 2024-09-11T06:22:00.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:22:00.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n snap3-clean ']' 2024-09-11T06:22:00.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ compare_image_snaps mirror test snap3-clean 2024-09-11T06:22:00.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:22:00.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:22:00.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap3-clean 2024-09-11T06:22:00.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local ret=0 2024-09-11T06:22:00.050 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local rmt_export=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export 2024-09-11T06:22:00.050 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loc_export=/tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:22:00.050 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:22:00.050 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3-clean /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export 2024-09-11T06:22:00.623 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T06:22:00.637 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3-clean /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:22:02.077 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T06:22:02.096 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmp /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:22:02.373 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:22:02.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:22:02.513 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ compare_image_snaps mirror test snap4 2024-09-11T06:22:02.513 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:22:02.513 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:22:02.513 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap4 2024-09-11T06:22:02.513 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local ret=0 2024-09-11T06:22:02.513 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local rmt_export=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export 2024-09-11T06:22:02.513 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loc_export=/tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:22:02.513 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:22:02.514 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4 /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export 2024-09-11T06:22:03.177 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T06:22:03.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4 /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:22:04.726 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T06:22:04.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmp /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:22:05.031 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:22:05.170 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:22:05.170 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ clean_snap_name=snap4-clean 2024-09-11T06:22:05.170 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_snap cluster2 mirror test snap4-clean 2024-09-11T06:22:05.170 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:22:05.170 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:22:05.170 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:22:05.170 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap4-clean 2024-09-11T06:22:05.170 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap4-clean --debug-rbd=20 --debug-journaler=20 2024-09-11T06:22:06.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 10` 2024-09-11T06:22:06.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ stress_write_image cluster2 mirror test 2024-09-11T06:22:06.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:22:06.196 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:22:06.196 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:22:06.196 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T06:22:06.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local duration=5 2024-09-11T06:22:06.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set +e 2024-09-11T06:22:06.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-11T06:22:08.617 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:100 IOs, 43579392 bytes 2024-09-11T06:22:11.009 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:200 IOs, 103205888 bytes 2024-09-11T06:22:11.207 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ error_code=124 2024-09-11T06:22:11.207 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set -e 2024-09-11T06:22:11.207 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T06:22:11.207 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:22:11.207 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-11T06:22:11.207 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:22:11.208 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:22:11.208 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:22:11.208 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state_pattern=up+replaying 2024-09-11T06:22:11.208 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local description_pattern= 2024-09-11T06:22:11.208 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local service_pattern= 2024-09-11T06:22:11.208 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:22:11.208 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:22:12.208 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-11T06:22:12.208 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:22:12.208 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:22:12.209 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:22:12.209 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state_pattern=up+replaying 2024-09-11T06:22:12.209 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local description_pattern= 2024-09-11T06:22:12.209 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local service_pattern= 2024-09-11T06:22:12.209 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local status 2024-09-11T06:22:12.209 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ CEPH_ARGS= 2024-09-11T06:22:12.209 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-11T06:22:12.295 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ status='test: 2024-09-11T06:22:12.295 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: global_id: a0f7d515-874e-4c67-a26b-4958c7ab3083 2024-09-11T06:22:12.295 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: state: up+replaying 2024-09-11T06:22:12.295 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: description: replaying, {"bytes_per_second":3157358.53,"entries_behind_primary":2860,"entries_per_second":198.23,"non_primary_position":{"entry_tid":12269,"object_number":61,"tag_tid":11},"primary_position":{"entry_tid":1,"object_number":65,"tag_tid":12},"seconds_until_synced":14} 2024-09-11T06:22:12.296 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: service: mirror.0 on smithi112 2024-09-11T06:22:12.296 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: last_update: 2024-09-11 06:21:44 2024-09-11T06:22:12.296 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: peer_sites: 2024-09-11T06:22:12.296 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: name: cluster2 2024-09-11T06:22:12.296 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: state: up+stopped 2024-09-11T06:22:12.296 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: description: local image is primary 2024-09-11T06:22:12.296 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: last_update: 2024-09-11 06:22:09' 2024-09-11T06:22:12.296 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ grep '^ state: .*up+replaying' 2024-09-11T06:22:12.296 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: state: up+replaying 2024-09-11T06:22:12.297 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ grep '^ description: .*' 2024-09-11T06:22:12.297 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: description: replaying, {"bytes_per_second":3157358.53,"entries_behind_primary":2860,"entries_per_second":198.23,"non_primary_position":{"entry_tid":12269,"object_number":61,"tag_tid":11},"primary_position":{"entry_tid":1,"object_number":65,"tag_tid":12},"seconds_until_synced":14} 2024-09-11T06:22:12.297 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n '' ']' 2024-09-11T06:22:12.298 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ echo up+replaying 2024-09-11T06:22:12.298 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ grep '^up+' 2024-09-11T06:22:12.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:up+replaying 2024-09-11T06:22:12.300 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ grep 'service: *mirror..* on ' 2024-09-11T06:22:12.301 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: service: mirror.0 on smithi112 2024-09-11T06:22:12.301 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local last_update 2024-09-11T06:22:12.301 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-11T06:22:12.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_update='2024-09-11 06:21:44' 2024-09-11T06:22:12.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-11 06:21:44' 2024-09-11T06:22:12.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:22:12.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:22:12.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:22:12.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state_pattern=up+replaying 2024-09-11T06:22:12.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'prev_last_update=2024-09-11 06:21:44' 2024-09-11T06:22:12.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local status 2024-09-11T06:22:12.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ CEPH_ARGS= 2024-09-11T06:22:12.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-11T06:22:12.382 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ status='OKOKOK142554262mirror.0smithi11219.1.1-298-gb53b6d3etrueOK42594265mirror.1smithi11219.1.1-298-gb53b6d3efalseOKtesta0f7d515-874e-4c67-a26b-4958c7ab3083up+replayingreplaying, {"bytes_per_second":3157358.53,"entries_behind_primary":2860,"entries_per_second":198.23,"non_primary_position":{"entry_tid":12269,"object_number":61,"tag_tid":11},"primary_position":{"entry_tid":1,"object_number":65,"tag_tid":12},"seconds_until_synced":14}42554262mirror.0smithi1122024-09-11 06:21:44cluster20bfc6eda-f3c5-424a-8ff7-5c4d3d9fd488up+stoppedlocal image is primary2024-09-11 06:22:09' 2024-09-11T06:22:12.383 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local last_update state 2024-09-11T06:22:12.383 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-11T06:22:12.386 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_update='2024-09-11 06:21:44' 2024-09-11T06:22:12.387 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-11T06:22:12.389 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ state=up+replaying 2024-09-11T06:22:12.389 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ echo up+replaying 2024-09-11T06:22:12.389 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ grep up+replaying 2024-09-11T06:22:12.390 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:up+replaying 2024-09-11T06:22:12.391 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:22:12.391 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:22:12.391 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ snap_name=snap5 2024-09-11T06:22:12.391 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_snap cluster2 mirror test snap5 2024-09-11T06:22:12.391 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:22:12.391 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:22:12.391 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:22:12.391 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap5 2024-09-11T06:22:12.391 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap5 --debug-rbd=20 --debug-journaler=20 2024-09-11T06:22:27.667 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:22:27.664+0000 7fb45318f640 -1 reset not still connected to 0x561315ba5520 2024-09-11T06:22:43.283 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-11T06:22:43.284 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:22:43.284 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:22:43.284 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:22:43.284 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-11T06:22:43.284 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:22:43.284 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:22:43.284 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:22:43.284 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=started 2024-09-11T06:22:43.284 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:22:43.284 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:22:43.284 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:22:44.285 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-11T06:22:44.285 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:22:44.286 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:22:44.286 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:22:44.286 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=started 2024-09-11T06:22:44.286 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local status_result 2024-09-11T06:22:44.286 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=stopped 2024-09-11T06:22:44.287 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-11T06:22:44.287 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_instance=cluster1 2024-09-11T06:22:44.287 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:22:44.287 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:22:44.287 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance=cluster1 2024-09-11T06:22:44.287 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local loop_instance 2024-09-11T06:22:44.287 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:22:44.287 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sleep 0 2024-09-11T06:22:44.287 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -i state 2024-09-11T06:22:44.288 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:22:44.288 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ seq 0 1 2024-09-11T06:22:44.291 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:22:44.291 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-11T06:22:44.291 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1:0 2024-09-11T06:22:44.291 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:22:44.291 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:22:44.291 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:22:44.291 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:22:44.291 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=cluster 2024-09-11T06:22:44.291 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:22:44.291 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:22:44.291 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:22:44.291 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:22:44.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval cluster=cluster1 2024-09-11T06:22:44.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:22:44.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:22:44.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:22:44.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:22:44.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local local_cluster=cluster1:0 2024-09-11T06:22:44.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster=cluster1 2024-09-11T06:22:44.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance 2024-09-11T06:22:44.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:22:44.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local val=cluster1:0 2024-09-11T06:22:44.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T06:22:44.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance_var_name=instance 2024-09-11T06:22:44.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:22:44.293 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:22:44.293 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T06:22:44.293 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval local_cluster=cluster1 2024-09-11T06:22:44.293 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ local_cluster=cluster1 2024-09-11T06:22:44.294 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval instance=0 2024-09-11T06:22:44.294 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ instance=0 2024-09-11T06:22:44.294 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:22:44.312 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:22:44.312 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:22:44.312 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:22:44.312 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-11T06:22:44.427 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ return 0 2024-09-11T06:22:44.428 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T06:22:44.428 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ echo ' "state": "Replaying"' 2024-09-11T06:22:44.429 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ grep -i Replaying 2024-09-11T06:22:44.430 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: "state": "Replaying" 2024-09-11T06:22:44.432 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_state=started 2024-09-11T06:22:44.432 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test started = started 2024-09-11T06:22:44.432 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:22:44.432 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-11T06:22:44.432 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:22:44.432 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:22:44.432 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:22:44.432 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:22:44.432 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' journal = journal ']' 2024-09-11T06:22:44.432 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-09-11T06:22:44.432 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:22:44.433 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:22:44.433 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:22:44.433 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:22:44.433 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-11T06:22:44.433 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-11T06:22:44.433 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:22:44.433 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:22:44.433 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:22:44.634 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:22:44.634 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:22:44.634 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:22:44.634 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:22:44.634 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:22:44.634 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:22:44.634 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:22:44.634 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:22:44.635 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:22:44.635 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:22:44.635 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:22:44.635 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:22:44.635 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:22:44.635 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:22:44.635 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:22:44.635 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:22:44.635 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:22:44.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:22:44.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:22:44.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:22:44.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:22:44.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:22:44.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:22:44.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:22:44.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:22:44.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:22:44.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:22:44.640 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:22:44.640 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:22:44.640 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:22:44.640 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:22:44.640 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:22:44.640 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:22:44.640 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:22:44.640 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:22:44.640 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:22:44.640 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:22:44.640 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:22:44.640 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:22:44.640 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:22:44.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:22:44.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:22:44.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:22:44.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:22:44.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:22:44.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:22:44.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:22:44.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:22:44.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:22:44.664 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:22:44.664 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:22:44.664 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:22:44.664 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:22:44.798 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:22:44.798 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:22:44.798 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:22:44.798 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:22:44.798 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:22:44.799 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:22:44.799 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:22:44.799 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:22:44.799 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:22:44.799 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:22:44.799 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:22:44.800 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:22:44.800 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:22:44.802 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:22:44.802 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:22:44.802 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:22:44.848 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 17 2024-09-11T06:22:44.848 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 18 2024-09-11T06:22:44.848 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:22:44.848 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=15, entry_tid=1], [object_number=72, tag_tid=15, entry_tid=0], [object_number=71, tag_tid=14, entry_tid=5951], [object_number=70, tag_tid=14, entry_tid=5950]]], state=connected] 2024-09-11T06:22:44.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=73, tag_tid=15, entry_tid=1], [object_number=72, tag_tid=15, entry_tid=0], [object_number=71, tag_tid=14, entry_tid=5951], [object_number=70, tag_tid=14, entry_tid=5950]]], state=connected] 2024-09-11T06:22:44.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:22:44.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=73, tag_tid=15, entry_tid=1' 2024-09-11T06:22:44.854 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:22:44.854 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:22:44.854 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:22:44.854 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:22:44.854 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:22:44.854 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:22:44.854 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:22:44.854 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:22:44.854 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:22:44.855 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:22:44.855 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:22:44.855 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:22:44.855 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:22:44.855 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:22:44.855 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:22:44.901 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 17 2024-09-11T06:22:44.902 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 18 2024-09-11T06:22:44.902 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:22:44.902 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=73, tag_tid=15, entry_tid=1], [object_number=72, tag_tid=15, entry_tid=0], [object_number=71, tag_tid=14, entry_tid=5951], [object_number=70, tag_tid=14, entry_tid=5950]]], state=connected] 2024-09-11T06:22:44.902 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=73, tag_tid=15, entry_tid=1], [object_number=72, tag_tid=15, entry_tid=0], [object_number=71, tag_tid=14, entry_tid=5951], [object_number=70, tag_tid=14, entry_tid=5950]]], state=connected] 2024-09-11T06:22:44.905 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:22:44.907 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=73, tag_tid=15, entry_tid=1' 2024-09-11T06:22:44.907 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=73, tag_tid=15, entry_tid=1' -a 'object_number=73, tag_tid=15, entry_tid=1' = 'object_number=73, tag_tid=15, entry_tid=1' 2024-09-11T06:22:44.907 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:22:44.907 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_snap_present cluster1 mirror test snap5 2024-09-11T06:22:44.907 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:22:44.907 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:22:44.907 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:22:44.907 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap5 2024-09-11T06:22:44.907 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:22:44.907 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T06:22:44.908 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:22:45.908 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 info mirror/test@snap5 2024-09-11T06:22:45.975 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:rbd image 'test': 2024-09-11T06:22:45.975 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: size 512 MiB in 128 objects 2024-09-11T06:22:45.975 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: order 22 (4 MiB objects) 2024-09-11T06:22:45.975 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: snapshot_count: 9 2024-09-11T06:22:45.975 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: id: 10a618cfe337 2024-09-11T06:22:45.975 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: block_name_prefix: rbd_data.10a618cfe337 2024-09-11T06:22:45.975 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: format: 2 2024-09-11T06:22:45.975 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: features: layering, exclusive-lock, journaling 2024-09-11T06:22:45.976 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: op_features: 2024-09-11T06:22:45.976 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: flags: 2024-09-11T06:22:45.976 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: create_timestamp: Wed Sep 11 06:16:46 2024 2024-09-11T06:22:45.976 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: access_timestamp: Wed Sep 11 06:22:00 2024 2024-09-11T06:22:45.976 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: modify_timestamp: Wed Sep 11 06:21:50 2024 2024-09-11T06:22:45.976 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: protected: False 2024-09-11T06:22:45.976 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: journal: 10a618cfe337 2024-09-11T06:22:45.976 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring state: enabled 2024-09-11T06:22:45.976 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring mode: journal 2024-09-11T06:22:45.976 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring global id: a0f7d515-874e-4c67-a26b-4958c7ab3083 2024-09-11T06:22:45.976 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring primary: false 2024-09-11T06:22:45.980 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:22:45.980 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n snap4-clean ']' 2024-09-11T06:22:45.980 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ compare_image_snaps mirror test snap4-clean 2024-09-11T06:22:45.980 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:22:45.980 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:22:45.980 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap4-clean 2024-09-11T06:22:45.980 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local ret=0 2024-09-11T06:22:45.980 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local rmt_export=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export 2024-09-11T06:22:45.980 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loc_export=/tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:22:45.980 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:22:45.981 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4-clean /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export 2024-09-11T06:22:46.774 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T06:22:46.789 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4-clean /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:22:48.178 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T06:22:48.201 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmp /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:22:48.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:22:48.611 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:22:48.612 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ compare_image_snaps mirror test snap5 2024-09-11T06:22:48.612 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:22:48.612 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:22:48.612 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap5 2024-09-11T06:22:48.612 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local ret=0 2024-09-11T06:22:48.612 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local rmt_export=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export 2024-09-11T06:22:48.612 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loc_export=/tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:22:48.612 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:22:48.613 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5 /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export 2024-09-11T06:22:49.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T06:22:49.259 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5 /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:22:50.693 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T06:22:50.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmp /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:22:50.999 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:22:51.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:22:51.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ clean_snap_name=snap5-clean 2024-09-11T06:22:51.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_snap cluster2 mirror test snap5-clean 2024-09-11T06:22:51.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:22:51.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:22:51.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:22:51.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap5-clean 2024-09-11T06:22:51.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap5-clean --debug-rbd=20 --debug-journaler=20 2024-09-11T06:22:52.285 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 10` 2024-09-11T06:22:52.285 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ stress_write_image cluster2 mirror test 2024-09-11T06:22:52.285 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:22:52.285 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:22:52.285 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:22:52.286 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T06:22:52.289 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local duration=14 2024-09-11T06:22:52.289 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set +e 2024-09-11T06:22:52.289 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-11T06:22:55.215 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:100 IOs, 57930752 bytes 2024-09-11T06:22:56.121 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:200 IOs, 91635712 bytes 2024-09-11T06:22:59.974 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:300 IOs, 164875264 bytes 2024-09-11T06:23:00.054 INFO:tasks.ceph.cluster2.osd.1.smithi112.stderr:2024-09-11T06:23:00.055+0000 7f2cecaac640 -1 reset not still connected to 0x55da8e212820 2024-09-11T06:23:00.055 INFO:tasks.ceph.cluster2.osd.1.smithi112.stderr:2024-09-11T06:23:00.055+0000 7f2cecaac640 -1 reset not still connected to 0x55da8e41b930 2024-09-11T06:23:03.208 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:400 IOs, 244245504 bytes 2024-09-11T06:23:05.232 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:500 IOs, 307001344 bytes 2024-09-11T06:23:05.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:600 IOs, 307632128 bytes 2024-09-11T06:23:06.226 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:700 IOs, 333958144 bytes 2024-09-11T06:23:06.297 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ error_code=124 2024-09-11T06:23:06.297 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set -e 2024-09-11T06:23:06.297 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T06:23:06.297 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:23:06.297 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-11T06:23:06.298 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:23:06.298 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:23:06.298 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:23:06.298 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state_pattern=up+replaying 2024-09-11T06:23:06.298 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local description_pattern= 2024-09-11T06:23:06.298 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local service_pattern= 2024-09-11T06:23:06.298 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:23:06.298 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:23:07.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-11T06:23:07.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:23:07.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:23:07.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:23:07.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state_pattern=up+replaying 2024-09-11T06:23:07.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local description_pattern= 2024-09-11T06:23:07.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local service_pattern= 2024-09-11T06:23:07.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local status 2024-09-11T06:23:07.300 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ CEPH_ARGS= 2024-09-11T06:23:07.300 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-11T06:23:07.390 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ status='test: 2024-09-11T06:23:07.390 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: global_id: a0f7d515-874e-4c67-a26b-4958c7ab3083 2024-09-11T06:23:07.390 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: state: up+replaying 2024-09-11T06:23:07.390 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: description: replaying, {"bytes_per_second":2650376.53,"entries_behind_primary":0,"entries_per_second":165.17,"non_primary_position":{"entry_tid":5020,"object_number":68,"tag_tid":14},"primary_position":{"entry_tid":1,"object_number":65,"tag_tid":13}} 2024-09-11T06:23:07.390 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: service: mirror.0 on smithi112 2024-09-11T06:23:07.390 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: last_update: 2024-09-11 06:22:44 2024-09-11T06:23:07.390 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: peer_sites: 2024-09-11T06:23:07.391 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: name: cluster2 2024-09-11T06:23:07.391 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: state: up+stopped 2024-09-11T06:23:07.391 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: description: local image is primary 2024-09-11T06:23:07.391 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: last_update: 2024-09-11 06:23:05' 2024-09-11T06:23:07.391 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ grep '^ state: .*up+replaying' 2024-09-11T06:23:07.391 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: state: up+replaying 2024-09-11T06:23:07.391 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ grep '^ description: .*' 2024-09-11T06:23:07.392 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: description: replaying, {"bytes_per_second":2650376.53,"entries_behind_primary":0,"entries_per_second":165.17,"non_primary_position":{"entry_tid":5020,"object_number":68,"tag_tid":14},"primary_position":{"entry_tid":1,"object_number":65,"tag_tid":13}} 2024-09-11T06:23:07.393 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n '' ']' 2024-09-11T06:23:07.393 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ echo up+replaying 2024-09-11T06:23:07.393 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ grep '^up+' 2024-09-11T06:23:07.394 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:up+replaying 2024-09-11T06:23:07.394 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ grep 'service: *mirror..* on ' 2024-09-11T06:23:07.395 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: service: mirror.0 on smithi112 2024-09-11T06:23:07.395 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local last_update 2024-09-11T06:23:07.395 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-11T06:23:07.397 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_update='2024-09-11 06:22:44' 2024-09-11T06:23:07.398 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-11 06:22:44' 2024-09-11T06:23:07.398 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:23:07.398 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:23:07.398 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:23:07.398 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state_pattern=up+replaying 2024-09-11T06:23:07.398 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'prev_last_update=2024-09-11 06:22:44' 2024-09-11T06:23:07.398 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local status 2024-09-11T06:23:07.398 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ CEPH_ARGS= 2024-09-11T06:23:07.398 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-11T06:23:07.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ status='OKOKOK142554262mirror.0smithi11219.1.1-298-gb53b6d3etrueOK42594265mirror.1smithi11219.1.1-298-gb53b6d3efalseOKtesta0f7d515-874e-4c67-a26b-4958c7ab3083up+replayingreplaying, {"bytes_per_second":2650376.53,"entries_behind_primary":0,"entries_per_second":165.17,"non_primary_position":{"entry_tid":5020,"object_number":68,"tag_tid":14},"primary_position":{"entry_tid":1,"object_number":65,"tag_tid":13}}42554262mirror.0smithi1122024-09-11 06:22:44cluster20bfc6eda-f3c5-424a-8ff7-5c4d3d9fd488up+stoppedlocal image is primary2024-09-11 06:23:05' 2024-09-11T06:23:07.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local last_update state 2024-09-11T06:23:07.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-11T06:23:07.491 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_update='2024-09-11 06:22:44' 2024-09-11T06:23:07.491 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-11T06:23:07.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ state=up+replaying 2024-09-11T06:23:07.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ echo up+replaying 2024-09-11T06:23:07.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ grep up+replaying 2024-09-11T06:23:07.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:up+replaying 2024-09-11T06:23:07.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:23:07.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:23:07.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ snap_name=snap6 2024-09-11T06:23:07.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_snap cluster2 mirror test snap6 2024-09-11T06:23:07.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:23:07.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:23:07.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:23:07.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap6 2024-09-11T06:23:07.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap6 --debug-rbd=20 --debug-journaler=20 2024-09-11T06:23:40.473 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-11T06:23:40.473 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:23:40.473 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:23:40.473 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:23:40.473 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-11T06:23:40.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:23:40.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:23:40.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:23:40.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=started 2024-09-11T06:23:40.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:23:40.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:23:40.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:23:41.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-11T06:23:41.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:23:41.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:23:41.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:23:41.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=started 2024-09-11T06:23:41.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local status_result 2024-09-11T06:23:41.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=stopped 2024-09-11T06:23:41.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-11T06:23:41.476 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_instance=cluster1 2024-09-11T06:23:41.476 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:23:41.476 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:23:41.476 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance=cluster1 2024-09-11T06:23:41.476 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local loop_instance 2024-09-11T06:23:41.476 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:23:41.476 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sleep 0 2024-09-11T06:23:41.476 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -i state 2024-09-11T06:23:41.476 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:23:41.477 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ seq 0 1 2024-09-11T06:23:41.478 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:23:41.478 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-11T06:23:41.478 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1:0 2024-09-11T06:23:41.478 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:23:41.478 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:23:41.479 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:23:41.479 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:23:41.479 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=cluster 2024-09-11T06:23:41.479 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:23:41.479 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:23:41.479 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:23:41.479 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:23:41.479 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval cluster=cluster1 2024-09-11T06:23:41.479 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:23:41.479 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:23:41.479 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:23:41.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:23:41.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local local_cluster=cluster1:0 2024-09-11T06:23:41.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster=cluster1 2024-09-11T06:23:41.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance 2024-09-11T06:23:41.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:23:41.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local val=cluster1:0 2024-09-11T06:23:41.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T06:23:41.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance_var_name=instance 2024-09-11T06:23:41.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:23:41.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:23:41.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T06:23:41.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval local_cluster=cluster1 2024-09-11T06:23:41.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ local_cluster=cluster1 2024-09-11T06:23:41.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval instance=0 2024-09-11T06:23:41.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ instance=0 2024-09-11T06:23:41.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:23:41.510 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:23:41.510 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:23:41.510 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:23:41.510 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-11T06:23:41.633 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ return 0 2024-09-11T06:23:41.633 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T06:23:41.634 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ grep -i Replaying 2024-09-11T06:23:41.634 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ echo ' "state": "Replaying"' 2024-09-11T06:23:41.635 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: "state": "Replaying" 2024-09-11T06:23:41.636 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_state=started 2024-09-11T06:23:41.636 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test started = started 2024-09-11T06:23:41.636 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:23:41.636 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-11T06:23:41.637 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:23:41.637 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:23:41.637 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:23:41.637 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:23:41.637 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' journal = journal ']' 2024-09-11T06:23:41.637 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-09-11T06:23:41.637 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:23:41.637 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:23:41.637 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:23:41.637 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:23:41.637 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-11T06:23:41.637 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-11T06:23:41.637 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:23:41.638 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:23:41.638 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:23:41.837 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:23:41.838 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:23:41.838 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:23:41.838 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:23:41.838 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:23:41.838 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:23:41.838 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:23:41.838 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:23:41.838 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:23:41.838 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:23:41.838 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:23:41.838 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:23:41.838 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:23:41.838 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:23:41.839 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:23:41.839 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:23:41.839 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:23:41.841 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:23:41.841 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:23:41.841 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:23:41.841 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:23:41.841 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:23:41.842 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:23:41.842 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:23:41.842 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:23:41.842 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:23:41.842 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:23:41.842 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:23:41.842 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:23:41.842 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:23:41.842 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:23:41.842 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:23:41.842 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:23:41.844 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:23:41.844 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:23:41.844 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:23:41.844 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:23:41.844 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:23:41.844 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:23:41.844 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:23:41.844 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:23:41.845 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:23:41.845 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:23:41.845 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:23:41.845 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:23:41.845 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:23:41.845 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:23:41.845 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:23:41.845 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:23:41.868 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:23:41.868 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:23:41.868 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:23:41.868 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:23:41.996 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:23:41.997 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:23:41.997 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:41.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:41.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:41.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:23:41.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:41.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:41.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:41.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:23:41.999 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:23:41.999 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:23:41.999 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:23:42.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:42.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:23:42.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:42.046 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 20 2024-09-11T06:23:42.046 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:23:42.046 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:23:42.046 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:23:42.047 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=83, tag_tid=17, entry_tid=8135], [object_number=82, tag_tid=17, entry_tid=8134], [object_number=81, tag_tid=17, entry_tid=8133], [object_number=80, tag_tid=17, entry_tid=8132]]], state=connected] 2024-09-11T06:23:42.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:42.051 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:23:42.052 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:23:42.052 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:42.052 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:42.052 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:42.052 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:23:42.053 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:42.053 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:42.053 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:42.053 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:23:42.053 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:23:42.053 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:23:42.053 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:23:42.053 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:42.054 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:23:42.054 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:42.101 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 20 2024-09-11T06:23:42.101 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:23:42.101 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:23:42.102 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:23:42.102 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=83, tag_tid=17, entry_tid=8135], [object_number=82, tag_tid=17, entry_tid=8134], [object_number=81, tag_tid=17, entry_tid=8133], [object_number=80, tag_tid=17, entry_tid=8132]]], state=connected] 2024-09-11T06:23:42.104 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:42.106 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=83, tag_tid=17, entry_tid=8135' 2024-09-11T06:23:42.107 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=83, tag_tid=17, entry_tid=8135' 2024-09-11T06:23:42.107 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=83, tag_tid=17, entry_tid=8135' '!=' '' 2024-09-11T06:23:42.107 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:23:42.107 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=83, tag_tid=17, entry_tid=8135' = '' 2024-09-11T06:23:42.107 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=83, tag_tid=17, entry_tid=8135' 2024-09-11T06:23:42.107 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:23:42.107 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:23:42.107 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:42.108 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:23:42.108 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=83, tag_tid=17, entry_tid=8135' 2024-09-11T06:23:42.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:23:42.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:42.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:23:42.110 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:23:42.110 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:23:42.110 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:42.112 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:23:42.112 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=83, tag_tid=17, entry_tid=8135' 2024-09-11T06:23:42.112 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:23:42.112 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:42.113 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=8135 2024-09-11T06:23:42.113 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 8135 2024-09-11T06:23:42.113 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:23:42.113 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:23:42.113 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:23:42.315 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:23:42.315 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:23:42.315 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:23:42.315 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:23:42.315 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:23:42.315 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:23:42.315 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:23:42.315 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:23:42.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:23:42.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:23:42.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:23:42.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:23:42.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:23:42.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:23:42.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:23:42.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:23:42.317 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:23:42.320 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:23:42.320 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:23:42.320 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:23:42.321 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:23:42.321 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:23:42.321 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:23:42.321 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:23:42.321 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:23:42.321 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:23:42.321 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:23:42.321 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:23:42.321 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:23:42.321 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:23:42.321 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:23:42.321 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:23:42.321 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:23:42.321 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:23:42.322 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:23:42.322 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:23:42.322 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:23:42.322 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:23:42.322 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:23:42.322 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:23:42.322 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:23:42.322 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:23:42.322 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:23:42.322 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:23:42.322 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:23:42.322 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:23:42.322 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:23:42.322 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:23:42.323 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:23:42.347 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:23:42.347 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:23:42.347 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:23:42.348 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:23:42.478 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:23:42.479 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:23:42.479 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:42.479 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:42.479 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:42.479 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:23:42.479 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:42.479 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:42.479 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:42.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:23:42.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:23:42.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:23:42.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:23:42.483 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:42.483 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:23:42.483 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:42.535 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 20 2024-09-11T06:23:42.535 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:23:42.535 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:23:42.535 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:23:42.536 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=82, tag_tid=17, entry_tid=8226], [object_number=81, tag_tid=17, entry_tid=8225], [object_number=80, tag_tid=17, entry_tid=8224], [object_number=83, tag_tid=17, entry_tid=8223]]], state=connected] 2024-09-11T06:23:42.538 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:42.540 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:23:42.541 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:23:42.541 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:42.541 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:42.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:42.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:23:42.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:42.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:42.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:42.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:23:42.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:23:42.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:23:42.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:23:42.543 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:42.543 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:23:42.543 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:42.590 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 20 2024-09-11T06:23:42.590 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:23:42.590 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:23:42.590 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:23:42.590 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=82, tag_tid=17, entry_tid=8226], [object_number=81, tag_tid=17, entry_tid=8225], [object_number=80, tag_tid=17, entry_tid=8224], [object_number=83, tag_tid=17, entry_tid=8223]]], state=connected] 2024-09-11T06:23:42.593 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:42.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=82, tag_tid=17, entry_tid=8226' 2024-09-11T06:23:42.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=82, tag_tid=17, entry_tid=8226' 2024-09-11T06:23:42.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=82, tag_tid=17, entry_tid=8226' '!=' 'object_number=83, tag_tid=17, entry_tid=8135' 2024-09-11T06:23:42.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:23:42.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=82, tag_tid=17, entry_tid=8226' = 'object_number=83, tag_tid=17, entry_tid=8135' 2024-09-11T06:23:42.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=82, tag_tid=17, entry_tid=8226' 2024-09-11T06:23:42.596 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:23:42.596 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:23:42.596 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:42.596 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:23:42.597 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=82, tag_tid=17, entry_tid=8226' 2024-09-11T06:23:42.597 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:23:42.598 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:42.598 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:23:42.599 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:23:42.599 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:23:42.599 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:42.600 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:23:42.601 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=82, tag_tid=17, entry_tid=8226' 2024-09-11T06:23:42.601 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:23:42.601 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:42.602 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=8226 2024-09-11T06:23:42.602 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 8226 2024-09-11T06:23:42.602 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:23:42.602 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:23:42.602 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:23:42.804 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:23:42.804 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:23:42.804 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:23:42.804 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:23:42.804 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:23:42.804 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:23:42.804 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:23:42.804 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:23:42.804 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:23:42.804 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:23:42.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:23:42.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:23:42.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:23:42.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:23:42.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:23:42.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:23:42.806 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:23:42.808 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:23:42.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:23:42.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:23:42.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:23:42.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:23:42.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:23:42.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:23:42.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:23:42.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:23:42.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:23:42.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:23:42.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:23:42.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:23:42.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:23:42.810 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:23:42.810 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:23:42.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:23:42.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:23:42.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:23:42.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:23:42.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:23:42.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:23:42.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:23:42.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:23:42.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:23:42.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:23:42.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:23:42.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:23:42.812 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:23:42.812 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:23:42.812 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:23:42.812 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:23:42.835 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:23:42.835 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:23:42.835 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:23:42.835 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:23:42.969 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:23:42.970 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:23:42.970 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:42.970 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:42.970 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:42.970 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:23:42.971 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:42.971 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:42.971 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:42.971 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:23:42.971 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:23:42.971 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:23:42.972 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:23:42.973 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:42.973 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:23:42.973 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:43.020 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 20 2024-09-11T06:23:43.020 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:23:43.020 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:23:43.020 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:23:43.020 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=83, tag_tid=17, entry_tid=8307], [object_number=82, tag_tid=17, entry_tid=8306], [object_number=81, tag_tid=17, entry_tid=8305], [object_number=80, tag_tid=17, entry_tid=8304]]], state=connected] 2024-09-11T06:23:43.023 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:43.026 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:23:43.026 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:23:43.026 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:43.026 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:43.026 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:43.026 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:23:43.026 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:43.026 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:43.026 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:43.026 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:23:43.026 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:23:43.026 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:23:43.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:23:43.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:43.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:23:43.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:43.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 20 2024-09-11T06:23:43.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:23:43.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:23:43.076 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:23:43.076 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=83, tag_tid=17, entry_tid=8307], [object_number=82, tag_tid=17, entry_tid=8306], [object_number=81, tag_tid=17, entry_tid=8305], [object_number=80, tag_tid=17, entry_tid=8304]]], state=connected] 2024-09-11T06:23:43.079 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:43.081 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=83, tag_tid=17, entry_tid=8307' 2024-09-11T06:23:43.081 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=83, tag_tid=17, entry_tid=8307' 2024-09-11T06:23:43.081 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=83, tag_tid=17, entry_tid=8307' '!=' 'object_number=82, tag_tid=17, entry_tid=8226' 2024-09-11T06:23:43.081 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:23:43.081 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=83, tag_tid=17, entry_tid=8307' = 'object_number=82, tag_tid=17, entry_tid=8226' 2024-09-11T06:23:43.081 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=83, tag_tid=17, entry_tid=8307' 2024-09-11T06:23:43.081 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:23:43.081 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:23:43.081 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:43.082 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:23:43.082 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=83, tag_tid=17, entry_tid=8307' 2024-09-11T06:23:43.082 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:23:43.083 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:43.083 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:23:43.084 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:23:43.084 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:23:43.084 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:43.085 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:23:43.086 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=83, tag_tid=17, entry_tid=8307' 2024-09-11T06:23:43.086 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:23:43.086 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:43.087 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=8307 2024-09-11T06:23:43.087 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 8307 2024-09-11T06:23:43.087 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:23:43.087 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:23:43.087 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:23:43.289 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:23:43.289 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:23:43.289 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:23:43.289 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:23:43.289 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:23:43.289 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:23:43.289 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:23:43.289 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:23:43.290 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:23:43.290 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:23:43.290 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:23:43.290 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:23:43.290 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:23:43.290 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:23:43.290 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:23:43.290 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:23:43.291 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:23:43.294 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:23:43.294 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:23:43.294 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:23:43.294 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:23:43.294 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:23:43.294 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:23:43.294 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:23:43.294 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:23:43.294 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:23:43.295 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:23:43.295 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:23:43.295 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:23:43.295 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:23:43.295 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:23:43.295 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:23:43.295 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:23:43.295 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:23:43.295 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:23:43.295 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:23:43.295 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:23:43.295 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:23:43.295 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:23:43.296 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:23:43.296 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:23:43.296 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:23:43.296 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:23:43.296 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:23:43.296 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:23:43.296 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:23:43.296 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:23:43.296 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:23:43.297 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:23:43.321 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:23:43.321 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:23:43.321 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:23:43.322 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:23:43.452 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:23:43.454 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:23:43.454 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:43.454 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:43.454 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:43.454 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:23:43.454 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:43.454 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:43.454 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:43.454 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:23:43.454 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:23:43.455 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:23:43.455 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:23:43.456 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:43.457 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:43.457 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:23:43.502 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 20 2024-09-11T06:23:43.503 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:23:43.503 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:23:43.503 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:23:43.503 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=82, tag_tid=17, entry_tid=8390], [object_number=81, tag_tid=17, entry_tid=8389], [object_number=80, tag_tid=17, entry_tid=8388], [object_number=83, tag_tid=17, entry_tid=8387]]], state=connected] 2024-09-11T06:23:43.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:43.507 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:23:43.508 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:23:43.509 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:43.509 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:43.509 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:43.509 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:23:43.509 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:43.509 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:43.509 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:43.509 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:23:43.509 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:23:43.509 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:23:43.509 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:23:43.510 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:43.510 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:23:43.510 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:43.556 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 20 2024-09-11T06:23:43.556 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:23:43.557 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:23:43.557 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:23:43.557 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=82, tag_tid=17, entry_tid=8390], [object_number=81, tag_tid=17, entry_tid=8389], [object_number=80, tag_tid=17, entry_tid=8388], [object_number=83, tag_tid=17, entry_tid=8387]]], state=connected] 2024-09-11T06:23:43.559 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:43.561 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=82, tag_tid=17, entry_tid=8390' 2024-09-11T06:23:43.561 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=82, tag_tid=17, entry_tid=8390' 2024-09-11T06:23:43.562 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=82, tag_tid=17, entry_tid=8390' '!=' 'object_number=83, tag_tid=17, entry_tid=8307' 2024-09-11T06:23:43.562 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:23:43.562 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=82, tag_tid=17, entry_tid=8390' = 'object_number=83, tag_tid=17, entry_tid=8307' 2024-09-11T06:23:43.562 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=82, tag_tid=17, entry_tid=8390' 2024-09-11T06:23:43.562 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:23:43.562 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:23:43.562 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:43.563 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:23:43.563 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=82, tag_tid=17, entry_tid=8390' 2024-09-11T06:23:43.563 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:23:43.563 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:43.564 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:23:43.565 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:23:43.565 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:23:43.565 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:43.566 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:23:43.566 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=82, tag_tid=17, entry_tid=8390' 2024-09-11T06:23:43.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:23:43.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:43.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=8390 2024-09-11T06:23:43.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 8390 2024-09-11T06:23:43.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:23:43.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:23:43.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:23:43.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:23:43.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:23:43.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:23:43.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:23:43.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:23:43.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:23:43.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:23:43.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:23:43.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:23:43.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:23:43.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:23:43.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:23:43.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:23:43.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:23:43.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:23:43.772 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:23:43.772 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:23:43.775 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:23:43.775 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:23:43.775 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:23:43.775 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:23:43.775 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:23:43.775 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:23:43.775 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:23:43.775 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:23:43.775 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:23:43.776 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:23:43.776 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:23:43.776 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:23:43.776 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:23:43.776 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:23:43.776 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:23:43.776 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:23:43.776 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:23:43.776 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:23:43.776 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:23:43.776 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:23:43.777 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:23:43.777 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:23:43.777 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:23:43.777 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:23:43.777 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:23:43.777 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:23:43.777 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:23:43.777 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:23:43.777 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:23:43.777 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:23:43.777 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:23:43.778 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:23:43.800 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:23:43.801 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:23:43.801 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:23:43.801 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:23:43.934 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:23:43.935 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:23:43.936 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:43.936 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:43.936 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:43.936 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:23:43.936 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:43.936 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:43.936 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:43.936 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:23:43.936 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:23:43.938 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:23:43.938 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:23:43.939 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:43.939 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:23:43.939 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:43.995 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 20 2024-09-11T06:23:43.995 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:23:43.995 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:23:43.995 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:23:43.996 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=80, tag_tid=17, entry_tid=8472], [object_number=83, tag_tid=17, entry_tid=8471], [object_number=82, tag_tid=17, entry_tid=8470], [object_number=81, tag_tid=17, entry_tid=8469]]], state=connected] 2024-09-11T06:23:43.999 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:44.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:23:44.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:23:44.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:44.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:44.002 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:44.002 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:23:44.002 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:44.002 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:44.002 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:44.002 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:23:44.002 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:23:44.002 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:23:44.002 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:23:44.003 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:44.003 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:23:44.003 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:44.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 20 2024-09-11T06:23:44.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:23:44.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:23:44.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:23:44.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=80, tag_tid=17, entry_tid=8472], [object_number=83, tag_tid=17, entry_tid=8471], [object_number=82, tag_tid=17, entry_tid=8470], [object_number=81, tag_tid=17, entry_tid=8469]]], state=connected] 2024-09-11T06:23:44.053 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:44.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=80, tag_tid=17, entry_tid=8472' 2024-09-11T06:23:44.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=80, tag_tid=17, entry_tid=8472' 2024-09-11T06:23:44.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=80, tag_tid=17, entry_tid=8472' '!=' 'object_number=82, tag_tid=17, entry_tid=8390' 2024-09-11T06:23:44.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:23:44.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=80, tag_tid=17, entry_tid=8472' = 'object_number=82, tag_tid=17, entry_tid=8390' 2024-09-11T06:23:44.056 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=80, tag_tid=17, entry_tid=8472' 2024-09-11T06:23:44.056 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:23:44.056 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:23:44.056 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:44.056 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:23:44.057 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=80, tag_tid=17, entry_tid=8472' 2024-09-11T06:23:44.057 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:23:44.057 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:44.059 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:23:44.059 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:23:44.059 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:23:44.059 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:44.060 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:23:44.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=80, tag_tid=17, entry_tid=8472' 2024-09-11T06:23:44.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:23:44.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:44.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=8472 2024-09-11T06:23:44.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 8472 2024-09-11T06:23:44.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:23:44.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:23:44.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:23:44.264 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:23:44.264 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:23:44.264 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:23:44.264 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:23:44.264 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:23:44.264 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:23:44.264 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:23:44.265 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:23:44.265 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:23:44.265 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:23:44.265 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:23:44.265 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:23:44.265 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:23:44.265 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:23:44.265 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:23:44.265 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:23:44.266 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:23:44.269 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:23:44.269 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:23:44.269 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:23:44.269 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:23:44.270 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:23:44.270 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:23:44.270 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:23:44.270 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:23:44.270 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:23:44.270 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:23:44.270 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:23:44.270 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:23:44.270 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:23:44.270 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:23:44.270 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:23:44.270 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:23:44.270 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:23:44.270 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:23:44.271 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:23:44.271 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:23:44.271 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:23:44.271 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:23:44.271 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:23:44.271 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:23:44.271 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:23:44.271 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:23:44.271 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:23:44.271 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:23:44.271 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:23:44.271 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:23:44.271 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:23:44.272 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:23:44.296 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:23:44.296 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:23:44.296 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:23:44.296 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:23:44.432 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:23:44.433 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:23:44.434 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:44.434 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:44.434 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:44.434 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:23:44.434 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:44.434 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:44.434 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:44.434 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:23:44.434 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:23:44.435 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:23:44.435 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:23:44.436 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:44.436 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:44.436 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:23:44.491 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 20 2024-09-11T06:23:44.491 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:23:44.492 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:23:44.492 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:23:44.492 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=82, tag_tid=17, entry_tid=8554], [object_number=81, tag_tid=17, entry_tid=8553], [object_number=80, tag_tid=17, entry_tid=8552], [object_number=83, tag_tid=17, entry_tid=8551]]], state=connected] 2024-09-11T06:23:44.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:44.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:23:44.498 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:23:44.498 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:44.498 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:44.498 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:44.498 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:23:44.498 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:44.498 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:44.498 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:44.498 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:23:44.499 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:23:44.499 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:23:44.499 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:23:44.499 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:44.499 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:23:44.499 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:44.547 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 20 2024-09-11T06:23:44.547 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:23:44.548 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:23:44.548 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:23:44.548 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=82, tag_tid=17, entry_tid=8554], [object_number=81, tag_tid=17, entry_tid=8553], [object_number=80, tag_tid=17, entry_tid=8552], [object_number=83, tag_tid=17, entry_tid=8551]]], state=connected] 2024-09-11T06:23:44.551 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:44.553 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=82, tag_tid=17, entry_tid=8554' 2024-09-11T06:23:44.553 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=82, tag_tid=17, entry_tid=8554' 2024-09-11T06:23:44.553 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=82, tag_tid=17, entry_tid=8554' '!=' 'object_number=80, tag_tid=17, entry_tid=8472' 2024-09-11T06:23:44.553 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:23:44.553 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=82, tag_tid=17, entry_tid=8554' = 'object_number=80, tag_tid=17, entry_tid=8472' 2024-09-11T06:23:44.553 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=82, tag_tid=17, entry_tid=8554' 2024-09-11T06:23:44.553 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:23:44.553 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:23:44.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:44.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:23:44.555 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=82, tag_tid=17, entry_tid=8554' 2024-09-11T06:23:44.555 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:23:44.555 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:44.556 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:23:44.556 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:23:44.556 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:23:44.557 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:44.557 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:23:44.558 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=82, tag_tid=17, entry_tid=8554' 2024-09-11T06:23:44.558 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:23:44.558 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:44.559 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=8554 2024-09-11T06:23:44.559 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 8554 2024-09-11T06:23:44.559 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:23:44.560 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:23:44.560 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:23:44.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:23:44.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:23:44.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:23:44.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:23:44.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:23:44.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:23:44.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:23:44.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:23:44.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:23:44.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:23:44.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:23:44.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:23:44.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:23:44.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:23:44.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:23:44.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:23:44.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:23:44.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:23:44.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:23:44.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:23:44.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:23:44.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:23:44.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:23:44.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:23:44.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:23:44.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:23:44.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:23:44.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:23:44.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:23:44.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:23:44.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:23:44.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:23:44.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:23:44.768 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:23:44.768 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:23:44.768 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:23:44.768 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:23:44.768 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:23:44.769 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:23:44.769 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:23:44.769 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:23:44.769 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:23:44.769 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:23:44.769 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:23:44.769 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:23:44.769 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:23:44.769 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:23:44.769 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:23:44.769 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:23:44.793 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:23:44.793 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:23:44.794 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:23:44.794 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:23:44.926 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:23:44.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:23:44.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:44.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:44.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:44.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:23:44.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:44.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:44.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:44.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:23:44.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:23:44.928 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:23:44.928 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:23:44.929 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:44.929 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:23:44.929 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:45.186 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 20 2024-09-11T06:23:45.186 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:23:45.186 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:23:45.186 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:23:45.186 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=82, tag_tid=17, entry_tid=8654], [object_number=81, tag_tid=17, entry_tid=8653], [object_number=80, tag_tid=17, entry_tid=8652], [object_number=83, tag_tid=17, entry_tid=8651]]], state=connected] 2024-09-11T06:23:45.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:45.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:23:45.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:23:45.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:45.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:45.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:45.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:23:45.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:45.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:45.196 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:45.196 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:23:45.196 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:23:45.196 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:23:45.196 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:23:45.197 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:45.197 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:23:45.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:45.254 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 20 2024-09-11T06:23:45.254 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:23:45.254 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:23:45.254 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:23:45.254 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=82, tag_tid=17, entry_tid=8654], [object_number=81, tag_tid=17, entry_tid=8653], [object_number=80, tag_tid=17, entry_tid=8652], [object_number=83, tag_tid=17, entry_tid=8651]]], state=connected] 2024-09-11T06:23:45.257 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:45.260 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=82, tag_tid=17, entry_tid=8654' 2024-09-11T06:23:45.260 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=82, tag_tid=17, entry_tid=8654' 2024-09-11T06:23:45.260 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=82, tag_tid=17, entry_tid=8654' '!=' 'object_number=82, tag_tid=17, entry_tid=8554' 2024-09-11T06:23:45.260 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:23:45.260 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=82, tag_tid=17, entry_tid=8654' = 'object_number=82, tag_tid=17, entry_tid=8554' 2024-09-11T06:23:45.260 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=82, tag_tid=17, entry_tid=8654' 2024-09-11T06:23:45.260 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:23:45.260 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:23:45.260 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:45.261 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:23:45.262 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=82, tag_tid=17, entry_tid=8654' 2024-09-11T06:23:45.262 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:23:45.262 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:45.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:23:45.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:23:45.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:23:45.264 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:45.264 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:23:45.265 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=82, tag_tid=17, entry_tid=8654' 2024-09-11T06:23:45.265 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:23:45.266 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:45.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=8654 2024-09-11T06:23:45.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 8654 2024-09-11T06:23:45.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:23:45.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:23:45.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:23:45.469 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:23:45.469 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:23:45.469 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:23:45.469 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:23:45.469 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:23:45.469 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:23:45.469 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:23:45.469 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:23:45.469 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:23:45.469 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:23:45.469 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:23:45.469 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:23:45.470 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:23:45.470 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:23:45.470 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:23:45.470 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:23:45.470 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:23:45.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:23:45.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:23:45.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:23:45.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:23:45.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:23:45.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:23:45.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:23:45.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:23:45.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:23:45.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:23:45.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:23:45.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:23:45.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:23:45.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:23:45.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:23:45.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:23:45.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:23:45.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:23:45.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:23:45.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:23:45.476 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:23:45.476 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:23:45.476 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:23:45.476 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:23:45.476 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:23:45.476 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:23:45.476 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:23:45.476 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:23:45.476 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:23:45.476 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:23:45.476 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:23:45.476 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:23:45.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:23:45.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:23:45.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:23:45.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:23:45.632 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:23:45.633 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:23:45.633 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:45.633 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:45.633 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:45.633 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:23:45.634 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:45.634 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:45.634 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:45.634 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:23:45.634 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:23:45.634 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:23:45.635 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:23:45.637 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:45.637 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:45.637 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:23:45.685 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 20 2024-09-11T06:23:45.685 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:23:45.685 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:23:45.685 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:23:45.685 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=80, tag_tid=17, entry_tid=8788], [object_number=83, tag_tid=17, entry_tid=8787], [object_number=82, tag_tid=17, entry_tid=8786], [object_number=81, tag_tid=17, entry_tid=8785]]], state=connected] 2024-09-11T06:23:45.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:45.690 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:23:45.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:23:45.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:45.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:45.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:45.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:23:45.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:45.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:45.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:45.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:23:45.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:23:45.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:23:45.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:23:45.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:45.693 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:23:45.693 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:45.949 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 20 2024-09-11T06:23:45.949 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:23:45.949 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:23:45.949 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:23:45.949 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=80, tag_tid=17, entry_tid=8788], [object_number=83, tag_tid=17, entry_tid=8787], [object_number=82, tag_tid=17, entry_tid=8786], [object_number=81, tag_tid=17, entry_tid=8785]]], state=connected] 2024-09-11T06:23:45.953 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:45.955 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=80, tag_tid=17, entry_tid=8788' 2024-09-11T06:23:45.955 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=80, tag_tid=17, entry_tid=8788' 2024-09-11T06:23:45.955 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=80, tag_tid=17, entry_tid=8788' '!=' 'object_number=82, tag_tid=17, entry_tid=8654' 2024-09-11T06:23:45.955 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:23:45.955 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=80, tag_tid=17, entry_tid=8788' = 'object_number=82, tag_tid=17, entry_tid=8654' 2024-09-11T06:23:45.955 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=80, tag_tid=17, entry_tid=8788' 2024-09-11T06:23:45.956 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:23:45.956 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:23:45.956 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:45.956 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:23:45.957 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=80, tag_tid=17, entry_tid=8788' 2024-09-11T06:23:45.957 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:23:45.957 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:45.958 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:23:45.959 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:23:45.959 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:23:45.959 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:45.960 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:23:45.961 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=80, tag_tid=17, entry_tid=8788' 2024-09-11T06:23:45.961 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:23:45.961 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:45.962 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=8788 2024-09-11T06:23:45.963 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 8788 2024-09-11T06:23:45.963 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:23:45.963 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:23:45.963 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:23:46.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:23:46.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:23:46.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:23:46.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:23:46.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:23:46.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:23:46.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:23:46.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:23:46.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:23:46.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:23:46.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:23:46.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:23:46.166 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:23:46.166 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:23:46.166 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:23:46.166 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:23:46.166 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:23:46.170 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:23:46.170 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:23:46.170 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:23:46.170 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:23:46.170 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:23:46.170 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:23:46.170 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:23:46.170 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:23:46.170 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:23:46.170 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:23:46.170 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:23:46.171 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:23:46.171 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:23:46.171 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:23:46.171 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:23:46.171 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:23:46.171 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:23:46.171 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:23:46.171 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:23:46.171 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:23:46.171 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:23:46.171 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:23:46.171 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:23:46.171 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:23:46.171 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:23:46.172 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:23:46.172 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:23:46.172 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:23:46.172 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:23:46.172 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:23:46.172 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:23:46.172 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:23:46.197 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:23:46.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:23:46.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:23:46.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:23:46.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:23:46.337 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:23:46.337 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:46.337 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:46.337 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:46.338 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:23:46.338 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:46.338 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:46.338 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:46.338 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:23:46.338 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:23:46.338 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:23:46.338 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:23:46.340 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:46.341 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:23:46.341 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:46.393 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 20 2024-09-11T06:23:46.393 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:23:46.393 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:23:46.393 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:23:46.393 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=80, tag_tid=17, entry_tid=8912], [object_number=83, tag_tid=17, entry_tid=8911], [object_number=82, tag_tid=17, entry_tid=8910], [object_number=81, tag_tid=17, entry_tid=8909]]], state=connected] 2024-09-11T06:23:46.397 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:46.399 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:23:46.399 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:23:46.399 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:46.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:46.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:46.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:23:46.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:46.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:46.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:46.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:23:46.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:23:46.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:23:46.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:23:46.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:46.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:23:46.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:46.659 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 20 2024-09-11T06:23:46.659 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:23:46.659 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:23:46.659 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:23:46.659 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=80, tag_tid=17, entry_tid=8912], [object_number=83, tag_tid=17, entry_tid=8911], [object_number=82, tag_tid=17, entry_tid=8910], [object_number=81, tag_tid=17, entry_tid=8909]]], state=connected] 2024-09-11T06:23:46.665 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:46.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=80, tag_tid=17, entry_tid=8912' 2024-09-11T06:23:46.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=80, tag_tid=17, entry_tid=8912' 2024-09-11T06:23:46.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=80, tag_tid=17, entry_tid=8912' '!=' 'object_number=80, tag_tid=17, entry_tid=8788' 2024-09-11T06:23:46.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:23:46.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=80, tag_tid=17, entry_tid=8912' = 'object_number=80, tag_tid=17, entry_tid=8788' 2024-09-11T06:23:46.668 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=80, tag_tid=17, entry_tid=8912' 2024-09-11T06:23:46.668 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:23:46.668 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:23:46.668 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:46.670 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:23:46.671 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=80, tag_tid=17, entry_tid=8912' 2024-09-11T06:23:46.671 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:23:46.671 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:46.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:23:46.674 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:23:46.674 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:23:46.674 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:46.675 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:23:46.676 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=80, tag_tid=17, entry_tid=8912' 2024-09-11T06:23:46.677 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:23:46.677 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:46.678 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=8912 2024-09-11T06:23:46.678 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 8912 2024-09-11T06:23:46.678 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:23:46.678 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:23:46.678 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:23:46.880 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:23:46.881 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:23:46.881 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:23:46.881 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:23:46.881 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:23:46.881 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:23:46.881 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:23:46.881 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:23:46.881 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:23:46.881 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:23:46.881 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:23:46.881 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:23:46.881 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:23:46.881 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:23:46.881 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:23:46.882 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:23:46.882 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:23:46.885 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:23:46.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:23:46.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:23:46.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:23:46.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:23:46.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:23:46.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:23:46.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:23:46.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:23:46.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:23:46.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:23:46.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:23:46.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:23:46.887 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:23:46.887 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:23:46.887 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:23:46.887 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:23:46.887 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:23:46.887 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:23:46.887 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:23:46.887 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:23:46.887 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:23:46.887 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:23:46.887 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:23:46.887 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:23:46.887 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:23:46.887 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:23:46.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:23:46.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:23:46.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:23:46.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:23:46.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:23:46.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:23:46.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:23:46.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:23:46.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:23:47.047 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:23:47.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:23:47.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:47.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:47.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:47.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:23:47.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:47.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:47.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:47.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:23:47.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:23:47.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:23:47.050 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:23:47.052 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:47.052 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:23:47.052 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:47.098 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 20 2024-09-11T06:23:47.099 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:23:47.099 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:23:47.099 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:23:47.099 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=82, tag_tid=17, entry_tid=9042], [object_number=81, tag_tid=17, entry_tid=9041], [object_number=80, tag_tid=17, entry_tid=9040], [object_number=83, tag_tid=17, entry_tid=9039]]], state=connected] 2024-09-11T06:23:47.102 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:47.103 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:23:47.104 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:23:47.104 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:47.104 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:47.104 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:47.105 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:23:47.105 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:47.105 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:47.105 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:47.105 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:23:47.105 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:23:47.105 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:23:47.105 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:23:47.106 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:47.106 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:23:47.106 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:47.151 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 20 2024-09-11T06:23:47.152 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:23:47.152 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:23:47.152 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:23:47.152 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=82, tag_tid=17, entry_tid=9042], [object_number=81, tag_tid=17, entry_tid=9041], [object_number=80, tag_tid=17, entry_tid=9040], [object_number=83, tag_tid=17, entry_tid=9039]]], state=connected] 2024-09-11T06:23:47.155 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:47.157 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=82, tag_tid=17, entry_tid=9042' 2024-09-11T06:23:47.157 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=82, tag_tid=17, entry_tid=9042' 2024-09-11T06:23:47.157 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=82, tag_tid=17, entry_tid=9042' '!=' 'object_number=80, tag_tid=17, entry_tid=8912' 2024-09-11T06:23:47.157 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:23:47.157 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=82, tag_tid=17, entry_tid=9042' = 'object_number=80, tag_tid=17, entry_tid=8912' 2024-09-11T06:23:47.157 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=82, tag_tid=17, entry_tid=9042' 2024-09-11T06:23:47.157 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:23:47.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:23:47.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:47.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:23:47.159 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=82, tag_tid=17, entry_tid=9042' 2024-09-11T06:23:47.159 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:23:47.159 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:47.160 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:23:47.160 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:23:47.161 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:23:47.161 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:47.161 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:23:47.162 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=82, tag_tid=17, entry_tid=9042' 2024-09-11T06:23:47.162 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:23:47.162 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:47.164 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=9042 2024-09-11T06:23:47.164 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 9042 2024-09-11T06:23:47.164 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:23:47.164 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:23:47.164 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:23:47.365 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:23:47.365 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:23:47.366 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:23:47.366 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:23:47.366 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:23:47.366 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:23:47.366 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:23:47.366 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:23:47.366 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:23:47.366 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:23:47.366 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:23:47.366 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:23:47.366 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:23:47.366 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:23:47.366 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:23:47.367 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:23:47.367 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:23:47.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:23:47.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:23:47.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:23:47.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:23:47.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:23:47.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:23:47.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:23:47.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:23:47.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:23:47.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:23:47.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:23:47.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:23:47.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:23:47.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:23:47.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:23:47.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:23:47.372 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:23:47.372 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:23:47.372 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:23:47.372 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:23:47.372 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:23:47.372 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:23:47.372 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:23:47.372 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:23:47.372 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:23:47.372 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:23:47.372 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:23:47.372 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:23:47.372 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:23:47.372 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:23:47.373 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:23:47.373 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:23:47.396 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:23:47.396 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:23:47.396 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:23:47.396 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:23:47.532 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:23:47.532 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:23:47.532 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:47.533 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:47.533 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:47.533 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:23:47.533 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:47.533 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:47.533 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:47.533 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:23:47.535 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:23:47.535 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:23:47.535 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:23:47.536 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:47.536 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:47.537 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:23:47.590 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 20 2024-09-11T06:23:47.591 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:23:47.591 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:23:47.591 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:23:47.591 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=80, tag_tid=17, entry_tid=9124], [object_number=83, tag_tid=17, entry_tid=9123], [object_number=82, tag_tid=17, entry_tid=9122], [object_number=81, tag_tid=17, entry_tid=9121]]], state=connected] 2024-09-11T06:23:47.594 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:47.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:23:47.596 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:23:47.596 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:47.596 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:47.596 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:47.596 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:23:47.596 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:47.596 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:47.597 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:47.597 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:23:47.597 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:23:47.597 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:23:47.597 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:23:47.597 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:47.598 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:23:47.598 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:47.645 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 20 2024-09-11T06:23:47.645 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:23:47.645 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:23:47.645 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:23:47.645 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=80, tag_tid=17, entry_tid=9124], [object_number=83, tag_tid=17, entry_tid=9123], [object_number=82, tag_tid=17, entry_tid=9122], [object_number=81, tag_tid=17, entry_tid=9121]]], state=connected] 2024-09-11T06:23:47.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:47.650 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=80, tag_tid=17, entry_tid=9124' 2024-09-11T06:23:47.650 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=80, tag_tid=17, entry_tid=9124' 2024-09-11T06:23:47.650 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=80, tag_tid=17, entry_tid=9124' '!=' 'object_number=82, tag_tid=17, entry_tid=9042' 2024-09-11T06:23:47.650 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:23:47.650 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=80, tag_tid=17, entry_tid=9124' = 'object_number=82, tag_tid=17, entry_tid=9042' 2024-09-11T06:23:47.651 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=80, tag_tid=17, entry_tid=9124' 2024-09-11T06:23:47.651 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:23:47.651 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:23:47.651 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:47.652 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:23:47.652 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=80, tag_tid=17, entry_tid=9124' 2024-09-11T06:23:47.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:23:47.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:47.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:23:47.654 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:23:47.654 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:23:47.654 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:47.655 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:23:47.656 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=80, tag_tid=17, entry_tid=9124' 2024-09-11T06:23:47.656 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:23:47.656 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:47.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=9124 2024-09-11T06:23:47.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 9124 2024-09-11T06:23:47.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:23:47.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:23:47.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:23:47.859 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:23:47.859 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:23:47.859 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:23:47.859 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:23:47.859 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:23:47.859 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:23:47.859 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:23:47.859 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:23:47.860 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:23:47.860 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:23:47.860 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:23:47.860 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:23:47.860 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:23:47.860 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:23:47.860 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:23:47.861 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:23:47.861 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:23:47.864 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:23:47.864 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:23:47.864 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:23:47.864 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:23:47.864 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:23:47.864 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:23:47.865 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:23:47.865 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:23:47.865 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:23:47.865 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:23:47.865 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:23:47.865 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:23:47.865 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:23:47.865 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:23:47.865 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:23:47.865 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:23:47.865 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:23:47.865 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:23:47.865 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:23:47.866 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:23:47.866 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:23:47.866 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:23:47.866 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:23:47.866 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:23:47.866 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:23:47.866 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:23:47.866 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:23:47.866 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:23:47.866 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:23:47.866 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:23:47.866 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:23:47.866 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:23:47.889 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:23:47.889 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:23:47.889 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:23:47.889 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:23:48.016 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:23:48.017 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:23:48.017 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:48.017 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:48.017 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:48.017 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:23:48.017 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:48.017 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:48.017 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:48.017 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:23:48.017 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:23:48.018 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:23:48.018 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:23:48.020 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:48.020 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:23:48.020 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:48.072 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 20 2024-09-11T06:23:48.072 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:23:48.072 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:23:48.072 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:23:48.072 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=83, tag_tid=17, entry_tid=9211], [object_number=82, tag_tid=17, entry_tid=9210], [object_number=81, tag_tid=17, entry_tid=9209], [object_number=80, tag_tid=17, entry_tid=9208]]], state=connected] 2024-09-11T06:23:48.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:48.077 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:23:48.078 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:23:48.078 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:48.078 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:48.078 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:48.078 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:23:48.078 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:48.078 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:48.079 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:48.079 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:23:48.079 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:23:48.079 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:23:48.079 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:23:48.079 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:48.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:23:48.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:48.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 20 2024-09-11T06:23:48.128 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:23:48.128 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:23:48.128 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:23:48.128 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=83, tag_tid=17, entry_tid=9211], [object_number=82, tag_tid=17, entry_tid=9210], [object_number=81, tag_tid=17, entry_tid=9209], [object_number=80, tag_tid=17, entry_tid=9208]]], state=connected] 2024-09-11T06:23:48.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:48.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=83, tag_tid=17, entry_tid=9211' 2024-09-11T06:23:48.133 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=83, tag_tid=17, entry_tid=9211' 2024-09-11T06:23:48.133 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=83, tag_tid=17, entry_tid=9211' '!=' 'object_number=80, tag_tid=17, entry_tid=9124' 2024-09-11T06:23:48.133 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:23:48.133 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=83, tag_tid=17, entry_tid=9211' = 'object_number=80, tag_tid=17, entry_tid=9124' 2024-09-11T06:23:48.133 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=83, tag_tid=17, entry_tid=9211' 2024-09-11T06:23:48.133 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:23:48.133 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:23:48.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:48.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:23:48.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=83, tag_tid=17, entry_tid=9211' 2024-09-11T06:23:48.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:23:48.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:48.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:23:48.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:23:48.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:23:48.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:48.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:23:48.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=83, tag_tid=17, entry_tid=9211' 2024-09-11T06:23:48.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:23:48.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:48.139 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=9211 2024-09-11T06:23:48.139 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 9211 2024-09-11T06:23:48.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:23:48.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:23:48.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:23:48.341 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:23:48.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:23:48.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:23:48.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:23:48.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:23:48.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:23:48.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:23:48.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:23:48.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:23:48.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:23:48.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:23:48.343 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:23:48.343 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:23:48.343 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:23:48.343 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:23:48.343 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:23:48.343 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:23:48.346 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:23:48.346 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:23:48.346 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:23:48.346 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:23:48.346 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:23:48.346 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:23:48.347 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:23:48.347 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:23:48.347 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:23:48.347 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:23:48.347 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:23:48.347 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:23:48.347 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:23:48.347 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:23:48.347 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:23:48.347 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:23:48.347 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:23:48.347 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:23:48.347 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:23:48.347 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:23:48.348 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:23:48.348 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:23:48.348 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:23:48.348 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:23:48.348 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:23:48.348 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:23:48.348 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:23:48.348 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:23:48.348 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:23:48.348 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:23:48.348 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:23:48.348 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:23:48.373 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:23:48.373 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:23:48.373 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:23:48.373 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:23:48.503 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:23:48.504 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:23:48.504 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:48.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:48.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:48.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:23:48.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:48.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:48.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:48.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:23:48.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:23:48.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:23:48.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:23:48.507 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:48.507 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:23:48.507 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:48.559 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 20 2024-09-11T06:23:48.559 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:23:48.559 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:23:48.559 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:23:48.559 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=81, tag_tid=17, entry_tid=9297], [object_number=80, tag_tid=17, entry_tid=9296], [object_number=83, tag_tid=17, entry_tid=9295], [object_number=82, tag_tid=17, entry_tid=9294]]], state=connected] 2024-09-11T06:23:48.562 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:48.563 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:23:48.565 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:23:48.565 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:48.565 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:48.565 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:48.565 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:23:48.565 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:48.565 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:48.565 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:48.566 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:23:48.566 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:23:48.566 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:23:48.566 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:23:48.566 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:48.566 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:23:48.566 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:48.614 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 20 2024-09-11T06:23:48.614 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:23:48.614 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:23:48.614 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:23:48.614 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=81, tag_tid=17, entry_tid=9297], [object_number=80, tag_tid=17, entry_tid=9296], [object_number=83, tag_tid=17, entry_tid=9295], [object_number=82, tag_tid=17, entry_tid=9294]]], state=connected] 2024-09-11T06:23:48.617 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:48.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=81, tag_tid=17, entry_tid=9297' 2024-09-11T06:23:48.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=81, tag_tid=17, entry_tid=9297' 2024-09-11T06:23:48.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=81, tag_tid=17, entry_tid=9297' '!=' 'object_number=83, tag_tid=17, entry_tid=9211' 2024-09-11T06:23:48.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:23:48.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=81, tag_tid=17, entry_tid=9297' = 'object_number=83, tag_tid=17, entry_tid=9211' 2024-09-11T06:23:48.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=81, tag_tid=17, entry_tid=9297' 2024-09-11T06:23:48.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:23:48.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:23:48.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:48.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:23:48.621 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=81, tag_tid=17, entry_tid=9297' 2024-09-11T06:23:48.621 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:23:48.621 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:48.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:23:48.623 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:23:48.623 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:23:48.623 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:48.624 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:23:48.624 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=81, tag_tid=17, entry_tid=9297' 2024-09-11T06:23:48.625 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:23:48.625 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:48.626 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=9297 2024-09-11T06:23:48.626 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 9297 2024-09-11T06:23:48.626 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:23:48.626 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:23:48.626 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:23:48.827 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:23:48.828 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:23:48.828 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:23:48.828 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:23:48.828 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:23:48.828 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:23:48.828 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:23:48.828 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:23:48.828 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:23:48.828 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:23:48.828 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:23:48.828 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:23:48.828 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:23:48.829 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:23:48.829 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:23:48.829 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:23:48.829 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:23:48.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:23:48.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:23:48.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:23:48.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:23:48.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:23:48.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:23:48.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:23:48.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:23:48.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:23:48.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:23:48.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:23:48.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:23:48.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:23:48.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:23:48.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:23:48.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:23:48.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:23:48.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:23:48.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:23:48.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:23:48.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:23:48.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:23:48.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:23:48.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:23:48.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:23:48.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:23:48.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:23:48.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:23:48.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:23:48.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:23:48.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:23:48.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:23:48.857 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:23:48.858 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:23:48.858 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:23:48.858 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:23:48.985 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:23:48.987 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:23:48.988 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:48.988 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:48.988 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:48.988 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:23:48.988 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:48.988 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:48.988 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:48.988 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:23:48.988 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:23:48.988 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:23:48.988 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:23:48.989 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:48.989 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:48.989 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:23:49.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 20 2024-09-11T06:23:49.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:23:49.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:23:49.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:23:49.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=80, tag_tid=17, entry_tid=9388], [object_number=83, tag_tid=17, entry_tid=9387], [object_number=82, tag_tid=17, entry_tid=9386], [object_number=81, tag_tid=17, entry_tid=9385]]], state=connected] 2024-09-11T06:23:49.045 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:49.047 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:23:49.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:23:49.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:49.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:49.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:49.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:23:49.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:49.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:49.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:49.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:23:49.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:23:49.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:23:49.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:23:49.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:49.050 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:23:49.050 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:49.097 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 20 2024-09-11T06:23:49.097 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:23:49.097 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:23:49.097 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:23:49.097 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=80, tag_tid=17, entry_tid=9388], [object_number=83, tag_tid=17, entry_tid=9387], [object_number=82, tag_tid=17, entry_tid=9386], [object_number=81, tag_tid=17, entry_tid=9385]]], state=connected] 2024-09-11T06:23:49.100 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:49.102 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=80, tag_tid=17, entry_tid=9388' 2024-09-11T06:23:49.102 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=80, tag_tid=17, entry_tid=9388' 2024-09-11T06:23:49.102 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=80, tag_tid=17, entry_tid=9388' '!=' 'object_number=81, tag_tid=17, entry_tid=9297' 2024-09-11T06:23:49.102 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:23:49.102 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=80, tag_tid=17, entry_tid=9388' = 'object_number=81, tag_tid=17, entry_tid=9297' 2024-09-11T06:23:49.102 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=80, tag_tid=17, entry_tid=9388' 2024-09-11T06:23:49.103 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:23:49.103 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:23:49.103 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:49.103 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:23:49.104 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=80, tag_tid=17, entry_tid=9388' 2024-09-11T06:23:49.104 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:23:49.104 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:49.105 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:23:49.106 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:23:49.106 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:23:49.106 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:49.108 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:23:49.108 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=80, tag_tid=17, entry_tid=9388' 2024-09-11T06:23:49.108 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:23:49.108 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:49.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=9388 2024-09-11T06:23:49.110 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 9388 2024-09-11T06:23:49.110 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:23:49.110 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:23:49.110 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:23:49.311 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:23:49.311 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:23:49.311 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:23:49.311 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:23:49.311 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:23:49.311 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:23:49.311 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:23:49.311 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:23:49.311 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:23:49.311 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:23:49.311 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:23:49.312 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:23:49.312 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:23:49.312 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:23:49.312 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:23:49.312 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:23:49.312 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:23:49.314 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:23:49.315 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:23:49.315 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:23:49.315 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:23:49.315 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:23:49.315 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:23:49.315 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:23:49.315 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:23:49.315 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:23:49.315 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:23:49.315 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:23:49.315 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:23:49.315 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:23:49.315 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:23:49.315 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:23:49.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:23:49.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:23:49.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:23:49.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:23:49.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:23:49.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:23:49.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:23:49.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:23:49.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:23:49.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:23:49.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:23:49.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:23:49.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:23:49.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:23:49.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:23:49.317 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:23:49.317 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:23:49.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:23:49.337 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:23:49.337 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:23:49.337 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:23:49.466 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:23:49.468 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:23:49.469 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:49.469 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:49.469 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:49.469 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:23:49.469 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:49.469 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:49.469 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:49.470 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:23:49.470 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:23:49.470 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:23:49.470 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:23:49.471 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:49.471 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:23:49.471 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:49.518 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 20 2024-09-11T06:23:49.519 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:23:49.519 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:23:49.519 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:23:49.520 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=80, tag_tid=17, entry_tid=9492], [object_number=83, tag_tid=17, entry_tid=9491], [object_number=82, tag_tid=17, entry_tid=9490], [object_number=81, tag_tid=17, entry_tid=9489]]], state=connected] 2024-09-11T06:23:49.522 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:49.523 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:23:49.525 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:23:49.525 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:49.525 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:49.525 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:49.525 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:23:49.525 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:49.525 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:49.525 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:49.525 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:23:49.525 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:23:49.525 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:23:49.525 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:23:49.526 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:49.526 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:23:49.526 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:49.572 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 20 2024-09-11T06:23:49.572 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:23:49.572 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:23:49.572 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:23:49.572 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=80, tag_tid=17, entry_tid=9492], [object_number=83, tag_tid=17, entry_tid=9491], [object_number=82, tag_tid=17, entry_tid=9490], [object_number=81, tag_tid=17, entry_tid=9489]]], state=connected] 2024-09-11T06:23:49.575 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:49.577 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=80, tag_tid=17, entry_tid=9492' 2024-09-11T06:23:49.577 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=80, tag_tid=17, entry_tid=9492' 2024-09-11T06:23:49.578 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=80, tag_tid=17, entry_tid=9492' '!=' 'object_number=80, tag_tid=17, entry_tid=9388' 2024-09-11T06:23:49.578 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:23:49.578 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=80, tag_tid=17, entry_tid=9492' = 'object_number=80, tag_tid=17, entry_tid=9388' 2024-09-11T06:23:49.578 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=80, tag_tid=17, entry_tid=9492' 2024-09-11T06:23:49.578 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:23:49.578 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:23:49.578 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:49.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:23:49.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=80, tag_tid=17, entry_tid=9492' 2024-09-11T06:23:49.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:23:49.580 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:49.580 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:23:49.581 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:23:49.581 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:23:49.581 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:49.582 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:23:49.583 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=80, tag_tid=17, entry_tid=9492' 2024-09-11T06:23:49.583 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:23:49.583 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:49.584 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=9492 2024-09-11T06:23:49.584 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 9492 2024-09-11T06:23:49.584 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:23:49.584 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:23:49.584 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:23:49.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:23:49.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:23:49.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:23:49.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:23:49.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:23:49.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:23:49.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:23:49.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:23:49.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:23:49.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:23:49.787 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:23:49.787 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:23:49.787 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:23:49.787 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:23:49.787 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:23:49.787 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:23:49.787 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:23:49.789 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:23:49.789 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:23:49.790 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:23:49.790 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:23:49.790 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:23:49.790 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:23:49.790 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:23:49.790 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:23:49.790 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:23:49.790 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:23:49.790 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:23:49.790 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:23:49.790 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:23:49.790 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:23:49.790 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:23:49.791 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:23:49.792 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:23:49.792 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:23:49.792 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:23:49.792 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:23:49.792 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:23:49.792 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:23:49.792 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:23:49.792 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:23:49.792 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:23:49.793 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:23:49.793 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:23:49.793 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:23:49.793 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:23:49.793 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:23:49.793 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:23:49.793 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:23:49.818 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:23:49.818 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:23:49.818 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:23:49.818 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:23:49.944 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:23:49.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:23:49.946 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:49.946 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:49.946 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:49.946 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:23:49.946 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:49.946 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:49.946 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:49.946 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:23:49.947 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:23:49.947 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:23:49.947 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:23:49.949 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:49.949 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:23:49.949 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:49.999 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 20 2024-09-11T06:23:49.999 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:23:49.999 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:23:49.999 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:23:49.999 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=82, tag_tid=17, entry_tid=9578], [object_number=81, tag_tid=17, entry_tid=9577], [object_number=80, tag_tid=17, entry_tid=9576], [object_number=83, tag_tid=17, entry_tid=9575]]], state=connected] 2024-09-11T06:23:50.004 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:50.005 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:23:50.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:23:50.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:50.007 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:50.007 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:50.007 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:23:50.007 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:50.007 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:50.007 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:50.007 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:23:50.007 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:23:50.007 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:23:50.007 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:23:50.008 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:50.008 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:23:50.008 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:50.057 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 20 2024-09-11T06:23:50.057 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:23:50.057 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:23:50.058 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:23:50.058 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=82, tag_tid=17, entry_tid=9578], [object_number=81, tag_tid=17, entry_tid=9577], [object_number=80, tag_tid=17, entry_tid=9576], [object_number=83, tag_tid=17, entry_tid=9575]]], state=connected] 2024-09-11T06:23:50.060 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:50.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=82, tag_tid=17, entry_tid=9578' 2024-09-11T06:23:50.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=82, tag_tid=17, entry_tid=9578' 2024-09-11T06:23:50.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=82, tag_tid=17, entry_tid=9578' '!=' 'object_number=80, tag_tid=17, entry_tid=9492' 2024-09-11T06:23:50.063 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:23:50.063 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=82, tag_tid=17, entry_tid=9578' = 'object_number=80, tag_tid=17, entry_tid=9492' 2024-09-11T06:23:50.063 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=82, tag_tid=17, entry_tid=9578' 2024-09-11T06:23:50.063 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:23:50.063 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:23:50.063 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:50.065 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:23:50.065 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=82, tag_tid=17, entry_tid=9578' 2024-09-11T06:23:50.065 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:23:50.065 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:50.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:23:50.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:23:50.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:23:50.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:50.067 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:23:50.068 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=82, tag_tid=17, entry_tid=9578' 2024-09-11T06:23:50.068 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:23:50.068 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:50.069 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=9578 2024-09-11T06:23:50.069 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 9578 2024-09-11T06:23:50.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:23:50.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:23:50.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:23:50.271 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:23:50.271 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:23:50.271 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:23:50.271 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:23:50.272 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:23:50.272 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:23:50.272 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:23:50.272 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:23:50.272 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:23:50.272 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:23:50.272 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:23:50.272 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:23:50.272 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:23:50.272 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:23:50.272 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:23:50.273 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:23:50.273 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:23:50.276 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:23:50.276 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:23:50.276 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:23:50.277 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:23:50.277 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:23:50.277 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:23:50.277 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:23:50.277 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:23:50.277 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:23:50.277 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:23:50.277 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:23:50.277 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:23:50.277 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:23:50.277 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:23:50.277 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:23:50.277 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:23:50.277 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:23:50.278 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:23:50.278 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:23:50.278 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:23:50.278 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:23:50.278 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:23:50.278 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:23:50.278 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:23:50.278 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:23:50.278 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:23:50.278 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:23:50.278 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:23:50.278 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:23:50.279 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:23:50.279 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:23:50.279 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:23:50.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:23:50.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:23:50.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:23:50.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:23:50.435 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:23:50.438 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:23:50.438 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:50.438 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:50.438 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:50.439 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:23:50.439 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:50.439 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:50.439 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:50.439 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:23:50.439 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:23:50.439 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:23:50.439 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:23:50.440 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:50.440 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:23:50.440 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:50.490 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 20 2024-09-11T06:23:50.491 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:23:50.491 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:23:50.491 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:23:50.491 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=81, tag_tid=17, entry_tid=9673], [object_number=80, tag_tid=17, entry_tid=9672], [object_number=83, tag_tid=17, entry_tid=9671], [object_number=82, tag_tid=17, entry_tid=9670]]], state=connected] 2024-09-11T06:23:50.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:50.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:23:50.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:23:50.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:50.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:50.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:50.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:23:50.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:50.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:50.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:50.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:23:50.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:23:50.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:23:50.498 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:23:50.498 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:50.498 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:23:50.498 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:50.545 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 20 2024-09-11T06:23:50.545 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:23:50.546 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:23:50.546 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:23:50.546 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=81, tag_tid=17, entry_tid=9673], [object_number=80, tag_tid=17, entry_tid=9672], [object_number=83, tag_tid=17, entry_tid=9671], [object_number=82, tag_tid=17, entry_tid=9670]]], state=connected] 2024-09-11T06:23:50.549 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:50.551 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=81, tag_tid=17, entry_tid=9673' 2024-09-11T06:23:50.551 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=81, tag_tid=17, entry_tid=9673' 2024-09-11T06:23:50.551 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=81, tag_tid=17, entry_tid=9673' '!=' 'object_number=82, tag_tid=17, entry_tid=9578' 2024-09-11T06:23:50.551 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:23:50.551 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=81, tag_tid=17, entry_tid=9673' = 'object_number=82, tag_tid=17, entry_tid=9578' 2024-09-11T06:23:50.551 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=81, tag_tid=17, entry_tid=9673' 2024-09-11T06:23:50.551 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:23:50.551 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:23:50.551 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:50.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:23:50.553 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=81, tag_tid=17, entry_tid=9673' 2024-09-11T06:23:50.553 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:23:50.553 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:50.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:23:50.555 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:23:50.555 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:23:50.555 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:50.556 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:23:50.556 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=81, tag_tid=17, entry_tid=9673' 2024-09-11T06:23:50.557 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:23:50.557 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:50.558 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=9673 2024-09-11T06:23:50.558 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 9673 2024-09-11T06:23:50.558 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:23:50.558 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:23:50.558 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:23:50.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:23:50.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:23:50.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:23:50.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:23:50.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:23:50.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:23:50.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:23:50.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:23:50.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:23:50.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:23:50.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:23:50.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:23:50.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:23:50.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:23:50.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:23:50.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:23:50.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:23:50.764 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:23:50.764 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:23:50.764 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:23:50.764 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:23:50.764 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:23:50.764 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:23:50.764 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:23:50.764 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:23:50.764 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:23:50.764 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:23:50.765 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:23:50.765 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:23:50.765 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:23:50.765 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:23:50.765 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:23:50.765 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:23:50.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:23:50.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:23:50.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:23:50.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:23:50.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:23:50.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:23:50.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:23:50.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:23:50.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:23:50.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:23:50.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:23:50.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:23:50.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:23:50.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:23:50.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:23:50.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:23:50.792 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:23:50.792 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:23:50.792 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:23:50.793 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:23:50.921 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:23:50.922 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:23:50.922 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:50.923 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:50.923 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:50.923 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:23:50.923 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:50.923 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:50.923 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:50.923 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:23:50.924 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:23:50.924 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:23:50.924 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:23:50.925 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:50.925 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:23:50.926 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:50.972 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 20 2024-09-11T06:23:50.972 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:23:50.972 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:23:50.972 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:23:50.972 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=82, tag_tid=17, entry_tid=9762], [object_number=81, tag_tid=17, entry_tid=9761], [object_number=80, tag_tid=17, entry_tid=9760], [object_number=83, tag_tid=17, entry_tid=9759]]], state=connected] 2024-09-11T06:23:50.976 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:50.977 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:23:50.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:23:50.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:50.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:50.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:50.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:23:50.979 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:50.979 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:50.979 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:50.979 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:23:50.979 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:23:50.979 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:23:50.979 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:23:50.980 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:50.980 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:23:50.980 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:51.026 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 20 2024-09-11T06:23:51.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:23:51.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:23:51.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:23:51.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=82, tag_tid=17, entry_tid=9762], [object_number=81, tag_tid=17, entry_tid=9761], [object_number=80, tag_tid=17, entry_tid=9760], [object_number=83, tag_tid=17, entry_tid=9759]]], state=connected] 2024-09-11T06:23:51.030 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:51.032 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=82, tag_tid=17, entry_tid=9762' 2024-09-11T06:23:51.032 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=82, tag_tid=17, entry_tid=9762' 2024-09-11T06:23:51.032 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=82, tag_tid=17, entry_tid=9762' '!=' 'object_number=81, tag_tid=17, entry_tid=9673' 2024-09-11T06:23:51.032 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:23:51.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=82, tag_tid=17, entry_tid=9762' = 'object_number=81, tag_tid=17, entry_tid=9673' 2024-09-11T06:23:51.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=82, tag_tid=17, entry_tid=9762' 2024-09-11T06:23:51.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:23:51.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:23:51.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:51.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:23:51.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=82, tag_tid=17, entry_tid=9762' 2024-09-11T06:23:51.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:23:51.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:51.035 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:23:51.036 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:23:51.036 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:23:51.036 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:51.037 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:23:51.038 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=82, tag_tid=17, entry_tid=9762' 2024-09-11T06:23:51.038 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:23:51.038 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:51.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=9762 2024-09-11T06:23:51.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 9762 2024-09-11T06:23:51.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:23:51.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:23:51.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:23:51.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:23:51.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:23:51.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:23:51.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:23:51.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:23:51.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:23:51.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:23:51.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:23:51.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:23:51.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:23:51.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:23:51.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:23:51.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:23:51.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:23:51.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:23:51.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:23:51.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:23:51.246 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:23:51.246 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:23:51.246 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:23:51.246 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:23:51.246 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:23:51.246 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:23:51.246 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:23:51.246 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:23:51.246 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:23:51.246 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:23:51.246 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:23:51.246 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:23:51.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:23:51.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:23:51.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:23:51.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:23:51.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:23:51.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:23:51.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:23:51.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:23:51.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:23:51.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:23:51.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:23:51.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:23:51.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:23:51.248 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:23:51.248 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:23:51.248 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:23:51.248 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:23:51.248 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:23:51.248 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:23:51.248 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:23:51.272 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:23:51.272 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:23:51.272 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:23:51.273 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:23:51.404 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:23:51.405 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:23:51.405 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:51.405 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:51.405 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:51.405 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:23:51.405 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:51.406 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:51.406 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:51.406 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:23:51.406 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:23:51.406 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:23:51.406 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:23:51.408 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:51.409 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:23:51.409 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:51.457 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 20 2024-09-11T06:23:51.457 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:23:51.457 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:23:51.457 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:23:51.457 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=80, tag_tid=17, entry_tid=9860], [object_number=83, tag_tid=17, entry_tid=9859], [object_number=82, tag_tid=17, entry_tid=9858], [object_number=81, tag_tid=17, entry_tid=9857]]], state=connected] 2024-09-11T06:23:51.461 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:51.462 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:23:51.464 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:23:51.464 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:51.464 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:51.464 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:51.464 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:23:51.464 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:51.464 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:51.464 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:51.464 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:23:51.464 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:23:51.464 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:23:51.464 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:23:51.465 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:51.465 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:23:51.465 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:51.511 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 20 2024-09-11T06:23:51.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:23:51.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:23:51.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:23:51.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=80, tag_tid=17, entry_tid=9860], [object_number=83, tag_tid=17, entry_tid=9859], [object_number=82, tag_tid=17, entry_tid=9858], [object_number=81, tag_tid=17, entry_tid=9857]]], state=connected] 2024-09-11T06:23:51.515 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:51.516 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=80, tag_tid=17, entry_tid=9860' 2024-09-11T06:23:51.517 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=80, tag_tid=17, entry_tid=9860' 2024-09-11T06:23:51.517 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=80, tag_tid=17, entry_tid=9860' '!=' 'object_number=82, tag_tid=17, entry_tid=9762' 2024-09-11T06:23:51.517 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:23:51.517 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=80, tag_tid=17, entry_tid=9860' = 'object_number=82, tag_tid=17, entry_tid=9762' 2024-09-11T06:23:51.517 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=80, tag_tid=17, entry_tid=9860' 2024-09-11T06:23:51.518 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:23:51.518 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:23:51.518 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:51.518 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:23:51.519 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=80, tag_tid=17, entry_tid=9860' 2024-09-11T06:23:51.519 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:23:51.519 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:51.520 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:23:51.520 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:23:51.520 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:23:51.520 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:51.522 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:23:51.522 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=80, tag_tid=17, entry_tid=9860' 2024-09-11T06:23:51.522 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:23:51.522 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:51.523 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=9860 2024-09-11T06:23:51.523 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 9860 2024-09-11T06:23:51.524 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:23:51.524 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:23:51.524 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:23:51.725 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:23:51.725 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:23:51.725 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:23:51.726 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:23:51.726 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:23:51.726 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:23:51.726 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:23:51.726 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:23:51.726 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:23:51.726 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:23:51.726 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:23:51.726 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:23:51.726 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:23:51.726 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:23:51.726 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:23:51.727 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:23:51.727 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:23:51.731 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:23:51.731 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:23:51.731 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:23:51.731 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:23:51.731 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:23:51.731 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:23:51.731 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:23:51.731 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:23:51.731 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:23:51.732 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:23:51.732 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:23:51.732 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:23:51.732 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:23:51.732 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:23:51.732 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:23:51.732 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:23:51.732 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:23:51.732 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:23:51.732 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:23:51.732 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:23:51.732 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:23:51.732 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:23:51.732 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:23:51.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:23:51.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:23:51.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:23:51.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:23:51.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:23:51.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:23:51.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:23:51.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:23:51.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:23:51.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:23:51.756 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:23:51.756 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:23:51.756 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:23:51.884 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:23:51.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:23:51.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:51.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:51.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:51.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:23:51.887 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:51.887 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:51.887 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:51.887 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:23:51.887 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:23:51.887 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:23:51.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:23:51.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:51.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:23:51.889 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:51.936 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 20 2024-09-11T06:23:51.936 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:23:51.936 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:23:51.936 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:23:51.936 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=83, tag_tid=17, entry_tid=9971], [object_number=82, tag_tid=17, entry_tid=9970], [object_number=81, tag_tid=17, entry_tid=9969], [object_number=80, tag_tid=17, entry_tid=9968]]], state=connected] 2024-09-11T06:23:51.939 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:51.941 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:23:51.942 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:23:51.942 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:51.942 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:51.942 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:51.942 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:23:51.942 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:51.942 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:51.942 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:51.942 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:23:51.942 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:23:51.943 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:23:51.943 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:23:51.943 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:51.943 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:23:51.943 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:51.989 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 20 2024-09-11T06:23:51.989 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:23:51.989 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:23:51.989 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:23:51.990 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=83, tag_tid=17, entry_tid=9971], [object_number=82, tag_tid=17, entry_tid=9970], [object_number=81, tag_tid=17, entry_tid=9969], [object_number=80, tag_tid=17, entry_tid=9968]]], state=connected] 2024-09-11T06:23:51.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:51.995 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=83, tag_tid=17, entry_tid=9971' 2024-09-11T06:23:51.995 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=83, tag_tid=17, entry_tid=9971' 2024-09-11T06:23:51.995 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=83, tag_tid=17, entry_tid=9971' '!=' 'object_number=80, tag_tid=17, entry_tid=9860' 2024-09-11T06:23:51.995 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:23:51.995 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=83, tag_tid=17, entry_tid=9971' = 'object_number=80, tag_tid=17, entry_tid=9860' 2024-09-11T06:23:51.995 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=83, tag_tid=17, entry_tid=9971' 2024-09-11T06:23:51.995 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:23:51.995 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:23:51.995 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:51.996 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:23:51.997 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=83, tag_tid=17, entry_tid=9971' 2024-09-11T06:23:51.997 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:23:51.997 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:51.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:23:51.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:23:51.999 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:23:51.999 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:51.999 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:23:52.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=83, tag_tid=17, entry_tid=9971' 2024-09-11T06:23:52.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:23:52.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:52.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=9971 2024-09-11T06:23:52.002 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 9971 2024-09-11T06:23:52.002 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:23:52.002 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:23:52.002 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:23:52.203 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:23:52.203 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:23:52.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:23:52.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:23:52.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:23:52.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:23:52.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:23:52.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:23:52.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:23:52.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:23:52.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:23:52.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:23:52.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:23:52.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:23:52.205 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:23:52.205 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:23:52.205 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:23:52.209 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:23:52.209 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:23:52.209 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:23:52.209 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:23:52.209 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:23:52.209 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:23:52.209 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:23:52.209 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:23:52.209 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:23:52.209 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:23:52.210 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:23:52.210 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:23:52.210 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:23:52.210 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:23:52.210 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:23:52.210 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:23:52.210 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:23:52.210 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:23:52.210 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:23:52.210 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:23:52.210 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:23:52.210 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:23:52.210 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:23:52.211 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:23:52.211 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:23:52.211 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:23:52.211 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:23:52.211 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:23:52.211 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:23:52.211 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:23:52.211 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:23:52.211 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:23:52.234 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:23:52.235 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:23:52.235 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:23:52.235 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:23:52.367 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:23:52.368 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:23:52.368 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:52.368 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:52.368 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:52.368 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:23:52.368 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:52.369 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:52.369 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:52.369 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:23:52.369 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:23:52.369 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:23:52.369 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:23:52.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:52.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:23:52.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:52.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 20 2024-09-11T06:23:52.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:23:52.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:23:52.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:23:52.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=80, tag_tid=17, entry_tid=10080], [object_number=83, tag_tid=17, entry_tid=10079], [object_number=82, tag_tid=17, entry_tid=10078], [object_number=81, tag_tid=17, entry_tid=10077]]], state=connected] 2024-09-11T06:23:52.430 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:52.432 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:23:52.432 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:23:52.432 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:52.433 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:52.433 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:52.433 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:23:52.433 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:52.433 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:52.433 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:52.433 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:23:52.433 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:23:52.433 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:23:52.433 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:23:52.434 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:52.434 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:23:52.434 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:52.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 20 2024-09-11T06:23:52.483 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:23:52.483 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:23:52.483 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:23:52.483 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=80, tag_tid=17, entry_tid=10080], [object_number=83, tag_tid=17, entry_tid=10079], [object_number=82, tag_tid=17, entry_tid=10078], [object_number=81, tag_tid=17, entry_tid=10077]]], state=connected] 2024-09-11T06:23:52.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:52.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=80, tag_tid=17, entry_tid=10080' 2024-09-11T06:23:52.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=80, tag_tid=17, entry_tid=10080' 2024-09-11T06:23:52.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=80, tag_tid=17, entry_tid=10080' '!=' 'object_number=83, tag_tid=17, entry_tid=9971' 2024-09-11T06:23:52.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:23:52.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=80, tag_tid=17, entry_tid=10080' = 'object_number=83, tag_tid=17, entry_tid=9971' 2024-09-11T06:23:52.490 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=80, tag_tid=17, entry_tid=10080' 2024-09-11T06:23:52.490 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:23:52.490 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:23:52.490 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:52.491 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:23:52.492 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=80, tag_tid=17, entry_tid=10080' 2024-09-11T06:23:52.492 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:23:52.492 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:52.493 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:23:52.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:23:52.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:23:52.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:52.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:23:52.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=80, tag_tid=17, entry_tid=10080' 2024-09-11T06:23:52.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:23:52.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:52.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=10080 2024-09-11T06:23:52.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 10080 2024-09-11T06:23:52.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:23:52.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:23:52.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:23:52.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:23:52.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:23:52.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:23:52.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:23:52.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:23:52.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:23:52.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:23:52.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:23:52.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:23:52.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:23:52.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:23:52.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:23:52.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:23:52.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:23:52.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:23:52.700 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:23:52.700 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:23:52.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:23:52.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:23:52.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:23:52.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:23:52.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:23:52.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:23:52.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:23:52.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:23:52.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:23:52.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:23:52.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:23:52.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:23:52.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:23:52.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:23:52.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:23:52.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:23:52.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:23:52.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:23:52.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:23:52.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:23:52.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:23:52.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:23:52.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:23:52.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:23:52.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:23:52.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:23:52.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:23:52.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:23:52.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:23:52.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:23:52.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:23:52.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:23:52.737 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:23:52.737 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:23:52.737 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:23:52.737 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:23:52.864 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:23:52.866 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:23:52.866 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:52.866 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:52.866 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:52.866 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:23:52.866 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:52.866 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:52.866 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:52.867 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:23:52.867 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:23:52.867 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:23:52.867 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:23:52.868 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:52.868 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:23:52.868 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:52.916 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 20 2024-09-11T06:23:52.916 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:23:52.916 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:23:52.917 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:23:52.917 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=85, tag_tid=17, entry_tid=10093], [object_number=84, tag_tid=17, entry_tid=10092], [object_number=87, tag_tid=17, entry_tid=10091], [object_number=82, tag_tid=17, entry_tid=10090]]], state=connected] 2024-09-11T06:23:52.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:52.921 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:23:52.922 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:23:52.922 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:52.922 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:52.922 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:52.922 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:23:52.922 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:52.922 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:52.922 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:52.923 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:23:52.923 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:23:52.923 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:23:52.923 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:23:52.923 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:52.923 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:23:52.924 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:52.970 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 20 2024-09-11T06:23:52.970 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:23:52.971 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:23:52.971 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:23:52.971 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=85, tag_tid=17, entry_tid=10093], [object_number=84, tag_tid=17, entry_tid=10092], [object_number=87, tag_tid=17, entry_tid=10091], [object_number=82, tag_tid=17, entry_tid=10090]]], state=connected] 2024-09-11T06:23:52.974 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:52.976 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=85, tag_tid=17, entry_tid=10093' 2024-09-11T06:23:52.977 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=85, tag_tid=17, entry_tid=10093' 2024-09-11T06:23:52.977 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=85, tag_tid=17, entry_tid=10093' '!=' 'object_number=80, tag_tid=17, entry_tid=10080' 2024-09-11T06:23:52.977 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:23:52.977 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=85, tag_tid=17, entry_tid=10093' = 'object_number=80, tag_tid=17, entry_tid=10080' 2024-09-11T06:23:52.977 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=85, tag_tid=17, entry_tid=10093' 2024-09-11T06:23:52.977 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:23:52.977 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:23:52.977 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:52.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:23:52.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=85, tag_tid=17, entry_tid=10093' 2024-09-11T06:23:52.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:23:52.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:52.979 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:23:52.979 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:23:52.980 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:23:52.980 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:52.981 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:23:52.981 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=85, tag_tid=17, entry_tid=10093' 2024-09-11T06:23:52.981 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:23:52.981 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:52.983 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=10093 2024-09-11T06:23:52.983 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 10093 2024-09-11T06:23:52.983 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:23:52.983 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:23:52.983 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:23:53.185 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:23:53.185 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:23:53.185 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:23:53.185 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:23:53.185 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:23:53.185 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:23:53.185 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:23:53.185 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:23:53.185 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:23:53.185 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:23:53.185 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:23:53.186 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:23:53.186 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:23:53.186 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:23:53.186 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:23:53.188 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:23:53.188 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:23:53.188 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:23:53.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:23:53.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:23:53.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:23:53.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:23:53.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:23:53.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:23:53.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:23:53.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:23:53.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:23:53.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:23:53.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:23:53.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:23:53.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:23:53.190 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:23:53.190 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:23:53.190 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:23:53.190 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:23:53.190 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:23:53.190 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:23:53.190 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:23:53.190 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:23:53.190 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:23:53.190 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:23:53.190 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:23:53.190 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:23:53.190 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:23:53.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:23:53.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:23:53.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:23:53.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:23:53.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:23:53.215 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:23:53.216 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:23:53.216 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:23:53.216 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:23:53.357 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:23:53.357 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:23:53.357 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:53.357 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:53.357 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:53.357 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:23:53.357 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:53.357 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:53.357 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:53.358 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:23:53.358 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:23:53.358 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:23:53.359 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:23:53.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:53.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:53.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:23:53.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 21 2024-09-11T06:23:53.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:23:53.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:23:53.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:23:53.418 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=85, tag_tid=17, entry_tid=10181], [object_number=84, tag_tid=17, entry_tid=10180], [object_number=87, tag_tid=17, entry_tid=10179], [object_number=86, tag_tid=17, entry_tid=10178]]], state=connected] 2024-09-11T06:23:53.421 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:53.422 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:23:53.423 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:23:53.423 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:53.423 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:53.423 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:53.423 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:23:53.424 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:53.424 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:53.424 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:53.424 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:23:53.424 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:23:53.424 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:23:53.424 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:23:53.425 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:53.425 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:23:53.425 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:53.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 21 2024-09-11T06:23:53.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:23:53.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:23:53.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:23:53.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=85, tag_tid=17, entry_tid=10181], [object_number=84, tag_tid=17, entry_tid=10180], [object_number=87, tag_tid=17, entry_tid=10179], [object_number=86, tag_tid=17, entry_tid=10178]]], state=connected] 2024-09-11T06:23:53.478 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:53.479 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=85, tag_tid=17, entry_tid=10181' 2024-09-11T06:23:53.479 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=85, tag_tid=17, entry_tid=10181' 2024-09-11T06:23:53.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=85, tag_tid=17, entry_tid=10181' '!=' 'object_number=85, tag_tid=17, entry_tid=10093' 2024-09-11T06:23:53.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:23:53.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=85, tag_tid=17, entry_tid=10181' = 'object_number=85, tag_tid=17, entry_tid=10093' 2024-09-11T06:23:53.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=85, tag_tid=17, entry_tid=10181' 2024-09-11T06:23:53.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:23:53.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:23:53.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:53.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:23:53.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=85, tag_tid=17, entry_tid=10181' 2024-09-11T06:23:53.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:23:53.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:53.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:23:53.483 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:23:53.483 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:23:53.483 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:53.484 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:23:53.484 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=85, tag_tid=17, entry_tid=10181' 2024-09-11T06:23:53.485 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:23:53.485 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:53.486 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=10181 2024-09-11T06:23:53.486 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 10181 2024-09-11T06:23:53.486 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:23:53.486 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:23:53.486 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:23:53.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:23:53.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:23:53.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:23:53.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:23:53.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:23:53.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:23:53.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:23:53.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:23:53.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:23:53.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:23:53.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:23:53.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:23:53.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:23:53.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:23:53.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:23:53.689 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:23:53.689 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:23:53.693 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:23:53.693 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:23:53.693 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:23:53.693 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:23:53.693 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:23:53.693 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:23:53.694 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:23:53.694 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:23:53.694 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:23:53.694 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:23:53.694 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:23:53.694 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:23:53.694 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:23:53.694 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:23:53.694 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:23:53.694 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:23:53.694 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:23:53.694 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:23:53.694 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:23:53.694 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:23:53.694 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:23:53.695 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:23:53.695 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:23:53.695 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:23:53.695 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:23:53.695 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:23:53.695 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:23:53.695 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:23:53.695 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:23:53.695 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:23:53.695 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:23:53.695 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:23:53.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:23:53.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:23:53.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:23:53.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:23:53.848 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:23:53.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:23:53.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:53.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:53.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:53.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:23:53.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:53.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:53.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:53.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:23:53.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:23:53.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:23:53.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:23:53.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:53.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:23:53.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:53.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 21 2024-09-11T06:23:53.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:23:53.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:23:53.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:23:53.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=84, tag_tid=17, entry_tid=10260], [object_number=87, tag_tid=17, entry_tid=10259], [object_number=86, tag_tid=17, entry_tid=10258], [object_number=85, tag_tid=17, entry_tid=10257]]], state=connected] 2024-09-11T06:23:53.907 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:53.908 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:23:53.909 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:23:53.909 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:53.909 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:53.910 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:53.910 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:23:53.910 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:53.910 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:53.910 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:53.910 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:23:53.910 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:23:53.910 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:23:53.910 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:23:53.911 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:53.911 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:23:53.911 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:53.957 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 21 2024-09-11T06:23:53.958 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:23:53.958 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:23:53.958 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:23:53.958 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=84, tag_tid=17, entry_tid=10260], [object_number=87, tag_tid=17, entry_tid=10259], [object_number=86, tag_tid=17, entry_tid=10258], [object_number=85, tag_tid=17, entry_tid=10257]]], state=connected] 2024-09-11T06:23:53.961 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:53.963 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=84, tag_tid=17, entry_tid=10260' 2024-09-11T06:23:53.963 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=84, tag_tid=17, entry_tid=10260' 2024-09-11T06:23:53.963 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=84, tag_tid=17, entry_tid=10260' '!=' 'object_number=85, tag_tid=17, entry_tid=10181' 2024-09-11T06:23:53.963 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:23:53.963 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=84, tag_tid=17, entry_tid=10260' = 'object_number=85, tag_tid=17, entry_tid=10181' 2024-09-11T06:23:53.963 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=84, tag_tid=17, entry_tid=10260' 2024-09-11T06:23:53.964 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:23:53.964 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:23:53.964 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:53.964 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:23:53.965 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=84, tag_tid=17, entry_tid=10260' 2024-09-11T06:23:53.965 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:23:53.965 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:53.966 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:23:53.967 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:23:53.967 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:23:53.967 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:53.968 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:23:53.968 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=84, tag_tid=17, entry_tid=10260' 2024-09-11T06:23:53.969 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:23:53.969 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:53.970 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=10260 2024-09-11T06:23:53.970 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 10260 2024-09-11T06:23:53.970 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:23:53.970 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:23:53.970 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:23:54.172 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:23:54.172 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:23:54.172 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:23:54.172 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:23:54.172 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:23:54.172 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:23:54.172 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:23:54.172 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:23:54.172 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:23:54.172 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:23:54.173 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:23:54.173 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:23:54.173 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:23:54.173 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:23:54.173 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:23:54.173 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:23:54.174 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:23:54.177 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:23:54.177 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:23:54.177 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:23:54.177 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:23:54.177 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:23:54.178 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:23:54.178 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:23:54.178 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:23:54.178 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:23:54.178 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:23:54.178 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:23:54.178 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:23:54.178 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:23:54.178 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:23:54.178 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:23:54.178 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:23:54.178 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:23:54.178 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:23:54.178 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:23:54.179 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:23:54.179 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:23:54.179 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:23:54.179 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:23:54.179 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:23:54.179 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:23:54.179 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:23:54.179 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:23:54.179 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:23:54.179 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:23:54.179 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:23:54.179 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:23:54.179 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:23:54.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:23:54.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:23:54.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:23:54.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:23:54.337 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:23:54.338 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:23:54.338 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:54.338 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:54.338 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:54.339 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:23:54.339 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:54.339 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:54.339 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:54.339 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:23:54.339 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:23:54.339 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:23:54.340 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:23:54.340 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:54.341 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:23:54.341 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:54.387 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 21 2024-09-11T06:23:54.387 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:23:54.387 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:23:54.387 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:23:54.387 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=84, tag_tid=17, entry_tid=10348], [object_number=87, tag_tid=17, entry_tid=10347], [object_number=86, tag_tid=17, entry_tid=10346], [object_number=85, tag_tid=17, entry_tid=10345]]], state=connected] 2024-09-11T06:23:54.390 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:54.392 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:23:54.393 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:23:54.393 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:54.393 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:54.393 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:54.393 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:23:54.394 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:54.394 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:54.394 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:54.394 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:23:54.394 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:23:54.394 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:23:54.394 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:23:54.395 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:54.395 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:23:54.395 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:54.444 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 21 2024-09-11T06:23:54.445 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:23:54.445 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:23:54.445 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:23:54.445 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=84, tag_tid=17, entry_tid=10348], [object_number=87, tag_tid=17, entry_tid=10347], [object_number=86, tag_tid=17, entry_tid=10346], [object_number=85, tag_tid=17, entry_tid=10345]]], state=connected] 2024-09-11T06:23:54.448 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:54.450 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=84, tag_tid=17, entry_tid=10348' 2024-09-11T06:23:54.451 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=84, tag_tid=17, entry_tid=10348' 2024-09-11T06:23:54.451 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=84, tag_tid=17, entry_tid=10348' '!=' 'object_number=84, tag_tid=17, entry_tid=10260' 2024-09-11T06:23:54.451 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:23:54.451 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=84, tag_tid=17, entry_tid=10348' = 'object_number=84, tag_tid=17, entry_tid=10260' 2024-09-11T06:23:54.451 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=84, tag_tid=17, entry_tid=10348' 2024-09-11T06:23:54.451 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:23:54.451 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:23:54.451 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:54.452 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:23:54.452 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=84, tag_tid=17, entry_tid=10348' 2024-09-11T06:23:54.452 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:23:54.452 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:54.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:23:54.454 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:23:54.454 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:23:54.454 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:54.455 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:23:54.456 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=84, tag_tid=17, entry_tid=10348' 2024-09-11T06:23:54.456 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:23:54.456 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:54.457 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=10348 2024-09-11T06:23:54.457 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 10348 2024-09-11T06:23:54.457 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:23:54.457 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:23:54.457 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:23:54.659 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:23:54.659 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:23:54.659 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:23:54.659 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:23:54.659 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:23:54.659 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:23:54.659 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:23:54.659 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:23:54.660 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:23:54.660 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:23:54.660 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:23:54.660 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:23:54.660 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:23:54.660 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:23:54.660 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:23:54.660 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:23:54.661 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:23:54.662 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:23:54.662 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:23:54.662 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:23:54.662 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:23:54.662 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:23:54.662 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:23:54.663 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:23:54.663 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:23:54.663 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:23:54.663 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:23:54.663 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:23:54.663 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:23:54.663 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:23:54.663 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:23:54.663 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:23:54.663 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:23:54.665 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:23:54.665 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:23:54.665 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:23:54.665 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:23:54.665 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:23:54.665 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:23:54.665 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:23:54.665 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:23:54.665 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:23:54.665 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:23:54.665 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:23:54.665 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:23:54.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:23:54.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:23:54.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:23:54.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:23:54.689 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:23:54.689 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:23:54.689 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:23:54.690 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:23:54.817 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:23:54.818 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:23:54.818 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:54.818 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:54.819 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:54.819 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:23:54.819 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:54.819 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:54.819 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:54.819 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:23:54.819 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:23:54.819 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:23:54.819 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:23:54.820 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:54.820 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:23:54.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:54.867 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 21 2024-09-11T06:23:54.867 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:23:54.867 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:23:54.867 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:23:54.867 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=85, tag_tid=17, entry_tid=10441], [object_number=84, tag_tid=17, entry_tid=10440], [object_number=87, tag_tid=17, entry_tid=10439], [object_number=86, tag_tid=17, entry_tid=10438]]], state=connected] 2024-09-11T06:23:54.871 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:54.873 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:23:54.873 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:23:54.873 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:54.873 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:54.873 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:54.873 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:23:54.873 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:54.873 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:54.874 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:54.874 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:23:54.874 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:23:54.874 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:23:54.874 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:23:54.874 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:54.874 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:23:54.875 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:54.921 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 21 2024-09-11T06:23:54.921 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:23:54.921 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:23:54.921 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:23:54.921 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=85, tag_tid=17, entry_tid=10441], [object_number=84, tag_tid=17, entry_tid=10440], [object_number=87, tag_tid=17, entry_tid=10439], [object_number=86, tag_tid=17, entry_tid=10438]]], state=connected] 2024-09-11T06:23:54.924 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:54.926 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=85, tag_tid=17, entry_tid=10441' 2024-09-11T06:23:54.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=85, tag_tid=17, entry_tid=10441' 2024-09-11T06:23:54.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=85, tag_tid=17, entry_tid=10441' '!=' 'object_number=84, tag_tid=17, entry_tid=10348' 2024-09-11T06:23:54.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:23:54.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=85, tag_tid=17, entry_tid=10441' = 'object_number=84, tag_tid=17, entry_tid=10348' 2024-09-11T06:23:54.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=85, tag_tid=17, entry_tid=10441' 2024-09-11T06:23:54.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:23:54.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:23:54.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:54.928 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:23:54.929 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=85, tag_tid=17, entry_tid=10441' 2024-09-11T06:23:54.929 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:23:54.929 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:54.929 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:23:54.930 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:23:54.930 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:23:54.930 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:54.931 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:23:54.931 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=85, tag_tid=17, entry_tid=10441' 2024-09-11T06:23:54.931 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:23:54.931 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:54.933 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=10441 2024-09-11T06:23:54.933 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 10441 2024-09-11T06:23:54.933 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:23:54.933 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:23:54.933 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:23:55.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:23:55.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:23:55.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:23:55.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:23:55.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:23:55.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:23:55.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:23:55.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:23:55.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:23:55.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:23:55.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:23:55.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:23:55.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:23:55.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:23:55.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:23:55.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:23:55.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:23:55.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:23:55.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:23:55.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:23:55.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:23:55.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:23:55.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:23:55.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:23:55.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:23:55.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:23:55.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:23:55.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:23:55.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:23:55.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:23:55.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:23:55.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:23:55.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:23:55.143 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:23:55.143 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:23:55.143 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:23:55.143 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:23:55.143 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:23:55.143 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:23:55.143 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:23:55.143 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:23:55.143 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:23:55.143 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:23:55.143 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:23:55.143 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:23:55.143 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:23:55.143 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:23:55.144 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:23:55.144 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:23:55.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:23:55.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:23:55.166 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:23:55.166 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:23:55.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:23:55.305 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:23:55.305 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:55.305 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:55.305 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:55.305 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:23:55.305 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:55.305 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:55.305 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:55.305 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:23:55.306 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:23:55.306 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:23:55.306 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:23:55.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:55.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:23:55.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:55.357 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 21 2024-09-11T06:23:55.357 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:23:55.357 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:23:55.358 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:23:55.358 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=86, tag_tid=17, entry_tid=10546], [object_number=85, tag_tid=17, entry_tid=10545], [object_number=84, tag_tid=17, entry_tid=10544], [object_number=87, tag_tid=17, entry_tid=10543]]], state=connected] 2024-09-11T06:23:55.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:55.363 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:23:55.363 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:23:55.364 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:55.364 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:55.364 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:55.364 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:23:55.364 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:55.364 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:55.364 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:55.364 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:23:55.364 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:23:55.364 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:23:55.364 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:23:55.365 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:55.365 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:23:55.365 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:55.411 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 21 2024-09-11T06:23:55.411 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:23:55.411 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:23:55.411 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:23:55.411 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=86, tag_tid=17, entry_tid=10546], [object_number=85, tag_tid=17, entry_tid=10545], [object_number=84, tag_tid=17, entry_tid=10544], [object_number=87, tag_tid=17, entry_tid=10543]]], state=connected] 2024-09-11T06:23:55.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:55.416 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=86, tag_tid=17, entry_tid=10546' 2024-09-11T06:23:55.416 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=86, tag_tid=17, entry_tid=10546' 2024-09-11T06:23:55.416 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=86, tag_tid=17, entry_tid=10546' '!=' 'object_number=85, tag_tid=17, entry_tid=10441' 2024-09-11T06:23:55.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:23:55.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=86, tag_tid=17, entry_tid=10546' = 'object_number=85, tag_tid=17, entry_tid=10441' 2024-09-11T06:23:55.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=86, tag_tid=17, entry_tid=10546' 2024-09-11T06:23:55.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:23:55.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:23:55.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:55.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:23:55.418 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=86, tag_tid=17, entry_tid=10546' 2024-09-11T06:23:55.418 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:23:55.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:55.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:23:55.420 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:23:55.420 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:23:55.420 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:55.421 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:23:55.422 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=86, tag_tid=17, entry_tid=10546' 2024-09-11T06:23:55.422 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:23:55.422 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:55.423 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=10546 2024-09-11T06:23:55.423 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 10546 2024-09-11T06:23:55.423 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:23:55.423 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:23:55.423 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:23:55.625 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:23:55.625 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:23:55.625 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:23:55.625 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:23:55.625 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:23:55.625 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:23:55.625 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:23:55.625 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:23:55.626 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:23:55.626 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:23:55.626 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:23:55.626 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:23:55.626 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:23:55.626 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:23:55.626 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:23:55.628 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:23:55.628 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:23:55.628 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:23:55.628 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:23:55.628 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:23:55.628 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:23:55.628 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:23:55.628 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:23:55.628 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:23:55.629 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:23:55.629 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:23:55.629 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:23:55.629 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:23:55.629 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:23:55.629 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:23:55.629 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:23:55.629 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:23:55.629 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:23:55.629 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:23:55.629 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:23:55.629 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:23:55.629 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:23:55.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:23:55.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:23:55.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:23:55.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:23:55.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:23:55.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:23:55.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:23:55.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:23:55.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:23:55.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:23:55.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:23:55.631 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:23:55.655 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:23:55.655 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:23:55.655 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:23:55.656 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:23:55.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:23:55.787 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:23:55.787 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:55.787 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:55.787 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:55.788 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:23:55.788 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:55.788 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:55.788 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:55.788 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:23:55.788 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:23:55.788 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:23:55.788 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:23:55.789 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:55.789 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:23:55.789 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:55.841 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 21 2024-09-11T06:23:55.841 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:23:55.841 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:23:55.842 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:23:55.842 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=84, tag_tid=17, entry_tid=10636], [object_number=87, tag_tid=17, entry_tid=10635], [object_number=86, tag_tid=17, entry_tid=10634], [object_number=85, tag_tid=17, entry_tid=10633]]], state=connected] 2024-09-11T06:23:55.845 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:55.846 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:23:55.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:23:55.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:55.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:55.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:55.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:23:55.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:55.848 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:55.848 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:55.848 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:23:55.848 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:23:55.848 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:23:55.848 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:23:55.848 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:55.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:23:55.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:55.895 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 21 2024-09-11T06:23:55.896 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:23:55.896 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:23:55.896 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:23:55.896 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=84, tag_tid=17, entry_tid=10636], [object_number=87, tag_tid=17, entry_tid=10635], [object_number=86, tag_tid=17, entry_tid=10634], [object_number=85, tag_tid=17, entry_tid=10633]]], state=connected] 2024-09-11T06:23:55.899 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:55.901 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=84, tag_tid=17, entry_tid=10636' 2024-09-11T06:23:55.901 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=84, tag_tid=17, entry_tid=10636' 2024-09-11T06:23:55.901 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=84, tag_tid=17, entry_tid=10636' '!=' 'object_number=86, tag_tid=17, entry_tid=10546' 2024-09-11T06:23:55.901 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:23:55.901 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=84, tag_tid=17, entry_tid=10636' = 'object_number=86, tag_tid=17, entry_tid=10546' 2024-09-11T06:23:55.901 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=84, tag_tid=17, entry_tid=10636' 2024-09-11T06:23:55.901 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:23:55.901 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:23:55.902 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:55.902 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:23:55.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=84, tag_tid=17, entry_tid=10636' 2024-09-11T06:23:55.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:23:55.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:55.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:23:55.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:23:55.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:23:55.905 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:55.905 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:23:55.906 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=84, tag_tid=17, entry_tid=10636' 2024-09-11T06:23:55.906 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:23:55.906 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:55.907 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=10636 2024-09-11T06:23:55.908 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 10636 2024-09-11T06:23:55.908 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:23:55.908 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:23:55.908 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:23:56.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:23:56.110 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:23:56.110 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:23:56.110 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:23:56.110 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:23:56.110 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:23:56.110 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:23:56.110 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:23:56.110 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:23:56.110 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:23:56.110 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:23:56.110 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:23:56.110 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:23:56.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:23:56.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:23:56.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:23:56.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:23:56.115 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:23:56.115 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:23:56.115 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:23:56.115 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:23:56.115 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:23:56.115 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:23:56.115 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:23:56.115 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:23:56.115 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:23:56.115 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:23:56.115 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:23:56.115 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:23:56.115 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:23:56.116 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:23:56.116 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:23:56.116 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:23:56.116 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:23:56.116 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:23:56.116 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:23:56.116 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:23:56.116 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:23:56.116 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:23:56.116 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:23:56.116 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:23:56.116 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:23:56.116 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:23:56.116 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:23:56.117 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:23:56.117 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:23:56.117 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:23:56.117 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:23:56.117 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:23:56.139 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:23:56.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:23:56.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:23:56.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:23:56.271 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:23:56.272 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:23:56.272 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:56.272 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:56.273 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:56.273 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:23:56.273 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:56.273 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:56.273 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:56.273 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:23:56.274 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:23:56.274 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:23:56.274 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:23:56.276 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:56.276 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:23:56.276 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:56.329 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 21 2024-09-11T06:23:56.329 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:23:56.329 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:23:56.329 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:23:56.330 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=85, tag_tid=17, entry_tid=10741], [object_number=84, tag_tid=17, entry_tid=10740], [object_number=87, tag_tid=17, entry_tid=10739], [object_number=86, tag_tid=17, entry_tid=10738]]], state=connected] 2024-09-11T06:23:56.333 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:56.335 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:23:56.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:23:56.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:56.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:56.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:56.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:23:56.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:56.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:56.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:56.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:23:56.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:23:56.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:23:56.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:23:56.337 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:56.337 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:23:56.337 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:56.384 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 21 2024-09-11T06:23:56.384 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:23:56.384 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:23:56.384 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:23:56.384 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=85, tag_tid=17, entry_tid=10741], [object_number=84, tag_tid=17, entry_tid=10740], [object_number=87, tag_tid=17, entry_tid=10739], [object_number=86, tag_tid=17, entry_tid=10738]]], state=connected] 2024-09-11T06:23:56.388 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:56.390 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=85, tag_tid=17, entry_tid=10741' 2024-09-11T06:23:56.390 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=85, tag_tid=17, entry_tid=10741' 2024-09-11T06:23:56.390 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=85, tag_tid=17, entry_tid=10741' '!=' 'object_number=84, tag_tid=17, entry_tid=10636' 2024-09-11T06:23:56.390 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:23:56.390 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=85, tag_tid=17, entry_tid=10741' = 'object_number=84, tag_tid=17, entry_tid=10636' 2024-09-11T06:23:56.390 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=85, tag_tid=17, entry_tid=10741' 2024-09-11T06:23:56.391 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:23:56.391 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:23:56.391 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:56.391 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:23:56.392 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=85, tag_tid=17, entry_tid=10741' 2024-09-11T06:23:56.392 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:23:56.392 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:56.393 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:23:56.393 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:23:56.394 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:23:56.394 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:56.395 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:23:56.395 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=85, tag_tid=17, entry_tid=10741' 2024-09-11T06:23:56.395 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:23:56.395 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:56.397 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=10741 2024-09-11T06:23:56.397 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 10741 2024-09-11T06:23:56.397 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:23:56.397 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:23:56.397 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:23:56.598 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:23:56.599 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:23:56.599 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:23:56.599 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:23:56.599 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:23:56.599 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:23:56.599 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:23:56.599 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:23:56.599 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:23:56.599 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:23:56.599 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:23:56.599 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:23:56.599 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:23:56.599 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:23:56.600 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:23:56.600 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:23:56.600 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:23:56.602 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:23:56.603 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:23:56.603 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:23:56.603 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:23:56.603 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:23:56.603 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:23:56.603 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:23:56.603 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:23:56.603 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:23:56.603 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:23:56.603 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:23:56.603 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:23:56.603 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:23:56.603 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:23:56.604 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:23:56.604 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:23:56.604 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:23:56.604 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:23:56.604 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:23:56.604 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:23:56.604 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:23:56.604 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:23:56.604 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:23:56.604 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:23:56.604 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:23:56.604 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:23:56.604 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:23:56.604 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:23:56.605 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:23:56.605 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:23:56.605 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:23:56.605 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:23:56.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:23:56.631 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:23:56.631 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:23:56.631 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:23:56.757 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:23:56.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:23:56.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:56.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:56.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:56.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:23:56.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:56.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:56.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:56.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:23:56.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:23:56.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:23:56.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:23:56.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:56.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:23:56.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:56.813 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 21 2024-09-11T06:23:56.813 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:23:56.813 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:23:56.813 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:23:56.813 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=85, tag_tid=17, entry_tid=10841], [object_number=84, tag_tid=17, entry_tid=10840], [object_number=87, tag_tid=17, entry_tid=10839], [object_number=86, tag_tid=17, entry_tid=10838]]], state=connected] 2024-09-11T06:23:56.817 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:56.819 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:23:56.819 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:23:56.819 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:56.819 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:56.819 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:56.819 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:23:56.819 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:56.820 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:56.820 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:56.820 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:23:56.820 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:23:56.820 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:23:56.820 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:23:56.820 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:56.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:23:56.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:56.867 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 21 2024-09-11T06:23:56.868 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:23:56.868 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:23:56.868 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:23:56.868 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=85, tag_tid=17, entry_tid=10841], [object_number=84, tag_tid=17, entry_tid=10840], [object_number=87, tag_tid=17, entry_tid=10839], [object_number=86, tag_tid=17, entry_tid=10838]]], state=connected] 2024-09-11T06:23:56.871 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:56.872 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=85, tag_tid=17, entry_tid=10841' 2024-09-11T06:23:56.873 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=85, tag_tid=17, entry_tid=10841' 2024-09-11T06:23:56.873 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=85, tag_tid=17, entry_tid=10841' '!=' 'object_number=85, tag_tid=17, entry_tid=10741' 2024-09-11T06:23:56.873 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:23:56.873 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=85, tag_tid=17, entry_tid=10841' = 'object_number=85, tag_tid=17, entry_tid=10741' 2024-09-11T06:23:56.873 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=85, tag_tid=17, entry_tid=10841' 2024-09-11T06:23:56.873 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:23:56.873 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:23:56.874 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:56.874 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:23:56.875 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=85, tag_tid=17, entry_tid=10841' 2024-09-11T06:23:56.875 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:23:56.875 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:56.876 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:23:56.876 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:23:56.877 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:23:56.877 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:56.877 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:23:56.878 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=85, tag_tid=17, entry_tid=10841' 2024-09-11T06:23:56.878 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:23:56.878 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:56.880 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=10841 2024-09-11T06:23:56.880 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 10841 2024-09-11T06:23:56.880 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:23:56.880 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:23:56.880 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:23:57.082 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:23:57.082 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:23:57.082 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:23:57.082 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:23:57.082 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:23:57.082 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:23:57.082 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:23:57.082 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:23:57.082 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:23:57.082 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:23:57.082 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:23:57.082 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:23:57.083 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:23:57.083 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:23:57.083 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:23:57.083 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:23:57.083 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:23:57.086 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:23:57.086 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:23:57.086 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:23:57.086 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:23:57.086 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:23:57.086 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:23:57.086 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:23:57.086 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:23:57.087 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:23:57.087 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:23:57.087 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:23:57.087 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:23:57.087 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:23:57.087 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:23:57.087 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:23:57.087 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:23:57.087 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:23:57.087 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:23:57.087 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:23:57.087 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:23:57.087 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:23:57.087 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:23:57.088 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:23:57.088 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:23:57.088 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:23:57.088 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:23:57.088 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:23:57.088 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:23:57.088 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:23:57.088 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:23:57.088 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:23:57.088 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:23:57.113 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:23:57.113 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:23:57.113 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:23:57.113 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:23:57.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:23:57.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:23:57.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:57.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:57.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:57.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:23:57.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:57.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:57.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:57.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:23:57.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:23:57.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:23:57.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:23:57.244 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:57.245 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:23:57.245 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:57.295 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 21 2024-09-11T06:23:57.295 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:23:57.295 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:23:57.295 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:23:57.295 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=87, tag_tid=17, entry_tid=10931], [object_number=86, tag_tid=17, entry_tid=10930], [object_number=85, tag_tid=17, entry_tid=10929], [object_number=84, tag_tid=17, entry_tid=10928]]], state=connected] 2024-09-11T06:23:57.298 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:57.301 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:23:57.301 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:23:57.301 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:57.301 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:57.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:57.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:23:57.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:57.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:57.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:57.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:23:57.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:23:57.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:23:57.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:23:57.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:57.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:23:57.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:57.349 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 21 2024-09-11T06:23:57.349 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:23:57.349 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:23:57.349 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:23:57.349 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=87, tag_tid=17, entry_tid=10931], [object_number=86, tag_tid=17, entry_tid=10930], [object_number=85, tag_tid=17, entry_tid=10929], [object_number=84, tag_tid=17, entry_tid=10928]]], state=connected] 2024-09-11T06:23:57.352 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:57.354 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=87, tag_tid=17, entry_tid=10931' 2024-09-11T06:23:57.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=87, tag_tid=17, entry_tid=10931' 2024-09-11T06:23:57.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=87, tag_tid=17, entry_tid=10931' '!=' 'object_number=85, tag_tid=17, entry_tid=10841' 2024-09-11T06:23:57.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:23:57.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=87, tag_tid=17, entry_tid=10931' = 'object_number=85, tag_tid=17, entry_tid=10841' 2024-09-11T06:23:57.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=87, tag_tid=17, entry_tid=10931' 2024-09-11T06:23:57.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:23:57.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:23:57.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:57.356 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:23:57.356 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=87, tag_tid=17, entry_tid=10931' 2024-09-11T06:23:57.356 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:23:57.357 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:57.357 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:23:57.358 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:23:57.358 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:23:57.358 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:57.359 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:23:57.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=87, tag_tid=17, entry_tid=10931' 2024-09-11T06:23:57.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:23:57.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:57.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=10931 2024-09-11T06:23:57.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 10931 2024-09-11T06:23:57.362 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:23:57.362 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:23:57.362 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:23:57.563 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:23:57.563 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:23:57.564 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:23:57.564 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:23:57.564 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:23:57.564 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:23:57.564 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:23:57.564 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:23:57.564 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:23:57.564 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:23:57.564 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:23:57.564 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:23:57.564 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:23:57.564 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:23:57.564 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:23:57.565 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:23:57.565 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:23:57.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:23:57.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:23:57.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:23:57.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:23:57.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:23:57.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:23:57.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:23:57.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:23:57.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:23:57.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:23:57.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:23:57.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:23:57.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:23:57.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:23:57.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:23:57.569 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:23:57.569 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:23:57.570 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:23:57.570 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:23:57.570 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:23:57.570 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:23:57.570 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:23:57.570 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:23:57.570 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:23:57.570 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:23:57.570 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:23:57.570 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:23:57.570 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:23:57.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:23:57.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:23:57.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:23:57.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:23:57.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:23:57.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:23:57.596 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:23:57.596 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:23:57.727 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:23:57.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:23:57.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:57.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:57.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:57.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:23:57.729 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:57.729 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:57.729 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:57.729 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:23:57.729 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:23:57.729 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:23:57.729 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:23:57.731 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:57.732 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:23:57.732 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:58.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 21 2024-09-11T06:23:58.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:23:58.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:23:58.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:23:58.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=87, tag_tid=17, entry_tid=11015], [object_number=86, tag_tid=17, entry_tid=11014], [object_number=85, tag_tid=17, entry_tid=11013], [object_number=84, tag_tid=17, entry_tid=11012]]], state=connected] 2024-09-11T06:23:58.005 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:58.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:23:58.008 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:23:58.008 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:58.008 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:58.008 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:58.008 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:23:58.008 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:58.009 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:58.009 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:58.009 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:23:58.009 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:23:58.009 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:23:58.009 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:23:58.010 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:58.010 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:23:58.010 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:58.064 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 21 2024-09-11T06:23:58.064 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:23:58.064 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:23:58.065 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:23:58.065 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=87, tag_tid=17, entry_tid=11015], [object_number=86, tag_tid=17, entry_tid=11014], [object_number=85, tag_tid=17, entry_tid=11013], [object_number=84, tag_tid=17, entry_tid=11012]]], state=connected] 2024-09-11T06:23:58.068 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:58.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=87, tag_tid=17, entry_tid=11015' 2024-09-11T06:23:58.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=87, tag_tid=17, entry_tid=11015' 2024-09-11T06:23:58.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=87, tag_tid=17, entry_tid=11015' '!=' 'object_number=87, tag_tid=17, entry_tid=10931' 2024-09-11T06:23:58.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:23:58.071 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=87, tag_tid=17, entry_tid=11015' = 'object_number=87, tag_tid=17, entry_tid=10931' 2024-09-11T06:23:58.071 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=87, tag_tid=17, entry_tid=11015' 2024-09-11T06:23:58.071 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:23:58.071 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:23:58.071 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:58.071 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:23:58.072 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=87, tag_tid=17, entry_tid=11015' 2024-09-11T06:23:58.072 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:23:58.072 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:58.073 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:23:58.074 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:23:58.074 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:23:58.074 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:58.076 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:23:58.076 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=87, tag_tid=17, entry_tid=11015' 2024-09-11T06:23:58.076 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:23:58.076 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:58.077 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=11015 2024-09-11T06:23:58.077 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 11015 2024-09-11T06:23:58.077 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:23:58.077 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:23:58.078 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:23:58.279 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:23:58.325 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:23:58.325 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:23:58.325 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:23:58.325 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:23:58.325 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:23:58.325 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:23:58.325 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:23:58.325 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:23:58.326 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:23:58.326 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:23:58.326 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:23:58.326 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:23:58.326 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:23:58.326 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:23:58.329 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:23:58.329 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:23:58.329 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:23:58.329 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:23:58.329 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:23:58.329 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:23:58.329 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:23:58.329 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:23:58.329 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:23:58.329 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:23:58.330 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:23:58.330 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:23:58.330 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:23:58.330 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:23:58.330 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:23:58.330 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:23:58.330 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:23:58.330 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:23:58.330 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:23:58.331 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:23:58.331 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:23:58.331 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:23:58.331 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:23:58.331 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:23:58.331 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:23:58.331 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:23:58.331 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:23:58.331 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:23:58.331 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:23:58.331 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:23:58.331 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:23:58.331 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:23:58.331 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:23:58.332 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:23:58.332 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:23:58.332 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:23:58.332 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:23:58.332 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:23:58.438 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:23:58.440 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:23:58.440 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:58.440 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:58.440 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:58.440 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:23:58.440 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:58.440 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:58.440 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:58.440 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:23:58.440 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:23:58.441 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:23:58.441 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:23:58.441 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:58.442 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:23:58.442 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:58.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 21 2024-09-11T06:23:58.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:23:58.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:23:58.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:23:58.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=86, tag_tid=17, entry_tid=11138], [object_number=85, tag_tid=17, entry_tid=11137], [object_number=84, tag_tid=17, entry_tid=11136], [object_number=87, tag_tid=17, entry_tid=11135]]], state=connected] 2024-09-11T06:23:58.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:58.499 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:23:58.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:23:58.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:58.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:58.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:58.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:23:58.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:58.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:58.501 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:58.501 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:23:58.501 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:23:58.501 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:23:58.501 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:23:58.501 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:58.502 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:23:58.502 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:58.548 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 21 2024-09-11T06:23:58.548 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:23:58.548 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:23:58.548 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:23:58.548 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=86, tag_tid=17, entry_tid=11138], [object_number=85, tag_tid=17, entry_tid=11137], [object_number=84, tag_tid=17, entry_tid=11136], [object_number=87, tag_tid=17, entry_tid=11135]]], state=connected] 2024-09-11T06:23:58.551 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:58.553 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=86, tag_tid=17, entry_tid=11138' 2024-09-11T06:23:58.553 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=86, tag_tid=17, entry_tid=11138' 2024-09-11T06:23:58.553 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=86, tag_tid=17, entry_tid=11138' '!=' 'object_number=87, tag_tid=17, entry_tid=11015' 2024-09-11T06:23:58.553 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:23:58.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=86, tag_tid=17, entry_tid=11138' = 'object_number=87, tag_tid=17, entry_tid=11015' 2024-09-11T06:23:58.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=86, tag_tid=17, entry_tid=11138' 2024-09-11T06:23:58.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:23:58.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:23:58.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:58.555 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:23:58.555 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=86, tag_tid=17, entry_tid=11138' 2024-09-11T06:23:58.555 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:23:58.555 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:58.556 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:23:58.557 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:23:58.557 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:23:58.557 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:58.558 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:23:58.559 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=86, tag_tid=17, entry_tid=11138' 2024-09-11T06:23:58.559 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:23:58.559 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:58.560 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=11138 2024-09-11T06:23:58.560 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 11138 2024-09-11T06:23:58.560 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:23:58.560 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:23:58.560 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:23:58.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:23:58.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:23:58.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:23:58.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:23:58.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:23:58.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:23:58.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:23:58.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:23:58.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:23:58.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:23:58.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:23:58.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:23:58.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:23:58.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:23:58.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:23:58.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:23:58.764 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:23:58.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:23:58.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:23:58.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:23:58.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:23:58.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:23:58.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:23:58.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:23:58.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:23:58.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:23:58.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:23:58.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:23:58.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:23:58.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:23:58.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:23:58.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:23:58.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:23:58.768 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:23:58.768 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:23:58.768 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:23:58.768 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:23:58.768 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:23:58.768 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:23:58.768 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:23:58.769 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:23:58.769 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:23:58.769 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:23:58.769 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:23:58.769 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:23:58.769 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:23:58.769 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:23:58.769 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:23:58.769 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:23:58.793 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:23:58.793 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:23:58.793 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:23:58.794 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:23:58.934 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:23:58.935 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:23:58.936 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:58.936 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:58.936 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:58.936 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:23:58.936 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:58.936 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:58.936 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:58.936 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:23:58.937 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:23:58.937 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:23:58.937 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:23:58.939 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:58.939 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:23:58.939 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:58.994 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 21 2024-09-11T06:23:58.994 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:23:58.994 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:23:58.994 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:23:58.995 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=85, tag_tid=17, entry_tid=11229], [object_number=84, tag_tid=17, entry_tid=11228], [object_number=87, tag_tid=17, entry_tid=11227], [object_number=86, tag_tid=17, entry_tid=11226]]], state=connected] 2024-09-11T06:23:58.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:59.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:23:59.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:23:59.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:59.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:59.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:59.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:23:59.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:59.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:59.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:59.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:23:59.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:23:59.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:23:59.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:23:59.002 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:59.002 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:23:59.002 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:59.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 21 2024-09-11T06:23:59.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:23:59.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:23:59.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:23:59.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=85, tag_tid=17, entry_tid=11229], [object_number=84, tag_tid=17, entry_tid=11228], [object_number=87, tag_tid=17, entry_tid=11227], [object_number=86, tag_tid=17, entry_tid=11226]]], state=connected] 2024-09-11T06:23:59.052 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:59.054 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=85, tag_tid=17, entry_tid=11229' 2024-09-11T06:23:59.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=85, tag_tid=17, entry_tid=11229' 2024-09-11T06:23:59.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=85, tag_tid=17, entry_tid=11229' '!=' 'object_number=86, tag_tid=17, entry_tid=11138' 2024-09-11T06:23:59.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:23:59.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=85, tag_tid=17, entry_tid=11229' = 'object_number=86, tag_tid=17, entry_tid=11138' 2024-09-11T06:23:59.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=85, tag_tid=17, entry_tid=11229' 2024-09-11T06:23:59.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:23:59.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:23:59.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:59.056 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:23:59.056 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=85, tag_tid=17, entry_tid=11229' 2024-09-11T06:23:59.056 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:23:59.057 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:59.057 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:23:59.058 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:23:59.059 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:23:59.059 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:59.059 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:23:59.060 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=85, tag_tid=17, entry_tid=11229' 2024-09-11T06:23:59.060 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:23:59.060 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:59.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=11229 2024-09-11T06:23:59.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 11229 2024-09-11T06:23:59.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:23:59.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:23:59.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:23:59.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:23:59.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:23:59.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:23:59.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:23:59.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:23:59.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:23:59.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:23:59.264 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:23:59.264 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:23:59.264 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:23:59.264 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:23:59.264 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:23:59.264 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:23:59.264 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:23:59.264 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:23:59.265 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:23:59.265 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:23:59.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:23:59.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:23:59.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:23:59.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:23:59.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:23:59.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:23:59.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:23:59.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:23:59.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:23:59.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:23:59.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:23:59.269 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:23:59.269 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:23:59.269 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:23:59.269 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:23:59.269 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:23:59.269 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:23:59.269 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:23:59.269 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:23:59.269 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:23:59.269 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:23:59.269 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:23:59.269 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:23:59.270 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:23:59.270 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:23:59.270 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:23:59.270 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:23:59.270 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:23:59.270 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:23:59.270 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:23:59.270 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:23:59.270 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:23:59.293 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:23:59.293 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:23:59.293 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:23:59.293 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:23:59.435 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:23:59.437 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:23:59.437 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:59.437 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:59.437 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:59.437 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:23:59.437 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:59.438 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:59.438 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:59.438 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:23:59.438 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:23:59.438 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:23:59.438 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:23:59.440 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:59.440 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:23:59.440 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:59.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 21 2024-09-11T06:23:59.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:23:59.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:23:59.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:23:59.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=87, tag_tid=17, entry_tid=11315], [object_number=86, tag_tid=17, entry_tid=11314], [object_number=85, tag_tid=17, entry_tid=11313], [object_number=84, tag_tid=17, entry_tid=11312]]], state=connected] 2024-09-11T06:23:59.491 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:59.493 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:23:59.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:23:59.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:59.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:59.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:59.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:23:59.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:59.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:59.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:59.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:23:59.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:23:59.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:23:59.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:23:59.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:59.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:23:59.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:59.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 21 2024-09-11T06:23:59.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:23:59.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:23:59.543 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:23:59.543 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=87, tag_tid=17, entry_tid=11315], [object_number=86, tag_tid=17, entry_tid=11314], [object_number=85, tag_tid=17, entry_tid=11313], [object_number=84, tag_tid=17, entry_tid=11312]]], state=connected] 2024-09-11T06:23:59.546 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:59.547 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=87, tag_tid=17, entry_tid=11315' 2024-09-11T06:23:59.547 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=87, tag_tid=17, entry_tid=11315' 2024-09-11T06:23:59.547 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=87, tag_tid=17, entry_tid=11315' '!=' 'object_number=85, tag_tid=17, entry_tid=11229' 2024-09-11T06:23:59.547 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:23:59.548 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=87, tag_tid=17, entry_tid=11315' = 'object_number=85, tag_tid=17, entry_tid=11229' 2024-09-11T06:23:59.548 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=87, tag_tid=17, entry_tid=11315' 2024-09-11T06:23:59.548 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:23:59.548 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:23:59.548 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:59.549 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:23:59.550 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=87, tag_tid=17, entry_tid=11315' 2024-09-11T06:23:59.550 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:23:59.550 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:59.551 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:23:59.551 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:23:59.551 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:23:59.551 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:59.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:23:59.553 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=87, tag_tid=17, entry_tid=11315' 2024-09-11T06:23:59.553 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:23:59.553 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:23:59.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=11315 2024-09-11T06:23:59.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 11315 2024-09-11T06:23:59.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:23:59.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:23:59.555 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:23:59.756 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:23:59.757 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:23:59.757 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:23:59.757 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:23:59.757 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:23:59.757 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:23:59.757 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:23:59.757 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:23:59.757 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:23:59.757 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:23:59.757 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:23:59.757 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:23:59.757 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:23:59.758 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:23:59.758 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:23:59.758 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:23:59.758 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:23:59.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:23:59.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:23:59.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:23:59.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:23:59.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:23:59.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:23:59.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:23:59.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:23:59.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:23:59.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:23:59.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:23:59.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:23:59.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:23:59.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:23:59.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:23:59.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:23:59.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:23:59.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:23:59.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:23:59.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:23:59.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:23:59.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:23:59.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:23:59.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:23:59.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:23:59.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:23:59.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:23:59.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:23:59.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:23:59.764 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:23:59.764 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:23:59.764 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:23:59.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:23:59.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:23:59.787 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:23:59.787 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:23:59.921 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:23:59.921 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:23:59.921 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:59.921 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:59.921 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:59.921 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:23:59.921 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:59.921 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:59.921 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:59.921 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:23:59.922 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:23:59.922 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:23:59.922 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:23:59.923 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:59.924 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:23:59.924 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:59.976 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 21 2024-09-11T06:23:59.976 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:23:59.976 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:23:59.976 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:23:59.977 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=86, tag_tid=17, entry_tid=11398], [object_number=85, tag_tid=17, entry_tid=11397], [object_number=84, tag_tid=17, entry_tid=11396], [object_number=87, tag_tid=17, entry_tid=11395]]], state=connected] 2024-09-11T06:23:59.979 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:59.981 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:23:59.982 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:23:59.982 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:59.982 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:59.982 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:59.983 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:23:59.983 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:23:59.983 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:23:59.983 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:23:59.983 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:23:59.983 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:23:59.983 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:23:59.983 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:23:59.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:23:59.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:23:59.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:00.032 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 21 2024-09-11T06:24:00.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:00.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:00.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:00.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=86, tag_tid=17, entry_tid=11398], [object_number=85, tag_tid=17, entry_tid=11397], [object_number=84, tag_tid=17, entry_tid=11396], [object_number=87, tag_tid=17, entry_tid=11395]]], state=connected] 2024-09-11T06:24:00.036 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:00.038 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=86, tag_tid=17, entry_tid=11398' 2024-09-11T06:24:00.038 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=86, tag_tid=17, entry_tid=11398' 2024-09-11T06:24:00.038 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=86, tag_tid=17, entry_tid=11398' '!=' 'object_number=87, tag_tid=17, entry_tid=11315' 2024-09-11T06:24:00.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:24:00.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=86, tag_tid=17, entry_tid=11398' = 'object_number=87, tag_tid=17, entry_tid=11315' 2024-09-11T06:24:00.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=86, tag_tid=17, entry_tid=11398' 2024-09-11T06:24:00.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:00.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:00.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:00.040 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:24:00.040 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=86, tag_tid=17, entry_tid=11398' 2024-09-11T06:24:00.040 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:00.040 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:00.041 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:24:00.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:00.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:00.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:00.043 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:24:00.043 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=86, tag_tid=17, entry_tid=11398' 2024-09-11T06:24:00.043 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:00.043 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:00.045 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=11398 2024-09-11T06:24:00.045 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 11398 2024-09-11T06:24:00.045 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:24:00.045 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:24:00.045 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:24:00.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:24:00.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:00.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:24:00.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:24:00.248 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:24:00.248 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:24:00.248 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:24:00.248 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:24:00.248 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:24:00.248 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:00.248 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:00.248 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:24:00.248 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:24:00.248 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:24:00.248 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:24:00.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:24:00.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:24:00.252 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:24:00.252 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:24:00.252 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:24:00.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:00.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:24:00.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:24:00.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:24:00.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:24:00.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:24:00.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:24:00.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:24:00.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:00.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:24:00.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:00.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:24:00.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:00.254 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:24:00.254 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:24:00.254 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:24:00.254 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:24:00.254 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:24:00.254 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:24:00.254 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:24:00.254 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:24:00.254 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:00.254 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:00.254 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:00.254 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:24:00.254 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:24:00.255 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:24:00.255 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:24:00.255 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:24:00.277 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:00.277 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:00.278 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:00.278 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:24:00.610 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:24:00.611 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:24:00.611 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:00.611 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:00.611 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:00.612 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:24:00.612 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:00.612 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:00.612 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:00.612 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:24:00.612 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:00.613 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:00.613 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:00.615 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:00.615 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:00.615 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:00.672 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 21 2024-09-11T06:24:00.672 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:00.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:00.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:00.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=84, tag_tid=17, entry_tid=11500], [object_number=87, tag_tid=17, entry_tid=11499], [object_number=86, tag_tid=17, entry_tid=11498], [object_number=85, tag_tid=17, entry_tid=11497]]], state=connected] 2024-09-11T06:24:00.676 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:00.677 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:00.678 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:24:00.679 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:00.679 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:00.679 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:00.679 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:24:00.679 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:00.679 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:00.679 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:00.679 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:24:00.680 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:00.680 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:00.680 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:00.681 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:00.681 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:00.681 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:00.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 21 2024-09-11T06:24:00.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:00.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:00.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:00.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=84, tag_tid=17, entry_tid=11500], [object_number=87, tag_tid=17, entry_tid=11499], [object_number=86, tag_tid=17, entry_tid=11498], [object_number=85, tag_tid=17, entry_tid=11497]]], state=connected] 2024-09-11T06:24:00.731 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:00.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=84, tag_tid=17, entry_tid=11500' 2024-09-11T06:24:00.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=84, tag_tid=17, entry_tid=11500' 2024-09-11T06:24:00.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=84, tag_tid=17, entry_tid=11500' '!=' 'object_number=86, tag_tid=17, entry_tid=11398' 2024-09-11T06:24:00.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:24:00.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=84, tag_tid=17, entry_tid=11500' = 'object_number=86, tag_tid=17, entry_tid=11398' 2024-09-11T06:24:00.734 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=84, tag_tid=17, entry_tid=11500' 2024-09-11T06:24:00.734 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:00.734 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:00.734 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:00.735 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:24:00.736 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=84, tag_tid=17, entry_tid=11500' 2024-09-11T06:24:00.736 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:00.736 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:00.737 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:24:00.741 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:00.741 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:00.741 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:00.741 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:24:00.741 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=84, tag_tid=17, entry_tid=11500' 2024-09-11T06:24:00.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:00.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:00.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=11500 2024-09-11T06:24:00.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 11500 2024-09-11T06:24:00.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:24:00.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:24:00.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:24:00.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:24:00.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:00.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:24:00.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:24:00.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:24:00.946 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:24:00.946 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:24:00.946 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:24:00.946 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:24:00.946 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:00.946 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:00.946 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:24:00.946 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:24:00.946 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:24:00.946 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:24:00.946 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:24:00.947 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:24:00.949 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:24:00.949 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:24:00.949 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:24:00.949 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:00.949 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:24:00.949 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:24:00.949 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:24:00.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:24:00.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:24:00.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:24:00.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:24:00.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:00.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:24:00.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:00.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:24:00.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:00.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:24:00.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:24:00.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:24:00.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:24:00.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:24:00.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:24:00.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:24:00.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:24:00.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:00.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:00.952 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:00.952 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:24:00.952 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:24:00.952 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:24:00.952 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:24:00.952 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:24:00.972 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:00.973 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:00.973 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:00.973 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:24:01.098 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:24:01.099 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:24:01.099 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:01.099 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:01.099 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:01.099 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:24:01.099 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:01.100 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:01.100 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:01.100 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:24:01.100 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:01.100 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:01.100 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:01.103 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:01.103 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:01.103 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:01.155 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 21 2024-09-11T06:24:01.155 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:01.155 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:01.155 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:01.155 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=86, tag_tid=17, entry_tid=11602], [object_number=85, tag_tid=17, entry_tid=11601], [object_number=84, tag_tid=17, entry_tid=11600], [object_number=87, tag_tid=17, entry_tid=11599]]], state=connected] 2024-09-11T06:24:01.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:01.160 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:01.161 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:24:01.161 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:01.161 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:01.161 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:01.161 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:24:01.161 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:01.161 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:01.161 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:01.161 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:24:01.162 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:01.162 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:01.162 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:01.162 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:01.162 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:01.162 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:01.210 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 21 2024-09-11T06:24:01.210 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:01.210 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:01.210 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:01.210 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=86, tag_tid=17, entry_tid=11602], [object_number=85, tag_tid=17, entry_tid=11601], [object_number=84, tag_tid=17, entry_tid=11600], [object_number=87, tag_tid=17, entry_tid=11599]]], state=connected] 2024-09-11T06:24:01.213 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:01.215 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=86, tag_tid=17, entry_tid=11602' 2024-09-11T06:24:01.215 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=86, tag_tid=17, entry_tid=11602' 2024-09-11T06:24:01.215 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=86, tag_tid=17, entry_tid=11602' '!=' 'object_number=84, tag_tid=17, entry_tid=11500' 2024-09-11T06:24:01.215 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:24:01.215 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=86, tag_tid=17, entry_tid=11602' = 'object_number=84, tag_tid=17, entry_tid=11500' 2024-09-11T06:24:01.215 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=86, tag_tid=17, entry_tid=11602' 2024-09-11T06:24:01.216 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:01.216 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:01.216 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:01.216 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:24:01.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=86, tag_tid=17, entry_tid=11602' 2024-09-11T06:24:01.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:01.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:01.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:24:01.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:01.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:01.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:01.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:24:01.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=86, tag_tid=17, entry_tid=11602' 2024-09-11T06:24:01.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:01.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:01.222 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=11602 2024-09-11T06:24:01.222 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 11602 2024-09-11T06:24:01.222 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:24:01.222 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:24:01.222 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:24:01.424 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:24:01.424 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:01.424 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:24:01.424 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:24:01.424 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:24:01.424 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:24:01.424 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:24:01.424 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:24:01.424 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:24:01.424 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:01.424 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:01.425 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:24:01.425 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:24:01.425 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:24:01.425 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:24:01.425 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:24:01.425 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:24:01.428 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:24:01.428 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:24:01.428 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:24:01.428 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:01.428 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:24:01.428 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:24:01.428 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:24:01.428 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:24:01.429 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:24:01.429 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:24:01.429 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:24:01.429 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:01.429 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:24:01.429 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:01.429 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:24:01.429 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:01.430 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:24:01.430 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:24:01.431 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:24:01.431 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:24:01.431 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:24:01.431 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:24:01.431 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:24:01.431 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:24:01.431 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:01.431 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:01.431 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:01.431 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:24:01.431 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:24:01.431 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:24:01.432 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:24:01.432 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:24:01.457 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:01.457 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:01.457 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:01.457 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:24:01.588 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:24:01.590 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:24:01.590 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:01.590 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:01.590 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:01.591 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:24:01.591 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:01.591 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:01.591 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:01.591 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:24:01.591 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:01.591 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:01.591 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:01.593 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:01.593 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:01.593 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:01.645 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 21 2024-09-11T06:24:01.645 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:01.645 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:01.645 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:01.645 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=87, tag_tid=17, entry_tid=11707], [object_number=86, tag_tid=17, entry_tid=11706], [object_number=85, tag_tid=17, entry_tid=11705], [object_number=84, tag_tid=17, entry_tid=11704]]], state=connected] 2024-09-11T06:24:01.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:01.650 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:01.651 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:24:01.651 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:01.651 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:01.651 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:01.651 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:24:01.651 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:01.651 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:01.651 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:01.651 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:24:01.651 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:01.651 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:01.652 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:01.652 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:01.652 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:01.652 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:01.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 21 2024-09-11T06:24:01.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:01.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:01.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:01.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=87, tag_tid=17, entry_tid=11707], [object_number=86, tag_tid=17, entry_tid=11706], [object_number=85, tag_tid=17, entry_tid=11705], [object_number=84, tag_tid=17, entry_tid=11704]]], state=connected] 2024-09-11T06:24:01.702 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:01.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=87, tag_tid=17, entry_tid=11707' 2024-09-11T06:24:01.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=87, tag_tid=17, entry_tid=11707' 2024-09-11T06:24:01.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=87, tag_tid=17, entry_tid=11707' '!=' 'object_number=86, tag_tid=17, entry_tid=11602' 2024-09-11T06:24:01.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:24:01.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=87, tag_tid=17, entry_tid=11707' = 'object_number=86, tag_tid=17, entry_tid=11602' 2024-09-11T06:24:01.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=87, tag_tid=17, entry_tid=11707' 2024-09-11T06:24:01.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:01.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:01.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:01.706 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:24:01.706 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=87, tag_tid=17, entry_tid=11707' 2024-09-11T06:24:01.706 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:01.707 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:01.708 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:24:01.708 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:01.708 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:01.708 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:01.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:24:01.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=87, tag_tid=17, entry_tid=11707' 2024-09-11T06:24:01.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:01.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:01.711 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=11707 2024-09-11T06:24:01.711 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 11707 2024-09-11T06:24:01.711 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:24:01.711 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:24:01.711 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:24:01.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:24:01.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:01.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:24:01.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:24:01.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:24:01.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:24:01.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:24:01.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:24:01.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:24:01.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:01.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:01.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:24:01.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:24:01.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:24:01.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:24:01.915 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:24:01.915 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:24:01.918 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:24:01.918 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:24:01.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:24:01.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:01.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:24:01.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:24:01.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:24:01.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:24:01.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:24:01.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:24:01.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:24:01.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:01.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:24:01.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:01.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:24:01.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:01.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:24:01.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:24:01.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:24:01.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:24:01.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:24:01.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:24:01.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:24:01.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:24:01.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:01.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:01.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:01.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:24:01.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:24:01.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:24:01.921 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:24:01.921 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:24:01.943 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:01.944 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:01.944 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:01.944 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:24:02.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:24:02.071 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:24:02.071 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:02.071 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:02.071 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:02.071 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:24:02.072 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:02.072 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:02.072 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:02.072 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:24:02.072 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:02.072 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:02.072 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:02.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:02.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:02.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:02.122 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 21 2024-09-11T06:24:02.122 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:02.122 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:02.122 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:02.122 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=86, tag_tid=17, entry_tid=11806], [object_number=85, tag_tid=17, entry_tid=11805], [object_number=84, tag_tid=17, entry_tid=11804], [object_number=87, tag_tid=17, entry_tid=11803]]], state=connected] 2024-09-11T06:24:02.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:02.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:02.128 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:24:02.128 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:02.128 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:02.128 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:02.128 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:24:02.128 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:02.128 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:02.128 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:02.129 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:24:02.129 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:02.129 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:02.129 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:02.129 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:02.129 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:02.130 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:02.178 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 21 2024-09-11T06:24:02.178 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:02.179 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:02.179 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:02.179 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=86, tag_tid=17, entry_tid=11806], [object_number=85, tag_tid=17, entry_tid=11805], [object_number=84, tag_tid=17, entry_tid=11804], [object_number=87, tag_tid=17, entry_tid=11803]]], state=connected] 2024-09-11T06:24:02.182 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:02.183 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=86, tag_tid=17, entry_tid=11806' 2024-09-11T06:24:02.183 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=86, tag_tid=17, entry_tid=11806' 2024-09-11T06:24:02.184 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=86, tag_tid=17, entry_tid=11806' '!=' 'object_number=87, tag_tid=17, entry_tid=11707' 2024-09-11T06:24:02.184 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:24:02.184 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=86, tag_tid=17, entry_tid=11806' = 'object_number=87, tag_tid=17, entry_tid=11707' 2024-09-11T06:24:02.184 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=86, tag_tid=17, entry_tid=11806' 2024-09-11T06:24:02.184 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:02.184 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:02.185 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:02.185 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:24:02.186 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=86, tag_tid=17, entry_tid=11806' 2024-09-11T06:24:02.186 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:02.186 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:02.187 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:24:02.188 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:02.188 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:02.188 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:02.188 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:24:02.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=86, tag_tid=17, entry_tid=11806' 2024-09-11T06:24:02.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:02.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:02.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=11806 2024-09-11T06:24:02.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 11806 2024-09-11T06:24:02.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:24:02.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:24:02.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:24:02.392 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:24:02.393 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:02.393 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:24:02.393 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:24:02.393 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:24:02.393 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:24:02.393 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:24:02.393 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:24:02.393 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:24:02.393 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:02.393 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:02.393 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:24:02.393 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:24:02.394 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:24:02.394 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:24:02.394 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:24:02.394 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:24:02.398 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:24:02.398 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:24:02.398 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:24:02.398 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:02.398 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:24:02.398 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:24:02.398 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:24:02.398 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:24:02.398 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:24:02.398 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:24:02.399 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:24:02.399 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:02.399 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:24:02.399 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:02.399 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:24:02.399 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:02.399 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:24:02.399 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:24:02.399 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:24:02.399 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:24:02.399 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:24:02.399 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:24:02.399 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:24:02.399 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:24:02.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:02.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:02.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:02.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:24:02.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:24:02.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:24:02.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:24:02.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:24:02.425 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:02.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:02.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:02.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:24:02.553 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:24:02.555 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:24:02.555 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:02.555 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:02.555 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:02.555 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:24:02.555 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:02.555 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:02.555 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:02.556 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:24:02.556 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:02.556 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:02.556 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:02.557 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:02.558 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:02.558 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:02.609 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 21 2024-09-11T06:24:02.610 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:02.610 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:02.610 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:02.610 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=84, tag_tid=17, entry_tid=11912], [object_number=87, tag_tid=17, entry_tid=11911], [object_number=86, tag_tid=17, entry_tid=11910], [object_number=85, tag_tid=17, entry_tid=11909]]], state=connected] 2024-09-11T06:24:02.613 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:02.614 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:02.616 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:24:02.616 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:02.616 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:02.616 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:02.616 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:24:02.616 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:02.616 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:02.616 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:02.616 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:24:02.616 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:02.616 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:02.617 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:02.617 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:02.617 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:02.617 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:02.665 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 21 2024-09-11T06:24:02.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:02.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:02.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:02.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=84, tag_tid=17, entry_tid=11912], [object_number=87, tag_tid=17, entry_tid=11911], [object_number=86, tag_tid=17, entry_tid=11910], [object_number=85, tag_tid=17, entry_tid=11909]]], state=connected] 2024-09-11T06:24:02.669 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:02.671 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=84, tag_tid=17, entry_tid=11912' 2024-09-11T06:24:02.671 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=84, tag_tid=17, entry_tid=11912' 2024-09-11T06:24:02.671 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=84, tag_tid=17, entry_tid=11912' '!=' 'object_number=86, tag_tid=17, entry_tid=11806' 2024-09-11T06:24:02.671 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:24:02.671 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=84, tag_tid=17, entry_tid=11912' = 'object_number=86, tag_tid=17, entry_tid=11806' 2024-09-11T06:24:02.672 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=84, tag_tid=17, entry_tid=11912' 2024-09-11T06:24:02.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:02.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:02.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:02.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:24:02.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=84, tag_tid=17, entry_tid=11912' 2024-09-11T06:24:02.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:02.674 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:02.674 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:24:02.675 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:02.675 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:02.675 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:02.676 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:24:02.676 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=84, tag_tid=17, entry_tid=11912' 2024-09-11T06:24:02.677 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:02.677 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:02.678 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=11912 2024-09-11T06:24:02.678 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 11912 2024-09-11T06:24:02.678 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:24:02.678 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:24:02.678 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:24:02.880 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:24:02.880 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:02.880 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:24:02.880 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:24:02.880 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:24:02.880 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:24:02.880 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:24:02.880 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:24:02.880 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:24:02.880 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:02.880 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:02.881 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:24:02.881 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:24:02.881 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:24:02.881 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:24:02.881 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:24:02.881 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:24:02.884 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:24:02.884 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:24:02.884 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:24:02.884 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:02.884 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:24:02.884 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:24:02.884 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:24:02.884 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:24:02.884 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:24:02.884 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:24:02.884 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:24:02.884 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:02.885 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:24:02.885 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:02.885 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:24:02.885 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:02.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:24:02.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:24:02.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:24:02.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:24:02.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:24:02.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:24:02.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:24:02.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:24:02.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:02.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:02.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:02.887 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:24:02.887 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:24:02.887 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:24:02.887 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:24:02.887 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:24:02.910 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:02.911 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:02.911 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:02.911 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:24:03.041 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:24:03.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:24:03.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:03.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:03.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:03.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:24:03.043 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:03.043 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:03.043 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:03.043 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:24:03.043 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:03.043 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:03.043 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:03.045 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:03.046 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:03.046 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:03.092 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 21 2024-09-11T06:24:03.092 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:03.092 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:03.092 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:03.092 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=84, tag_tid=17, entry_tid=12016], [object_number=87, tag_tid=17, entry_tid=12015], [object_number=86, tag_tid=17, entry_tid=12014], [object_number=85, tag_tid=17, entry_tid=12013]]], state=connected] 2024-09-11T06:24:03.095 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:03.096 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:03.097 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:24:03.097 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:03.097 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:03.098 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:03.098 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:24:03.098 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:03.098 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:03.098 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:03.098 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:24:03.099 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:03.099 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:03.099 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:03.099 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:03.099 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:03.099 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:03.145 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 21 2024-09-11T06:24:03.145 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:03.145 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:03.145 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:03.145 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=84, tag_tid=17, entry_tid=12016], [object_number=87, tag_tid=17, entry_tid=12015], [object_number=86, tag_tid=17, entry_tid=12014], [object_number=85, tag_tid=17, entry_tid=12013]]], state=connected] 2024-09-11T06:24:03.148 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:03.150 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=84, tag_tid=17, entry_tid=12016' 2024-09-11T06:24:03.151 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=84, tag_tid=17, entry_tid=12016' 2024-09-11T06:24:03.151 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=84, tag_tid=17, entry_tid=12016' '!=' 'object_number=84, tag_tid=17, entry_tid=11912' 2024-09-11T06:24:03.151 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:24:03.151 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=84, tag_tid=17, entry_tid=12016' = 'object_number=84, tag_tid=17, entry_tid=11912' 2024-09-11T06:24:03.151 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=84, tag_tid=17, entry_tid=12016' 2024-09-11T06:24:03.151 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:03.151 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:03.151 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:03.151 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:24:03.152 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=84, tag_tid=17, entry_tid=12016' 2024-09-11T06:24:03.152 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:03.152 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:03.153 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:24:03.154 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:03.154 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:03.154 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:03.155 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:24:03.155 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=84, tag_tid=17, entry_tid=12016' 2024-09-11T06:24:03.156 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:03.156 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:03.157 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=12016 2024-09-11T06:24:03.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 12016 2024-09-11T06:24:03.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:24:03.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:24:03.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:24:03.359 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:24:03.359 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:03.359 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:24:03.359 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:24:03.359 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:24:03.359 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:24:03.359 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:24:03.359 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:24:03.359 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:24:03.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:03.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:03.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:24:03.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:24:03.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:24:03.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:24:03.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:24:03.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:24:03.364 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:24:03.365 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:24:03.365 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:24:03.365 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:03.365 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:24:03.365 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:24:03.365 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:24:03.365 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:24:03.365 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:24:03.365 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:24:03.365 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:24:03.365 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:03.365 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:24:03.365 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:03.366 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:24:03.366 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:03.366 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:24:03.366 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:24:03.366 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:24:03.366 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:24:03.366 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:24:03.366 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:24:03.366 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:24:03.366 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:24:03.366 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:03.366 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:03.366 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:03.366 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:24:03.367 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:24:03.367 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:24:03.367 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:24:03.367 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:24:03.389 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:03.390 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:03.390 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:03.390 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:24:03.519 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:24:03.520 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:24:03.521 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:03.521 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:03.521 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:03.521 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:24:03.521 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:03.521 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:03.521 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:03.521 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:24:03.521 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:03.522 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:03.522 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:03.524 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:03.524 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:03.525 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:03.578 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 21 2024-09-11T06:24:03.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:03.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:03.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:03.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=84, tag_tid=17, entry_tid=12108], [object_number=87, tag_tid=17, entry_tid=12107], [object_number=86, tag_tid=17, entry_tid=12106], [object_number=85, tag_tid=17, entry_tid=12105]]], state=connected] 2024-09-11T06:24:03.582 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:03.583 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:03.584 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:24:03.584 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:03.584 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:03.584 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:03.585 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:24:03.585 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:03.585 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:03.585 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:03.585 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:24:03.585 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:03.585 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:03.585 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:03.586 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:03.586 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:03.586 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:03.632 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 21 2024-09-11T06:24:03.633 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:03.633 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:03.633 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:03.633 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=84, tag_tid=17, entry_tid=12108], [object_number=87, tag_tid=17, entry_tid=12107], [object_number=86, tag_tid=17, entry_tid=12106], [object_number=85, tag_tid=17, entry_tid=12105]]], state=connected] 2024-09-11T06:24:03.636 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:03.638 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=84, tag_tid=17, entry_tid=12108' 2024-09-11T06:24:03.638 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=84, tag_tid=17, entry_tid=12108' 2024-09-11T06:24:03.638 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=84, tag_tid=17, entry_tid=12108' '!=' 'object_number=84, tag_tid=17, entry_tid=12016' 2024-09-11T06:24:03.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:24:03.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=84, tag_tid=17, entry_tid=12108' = 'object_number=84, tag_tid=17, entry_tid=12016' 2024-09-11T06:24:03.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=84, tag_tid=17, entry_tid=12108' 2024-09-11T06:24:03.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:03.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:03.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:03.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:24:03.640 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=84, tag_tid=17, entry_tid=12108' 2024-09-11T06:24:03.640 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:03.640 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:03.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:24:03.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:03.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:03.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:03.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:24:03.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=84, tag_tid=17, entry_tid=12108' 2024-09-11T06:24:03.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:03.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:03.645 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=12108 2024-09-11T06:24:03.645 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 12108 2024-09-11T06:24:03.645 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:24:03.645 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:24:03.645 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:24:03.846 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:24:03.846 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:03.846 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:24:03.846 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:24:03.846 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:24:03.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:24:03.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:24:03.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:24:03.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:24:03.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:03.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:03.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:24:03.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:24:03.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:24:03.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:24:03.848 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:24:03.848 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:24:03.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:24:03.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:24:03.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:24:03.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:03.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:24:03.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:24:03.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:24:03.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:24:03.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:24:03.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:24:03.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:24:03.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:03.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:24:03.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:03.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:24:03.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:03.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:24:03.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:24:03.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:24:03.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:24:03.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:24:03.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:24:03.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:24:03.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:24:03.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:03.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:03.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:03.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:24:03.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:24:03.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:24:03.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:24:03.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:24:03.876 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:03.877 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:03.877 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:03.877 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:24:04.009 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:24:04.010 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:24:04.010 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:04.010 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:04.010 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:04.010 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:24:04.012 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:04.012 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:04.012 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:04.013 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:24:04.013 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:04.013 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:04.013 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:04.013 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:04.014 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:04.014 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:04.065 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 21 2024-09-11T06:24:04.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:04.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:04.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:04.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=87, tag_tid=17, entry_tid=12207], [object_number=86, tag_tid=17, entry_tid=12206], [object_number=85, tag_tid=17, entry_tid=12205], [object_number=84, tag_tid=17, entry_tid=12204]]], state=connected] 2024-09-11T06:24:04.069 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:04.072 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:04.072 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:24:04.072 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:04.072 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:04.072 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:04.072 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:24:04.072 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:04.072 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:04.072 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:04.072 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:24:04.073 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:04.073 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:04.073 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:04.073 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:04.073 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:04.073 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:04.121 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 21 2024-09-11T06:24:04.121 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:04.121 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:04.121 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:04.121 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=87, tag_tid=17, entry_tid=12207], [object_number=86, tag_tid=17, entry_tid=12206], [object_number=85, tag_tid=17, entry_tid=12205], [object_number=84, tag_tid=17, entry_tid=12204]]], state=connected] 2024-09-11T06:24:04.124 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:04.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=87, tag_tid=17, entry_tid=12207' 2024-09-11T06:24:04.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=87, tag_tid=17, entry_tid=12207' 2024-09-11T06:24:04.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=87, tag_tid=17, entry_tid=12207' '!=' 'object_number=84, tag_tid=17, entry_tid=12108' 2024-09-11T06:24:04.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:24:04.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=87, tag_tid=17, entry_tid=12207' = 'object_number=84, tag_tid=17, entry_tid=12108' 2024-09-11T06:24:04.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=87, tag_tid=17, entry_tid=12207' 2024-09-11T06:24:04.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:04.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:04.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:04.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:24:04.128 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=87, tag_tid=17, entry_tid=12207' 2024-09-11T06:24:04.128 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:04.128 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:04.129 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:24:04.130 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:04.130 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:04.130 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:04.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:24:04.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=87, tag_tid=17, entry_tid=12207' 2024-09-11T06:24:04.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:04.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:04.133 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=12207 2024-09-11T06:24:04.133 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 12207 2024-09-11T06:24:04.133 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:24:04.133 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:24:04.133 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:24:04.335 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:24:04.335 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:04.335 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:24:04.335 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:24:04.335 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:24:04.335 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:24:04.335 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:24:04.335 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:24:04.335 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:24:04.335 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:04.335 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:04.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:24:04.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:24:04.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:24:04.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:24:04.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:24:04.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:24:04.340 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:24:04.340 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:24:04.340 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:24:04.340 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:04.340 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:24:04.340 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:24:04.340 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:24:04.340 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:24:04.340 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:24:04.340 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:24:04.340 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:24:04.340 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:04.341 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:24:04.341 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:04.341 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:24:04.341 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:04.341 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:24:04.341 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:24:04.341 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:24:04.341 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:24:04.341 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:24:04.341 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:24:04.341 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:24:04.341 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:24:04.341 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:04.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:04.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:04.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:24:04.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:24:04.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:24:04.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:24:04.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:24:04.366 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:04.366 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:04.366 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:04.366 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:24:04.503 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:24:04.504 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:24:04.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:04.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:04.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:04.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:24:04.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:04.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:04.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:04.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:24:04.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:04.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:04.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:04.507 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:04.508 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:04.508 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:04.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 21 2024-09-11T06:24:04.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:04.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:04.555 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:04.555 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=86, tag_tid=17, entry_tid=12302], [object_number=85, tag_tid=17, entry_tid=12301], [object_number=84, tag_tid=17, entry_tid=12300], [object_number=87, tag_tid=17, entry_tid=12299]]], state=connected] 2024-09-11T06:24:04.557 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:04.559 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:04.560 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:24:04.560 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:04.560 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:04.560 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:04.560 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:24:04.560 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:04.560 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:04.561 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:04.561 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:24:04.561 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:04.561 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:04.561 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:04.561 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:04.562 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:04.562 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:04.609 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 21 2024-09-11T06:24:04.609 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:04.609 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:04.609 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:04.609 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=86, tag_tid=17, entry_tid=12302], [object_number=85, tag_tid=17, entry_tid=12301], [object_number=84, tag_tid=17, entry_tid=12300], [object_number=87, tag_tid=17, entry_tid=12299]]], state=connected] 2024-09-11T06:24:04.612 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:04.615 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=86, tag_tid=17, entry_tid=12302' 2024-09-11T06:24:04.615 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=86, tag_tid=17, entry_tid=12302' 2024-09-11T06:24:04.616 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=86, tag_tid=17, entry_tid=12302' '!=' 'object_number=87, tag_tid=17, entry_tid=12207' 2024-09-11T06:24:04.616 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:24:04.616 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=86, tag_tid=17, entry_tid=12302' = 'object_number=87, tag_tid=17, entry_tid=12207' 2024-09-11T06:24:04.616 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=86, tag_tid=17, entry_tid=12302' 2024-09-11T06:24:04.616 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:04.616 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:04.617 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:04.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:24:04.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=86, tag_tid=17, entry_tid=12302' 2024-09-11T06:24:04.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:04.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:04.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:24:04.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:04.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:04.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:04.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:24:04.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=86, tag_tid=17, entry_tid=12302' 2024-09-11T06:24:04.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:04.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:04.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=12302 2024-09-11T06:24:04.623 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 12302 2024-09-11T06:24:04.623 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:24:04.623 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:24:04.623 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:24:04.823 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:24:04.823 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:04.823 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:24:04.823 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:24:04.823 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:24:04.823 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:24:04.823 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:24:04.823 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:24:04.824 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:24:04.824 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:04.824 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:04.824 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:24:04.824 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:24:04.824 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:24:04.824 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:24:04.825 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:24:04.825 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:24:04.829 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:24:04.829 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:24:04.830 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:24:04.830 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:04.830 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:24:04.830 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:24:04.830 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:24:04.830 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:24:04.830 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:24:04.830 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:24:04.830 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:24:04.830 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:04.830 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:24:04.830 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:04.831 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:24:04.831 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:04.831 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:24:04.831 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:24:04.831 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:24:04.831 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:24:04.831 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:24:04.831 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:24:04.831 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:24:04.831 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:24:04.831 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:04.831 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:04.831 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:04.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:24:04.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:24:04.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:24:04.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:24:04.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:24:04.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:04.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:04.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:04.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:24:04.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:24:04.980 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:24:04.980 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:04.980 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:04.980 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:04.980 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:24:04.980 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:04.980 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:04.980 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:04.980 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:24:04.980 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:04.981 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:04.981 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:04.982 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:04.983 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:04.983 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:05.032 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 21 2024-09-11T06:24:05.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:05.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:05.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:05.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=85, tag_tid=17, entry_tid=12393], [object_number=84, tag_tid=17, entry_tid=12392], [object_number=87, tag_tid=17, entry_tid=12391], [object_number=86, tag_tid=17, entry_tid=12390]]], state=connected] 2024-09-11T06:24:05.036 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:05.037 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:05.038 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:24:05.038 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:05.038 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:05.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:05.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:24:05.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:05.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:05.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:05.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:24:05.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:05.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:05.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:05.040 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:05.040 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:05.040 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:05.084 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 21 2024-09-11T06:24:05.084 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:05.085 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:05.085 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:05.085 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=85, tag_tid=17, entry_tid=12393], [object_number=84, tag_tid=17, entry_tid=12392], [object_number=87, tag_tid=17, entry_tid=12391], [object_number=86, tag_tid=17, entry_tid=12390]]], state=connected] 2024-09-11T06:24:05.088 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:05.090 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=85, tag_tid=17, entry_tid=12393' 2024-09-11T06:24:05.090 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=85, tag_tid=17, entry_tid=12393' 2024-09-11T06:24:05.090 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=85, tag_tid=17, entry_tid=12393' '!=' 'object_number=86, tag_tid=17, entry_tid=12302' 2024-09-11T06:24:05.090 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:24:05.090 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=85, tag_tid=17, entry_tid=12393' = 'object_number=86, tag_tid=17, entry_tid=12302' 2024-09-11T06:24:05.090 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=85, tag_tid=17, entry_tid=12393' 2024-09-11T06:24:05.090 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:05.090 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:05.090 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:05.091 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:24:05.091 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=85, tag_tid=17, entry_tid=12393' 2024-09-11T06:24:05.091 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:05.092 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:05.092 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:24:05.093 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:05.093 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:05.093 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:05.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:24:05.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=85, tag_tid=17, entry_tid=12393' 2024-09-11T06:24:05.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:05.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:05.096 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=12393 2024-09-11T06:24:05.096 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 12393 2024-09-11T06:24:05.096 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:24:05.096 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:24:05.096 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:24:05.298 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:24:05.298 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:05.298 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:24:05.298 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:24:05.298 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:24:05.298 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:24:05.298 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:24:05.298 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:24:05.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:24:05.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:05.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:05.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:24:05.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:24:05.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:24:05.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:24:05.300 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:24:05.300 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:24:05.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:24:05.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:24:05.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:24:05.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:05.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:24:05.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:24:05.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:24:05.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:24:05.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:24:05.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:24:05.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:24:05.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:05.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:24:05.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:05.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:24:05.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:05.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:24:05.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:24:05.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:24:05.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:24:05.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:24:05.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:24:05.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:24:05.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:24:05.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:05.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:05.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:05.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:24:05.305 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:24:05.305 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:24:05.305 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:24:05.305 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:24:05.327 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:05.327 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:05.327 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:05.327 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:24:05.459 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:24:05.459 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:24:05.459 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:05.459 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:05.459 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:05.459 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:24:05.460 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:05.460 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:05.460 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:05.460 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:24:05.460 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:05.461 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:05.461 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:05.461 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:05.461 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:05.462 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:05.514 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 21 2024-09-11T06:24:05.514 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:05.514 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:05.514 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:05.514 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=84, tag_tid=17, entry_tid=12476], [object_number=87, tag_tid=17, entry_tid=12475], [object_number=86, tag_tid=17, entry_tid=12474], [object_number=85, tag_tid=17, entry_tid=12473]]], state=connected] 2024-09-11T06:24:05.517 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:05.519 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:05.520 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:24:05.520 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:05.520 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:05.520 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:05.520 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:24:05.520 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:05.520 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:05.520 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:05.520 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:24:05.520 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:05.520 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:05.520 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:05.521 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:05.521 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:05.521 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:05.565 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 21 2024-09-11T06:24:05.565 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:05.565 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:05.565 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:05.565 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=84, tag_tid=17, entry_tid=12476], [object_number=87, tag_tid=17, entry_tid=12475], [object_number=86, tag_tid=17, entry_tid=12474], [object_number=85, tag_tid=17, entry_tid=12473]]], state=connected] 2024-09-11T06:24:05.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:05.570 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=84, tag_tid=17, entry_tid=12476' 2024-09-11T06:24:05.570 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=84, tag_tid=17, entry_tid=12476' 2024-09-11T06:24:05.570 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=84, tag_tid=17, entry_tid=12476' '!=' 'object_number=85, tag_tid=17, entry_tid=12393' 2024-09-11T06:24:05.570 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:24:05.570 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=84, tag_tid=17, entry_tid=12476' = 'object_number=85, tag_tid=17, entry_tid=12393' 2024-09-11T06:24:05.570 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=84, tag_tid=17, entry_tid=12476' 2024-09-11T06:24:05.570 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:05.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:05.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:05.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:24:05.572 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=84, tag_tid=17, entry_tid=12476' 2024-09-11T06:24:05.572 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:05.572 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:05.573 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:24:05.573 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:05.573 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:05.573 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:05.574 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:24:05.575 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=84, tag_tid=17, entry_tid=12476' 2024-09-11T06:24:05.575 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:05.575 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:05.576 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=12476 2024-09-11T06:24:05.576 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 12476 2024-09-11T06:24:05.576 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:24:05.576 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:24:05.576 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:24:05.778 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:24:05.778 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:05.778 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:24:05.778 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:24:05.778 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:24:05.778 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:24:05.778 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:24:05.778 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:24:05.778 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:24:05.778 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:05.779 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:05.779 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:24:05.779 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:24:05.779 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:24:05.779 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:24:05.779 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:24:05.779 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:24:05.781 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:24:05.781 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:24:05.781 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:24:05.781 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:05.781 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:24:05.781 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:24:05.781 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:24:05.781 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:24:05.781 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:24:05.781 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:24:05.782 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:24:05.782 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:05.782 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:24:05.782 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:05.782 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:24:05.782 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:05.783 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:24:05.783 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:24:05.783 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:24:05.783 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:24:05.783 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:24:05.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:24:05.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:24:05.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:24:05.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:05.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:05.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:05.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:24:05.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:24:05.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:24:05.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:24:05.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:24:05.807 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:05.807 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:05.808 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:05.808 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:24:05.938 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:24:05.939 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:24:05.939 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:05.939 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:05.939 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:05.939 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:24:05.939 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:05.939 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:05.939 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:05.939 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:24:05.940 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:05.940 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:05.940 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:05.942 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:05.942 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:05.942 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:05.989 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 21 2024-09-11T06:24:05.989 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:05.989 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:05.989 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:05.989 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=85, tag_tid=17, entry_tid=12553], [object_number=84, tag_tid=17, entry_tid=12552], [object_number=87, tag_tid=17, entry_tid=12551], [object_number=86, tag_tid=17, entry_tid=12550]]], state=connected] 2024-09-11T06:24:05.992 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:05.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:05.994 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:24:05.994 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:05.994 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:05.994 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:05.994 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:24:05.995 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:05.995 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:05.995 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:05.995 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:24:05.995 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:05.995 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:05.995 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:05.996 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:05.996 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:05.996 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:06.043 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 21 2024-09-11T06:24:06.043 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:06.043 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:06.043 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:06.043 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=85, tag_tid=17, entry_tid=12553], [object_number=84, tag_tid=17, entry_tid=12552], [object_number=87, tag_tid=17, entry_tid=12551], [object_number=86, tag_tid=17, entry_tid=12550]]], state=connected] 2024-09-11T06:24:06.046 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:06.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=85, tag_tid=17, entry_tid=12553' 2024-09-11T06:24:06.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=85, tag_tid=17, entry_tid=12553' 2024-09-11T06:24:06.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=85, tag_tid=17, entry_tid=12553' '!=' 'object_number=84, tag_tid=17, entry_tid=12476' 2024-09-11T06:24:06.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:24:06.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=85, tag_tid=17, entry_tid=12553' = 'object_number=84, tag_tid=17, entry_tid=12476' 2024-09-11T06:24:06.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=85, tag_tid=17, entry_tid=12553' 2024-09-11T06:24:06.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:06.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:06.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:06.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:24:06.050 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=85, tag_tid=17, entry_tid=12553' 2024-09-11T06:24:06.050 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:06.050 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:06.051 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:24:06.051 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:06.052 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:06.052 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:06.052 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:24:06.053 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=85, tag_tid=17, entry_tid=12553' 2024-09-11T06:24:06.053 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:06.053 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:06.054 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=12553 2024-09-11T06:24:06.054 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 12553 2024-09-11T06:24:06.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:24:06.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:24:06.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:24:06.256 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:24:06.256 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:06.256 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:24:06.256 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:24:06.256 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:24:06.257 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:24:06.257 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:24:06.257 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:24:06.257 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:24:06.257 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:06.257 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:06.257 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:24:06.257 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:24:06.257 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:24:06.257 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:24:06.258 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:24:06.258 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:24:06.262 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:24:06.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:24:06.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:24:06.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:06.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:24:06.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:24:06.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:24:06.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:24:06.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:24:06.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:24:06.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:24:06.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:06.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:24:06.264 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:06.264 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:24:06.264 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:06.264 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:24:06.264 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:24:06.264 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:24:06.264 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:24:06.264 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:24:06.264 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:24:06.264 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:24:06.264 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:24:06.264 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:06.264 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:06.264 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:06.265 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:24:06.265 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:24:06.265 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:24:06.265 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:24:06.265 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:24:06.285 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:06.286 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:06.286 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:06.286 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:24:06.413 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:24:06.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:24:06.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:06.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:06.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:06.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:24:06.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:06.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:06.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:06.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:24:06.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:06.416 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:06.416 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:06.418 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:06.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:06.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:06.695 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 21 2024-09-11T06:24:06.695 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:06.695 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:06.695 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:06.695 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=85, tag_tid=17, entry_tid=12629], [object_number=84, tag_tid=17, entry_tid=12628], [object_number=87, tag_tid=17, entry_tid=12627], [object_number=86, tag_tid=17, entry_tid=12626]]], state=connected] 2024-09-11T06:24:06.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:06.701 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:06.701 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:24:06.702 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:06.702 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:06.702 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:06.702 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:24:06.702 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:06.702 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:06.702 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:06.702 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:24:06.702 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:06.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:06.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:06.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:06.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:06.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:06.756 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 21 2024-09-11T06:24:06.756 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:06.756 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:06.756 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:06.756 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=85, tag_tid=17, entry_tid=12629], [object_number=84, tag_tid=17, entry_tid=12628], [object_number=87, tag_tid=17, entry_tid=12627], [object_number=86, tag_tid=17, entry_tid=12626]]], state=connected] 2024-09-11T06:24:06.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:06.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=85, tag_tid=17, entry_tid=12629' 2024-09-11T06:24:06.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=85, tag_tid=17, entry_tid=12629' 2024-09-11T06:24:06.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=85, tag_tid=17, entry_tid=12629' '!=' 'object_number=85, tag_tid=17, entry_tid=12553' 2024-09-11T06:24:06.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:24:06.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=85, tag_tid=17, entry_tid=12629' = 'object_number=85, tag_tid=17, entry_tid=12553' 2024-09-11T06:24:06.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=85, tag_tid=17, entry_tid=12629' 2024-09-11T06:24:06.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:06.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:06.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:06.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:24:06.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=85, tag_tid=17, entry_tid=12629' 2024-09-11T06:24:06.764 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:06.764 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:06.765 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:24:06.765 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:06.765 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:06.765 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:06.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:24:06.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=85, tag_tid=17, entry_tid=12629' 2024-09-11T06:24:06.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:06.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:06.768 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=12629 2024-09-11T06:24:06.768 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 12629 2024-09-11T06:24:06.769 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:24:06.769 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:24:06.769 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:24:06.970 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:24:06.970 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:06.971 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:24:06.971 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:24:06.971 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:24:06.971 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:24:06.971 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:24:06.971 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:24:06.971 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:24:06.971 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:06.971 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:06.971 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:24:06.971 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:24:06.972 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:24:06.972 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:24:06.972 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:24:06.972 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:24:06.976 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:24:06.976 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:24:06.976 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:24:06.976 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:06.976 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:24:06.976 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:24:06.976 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:24:06.976 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:24:06.976 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:24:06.977 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:24:06.977 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:24:06.977 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:06.977 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:24:06.977 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:06.977 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:24:06.977 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:06.977 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:24:06.977 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:24:06.977 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:24:06.977 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:24:06.977 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:24:06.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:24:06.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:24:06.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:24:06.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:06.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:06.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:06.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:24:06.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:24:06.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:24:06.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:24:06.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:24:07.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:07.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:07.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:07.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:24:07.129 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:24:07.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:24:07.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:07.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:07.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:07.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:24:07.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:07.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:07.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:07.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:24:07.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:07.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:07.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:07.133 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:07.133 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:07.133 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:07.180 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 21 2024-09-11T06:24:07.180 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:07.180 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:07.180 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:07.180 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=87, tag_tid=17, entry_tid=12743], [object_number=86, tag_tid=17, entry_tid=12742], [object_number=85, tag_tid=17, entry_tid=12741], [object_number=84, tag_tid=17, entry_tid=12740]]], state=connected] 2024-09-11T06:24:07.183 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:07.185 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:07.186 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:24:07.186 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:07.186 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:07.186 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:07.186 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:24:07.186 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:07.186 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:07.186 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:07.186 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:24:07.186 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:07.187 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:07.187 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:07.187 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:07.187 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:07.187 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:07.235 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 21 2024-09-11T06:24:07.236 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:07.236 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:07.236 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:07.236 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=87, tag_tid=17, entry_tid=12743], [object_number=86, tag_tid=17, entry_tid=12742], [object_number=85, tag_tid=17, entry_tid=12741], [object_number=84, tag_tid=17, entry_tid=12740]]], state=connected] 2024-09-11T06:24:07.239 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:07.240 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=87, tag_tid=17, entry_tid=12743' 2024-09-11T06:24:07.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=87, tag_tid=17, entry_tid=12743' 2024-09-11T06:24:07.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=87, tag_tid=17, entry_tid=12743' '!=' 'object_number=85, tag_tid=17, entry_tid=12629' 2024-09-11T06:24:07.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:24:07.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=87, tag_tid=17, entry_tid=12743' = 'object_number=85, tag_tid=17, entry_tid=12629' 2024-09-11T06:24:07.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=87, tag_tid=17, entry_tid=12743' 2024-09-11T06:24:07.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:07.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:07.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:07.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:24:07.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=87, tag_tid=17, entry_tid=12743' 2024-09-11T06:24:07.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:07.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:07.244 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:24:07.245 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:07.245 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:07.245 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:07.246 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:24:07.246 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=87, tag_tid=17, entry_tid=12743' 2024-09-11T06:24:07.246 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:07.246 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:07.248 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=12743 2024-09-11T06:24:07.248 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 12743 2024-09-11T06:24:07.248 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:24:07.248 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:24:07.248 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:24:07.449 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:24:07.450 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:07.450 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:24:07.450 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:24:07.450 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:24:07.450 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:24:07.450 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:24:07.450 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:24:07.450 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:24:07.450 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:07.450 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:07.450 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:24:07.451 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:24:07.451 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:24:07.451 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:24:07.451 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:24:07.451 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:24:07.454 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:24:07.454 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:24:07.454 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:24:07.455 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:07.455 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:24:07.455 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:24:07.455 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:24:07.455 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:24:07.455 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:24:07.455 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:24:07.455 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:24:07.455 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:07.455 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:24:07.455 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:07.455 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:24:07.455 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:07.456 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:24:07.456 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:24:07.456 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:24:07.456 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:24:07.456 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:24:07.456 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:24:07.456 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:24:07.456 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:24:07.456 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:07.457 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:07.457 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:07.457 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:24:07.457 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:24:07.457 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:24:07.458 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:24:07.458 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:24:07.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:07.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:07.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:07.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:24:07.607 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:24:07.609 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:24:07.609 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:07.609 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:07.609 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:07.609 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:24:07.609 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:07.609 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:07.609 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:07.609 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:24:07.609 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:07.610 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:07.610 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:07.611 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:07.611 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:07.611 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:07.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 21 2024-09-11T06:24:07.658 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:07.658 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:07.658 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:07.658 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=87, tag_tid=17, entry_tid=12823], [object_number=86, tag_tid=17, entry_tid=12822], [object_number=85, tag_tid=17, entry_tid=12821], [object_number=84, tag_tid=17, entry_tid=12820]]], state=connected] 2024-09-11T06:24:07.662 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:07.663 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:07.664 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:24:07.664 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:07.664 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:07.664 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:07.665 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:24:07.665 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:07.665 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:07.665 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:07.665 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:24:07.665 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:07.665 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:07.665 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:07.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:07.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:07.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:07.712 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 21 2024-09-11T06:24:07.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:07.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:07.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:07.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=87, tag_tid=17, entry_tid=12823], [object_number=86, tag_tid=17, entry_tid=12822], [object_number=85, tag_tid=17, entry_tid=12821], [object_number=84, tag_tid=17, entry_tid=12820]]], state=connected] 2024-09-11T06:24:07.716 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:07.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=87, tag_tid=17, entry_tid=12823' 2024-09-11T06:24:07.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=87, tag_tid=17, entry_tid=12823' 2024-09-11T06:24:07.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=87, tag_tid=17, entry_tid=12823' '!=' 'object_number=87, tag_tid=17, entry_tid=12743' 2024-09-11T06:24:07.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:24:07.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=87, tag_tid=17, entry_tid=12823' = 'object_number=87, tag_tid=17, entry_tid=12743' 2024-09-11T06:24:07.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=87, tag_tid=17, entry_tid=12823' 2024-09-11T06:24:07.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:07.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:07.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:07.721 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:24:07.721 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=87, tag_tid=17, entry_tid=12823' 2024-09-11T06:24:07.721 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:07.721 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:07.722 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:24:07.723 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:07.723 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:07.724 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:07.724 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:24:07.724 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=87, tag_tid=17, entry_tid=12823' 2024-09-11T06:24:07.724 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:07.724 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:07.725 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=12823 2024-09-11T06:24:07.725 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 12823 2024-09-11T06:24:07.725 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:24:07.725 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:24:07.725 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:24:07.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:24:07.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:07.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:24:07.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:24:07.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:24:07.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:24:07.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:24:07.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:24:07.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:24:07.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:07.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:07.928 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:24:07.928 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:24:07.928 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:24:07.928 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:24:07.928 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:24:07.928 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:24:07.931 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:24:07.931 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:24:07.931 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:24:07.931 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:07.931 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:24:07.932 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:24:07.932 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:24:07.932 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:24:07.932 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:24:07.932 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:24:07.932 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:24:07.932 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:07.932 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:24:07.932 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:07.932 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:24:07.932 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:07.932 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:24:07.933 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:24:07.933 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:24:07.933 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:24:07.933 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:24:07.933 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:24:07.933 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:24:07.933 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:24:07.933 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:07.933 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:07.933 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:07.933 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:24:07.933 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:24:07.933 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:24:07.933 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:24:07.934 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:24:07.957 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:07.957 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:07.957 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:07.958 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:24:08.093 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:24:08.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:24:08.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:08.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:08.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:08.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:24:08.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:08.095 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:08.095 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:08.095 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:24:08.095 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:08.095 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:08.095 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:08.097 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:08.097 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:08.097 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:08.149 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 21 2024-09-11T06:24:08.149 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:08.150 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:08.150 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:08.150 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=86, tag_tid=17, entry_tid=12906], [object_number=85, tag_tid=17, entry_tid=12905], [object_number=84, tag_tid=17, entry_tid=12904], [object_number=87, tag_tid=17, entry_tid=12903]]], state=connected] 2024-09-11T06:24:08.153 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:08.154 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:08.155 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:24:08.156 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:08.156 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:08.156 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:08.156 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:24:08.156 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:08.156 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:08.156 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:08.156 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:24:08.156 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:08.156 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:08.156 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:08.157 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:08.157 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:08.157 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:08.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 21 2024-09-11T06:24:08.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:08.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:08.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:08.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=86, tag_tid=17, entry_tid=12906], [object_number=85, tag_tid=17, entry_tid=12905], [object_number=84, tag_tid=17, entry_tid=12904], [object_number=87, tag_tid=17, entry_tid=12903]]], state=connected] 2024-09-11T06:24:08.207 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:08.209 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=86, tag_tid=17, entry_tid=12906' 2024-09-11T06:24:08.209 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=86, tag_tid=17, entry_tid=12906' 2024-09-11T06:24:08.209 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=86, tag_tid=17, entry_tid=12906' '!=' 'object_number=87, tag_tid=17, entry_tid=12823' 2024-09-11T06:24:08.209 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:24:08.210 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=86, tag_tid=17, entry_tid=12906' = 'object_number=87, tag_tid=17, entry_tid=12823' 2024-09-11T06:24:08.210 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=86, tag_tid=17, entry_tid=12906' 2024-09-11T06:24:08.210 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:08.210 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:08.210 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:08.210 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:24:08.211 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=86, tag_tid=17, entry_tid=12906' 2024-09-11T06:24:08.211 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:08.211 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:08.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:24:08.213 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:08.213 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:08.213 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:08.214 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:24:08.214 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=86, tag_tid=17, entry_tid=12906' 2024-09-11T06:24:08.215 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:08.215 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:08.216 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=12906 2024-09-11T06:24:08.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 12906 2024-09-11T06:24:08.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:24:08.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:24:08.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:24:08.418 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:24:08.418 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:08.418 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:24:08.418 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:24:08.418 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:24:08.418 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:24:08.418 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:24:08.418 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:24:08.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:24:08.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:08.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:08.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:24:08.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:24:08.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:24:08.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:24:08.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:24:08.420 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:24:08.425 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:24:08.425 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:24:08.425 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:24:08.425 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:08.425 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:24:08.425 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:24:08.425 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:24:08.425 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:24:08.425 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:24:08.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:24:08.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:24:08.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:08.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:24:08.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:08.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:24:08.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:08.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:24:08.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:24:08.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:24:08.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:24:08.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:24:08.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:24:08.427 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:24:08.427 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:24:08.427 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:08.427 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:08.427 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:08.427 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:24:08.427 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:24:08.427 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:24:08.427 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:24:08.427 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:24:08.449 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:08.450 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:08.450 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:08.450 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:24:08.585 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:24:08.586 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:24:08.586 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:08.587 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:08.587 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:08.587 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:24:08.587 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:08.587 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:08.587 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:08.587 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:24:08.588 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:08.588 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:08.588 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:08.590 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:08.591 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:08.591 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:08.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 21 2024-09-11T06:24:08.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:08.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:08.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:08.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=84, tag_tid=17, entry_tid=12992], [object_number=87, tag_tid=17, entry_tid=12991], [object_number=86, tag_tid=17, entry_tid=12990], [object_number=85, tag_tid=17, entry_tid=12989]]], state=connected] 2024-09-11T06:24:08.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:08.858 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:08.859 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:24:08.859 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:08.859 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:08.859 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:08.860 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:24:08.860 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:08.860 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:08.860 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:08.860 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:24:08.860 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:08.860 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:08.860 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:08.862 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:08.862 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:08.862 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:08.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 21 2024-09-11T06:24:08.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:08.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:08.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:08.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=84, tag_tid=17, entry_tid=12992], [object_number=87, tag_tid=17, entry_tid=12991], [object_number=86, tag_tid=17, entry_tid=12990], [object_number=85, tag_tid=17, entry_tid=12989]]], state=connected] 2024-09-11T06:24:08.923 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:08.926 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=84, tag_tid=17, entry_tid=12992' 2024-09-11T06:24:08.928 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=84, tag_tid=17, entry_tid=12992' 2024-09-11T06:24:08.928 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=84, tag_tid=17, entry_tid=12992' '!=' 'object_number=86, tag_tid=17, entry_tid=12906' 2024-09-11T06:24:08.928 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:24:08.928 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=84, tag_tid=17, entry_tid=12992' = 'object_number=86, tag_tid=17, entry_tid=12906' 2024-09-11T06:24:08.928 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=84, tag_tid=17, entry_tid=12992' 2024-09-11T06:24:08.929 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:08.929 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:08.929 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:08.930 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:24:08.930 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=84, tag_tid=17, entry_tid=12992' 2024-09-11T06:24:08.930 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:08.930 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:08.930 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:24:08.930 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:08.930 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:08.930 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:08.931 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:24:08.931 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=84, tag_tid=17, entry_tid=12992' 2024-09-11T06:24:08.932 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:08.932 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:08.932 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=12992 2024-09-11T06:24:08.933 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 12992 2024-09-11T06:24:08.933 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:24:08.933 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:24:08.933 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:24:09.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:24:09.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:09.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:24:09.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:24:09.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:24:09.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:24:09.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:24:09.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:24:09.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:24:09.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:09.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:09.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:24:09.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:24:09.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:24:09.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:24:09.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:24:09.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:24:09.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:24:09.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:24:09.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:24:09.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:09.139 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:24:09.139 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:24:09.139 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:24:09.139 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:24:09.139 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:24:09.139 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:24:09.139 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:24:09.139 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:09.139 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:24:09.139 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:09.139 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:24:09.139 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:09.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:24:09.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:24:09.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:24:09.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:24:09.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:24:09.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:24:09.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:24:09.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:24:09.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:09.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:09.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:09.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:24:09.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:24:09.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:24:09.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:24:09.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:24:09.166 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:09.167 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:09.167 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:09.167 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:24:09.297 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:24:09.298 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:24:09.298 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:09.298 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:09.298 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:09.298 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:24:09.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:09.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:09.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:09.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:24:09.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:09.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:09.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:09.300 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:09.301 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:09.301 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:09.348 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 21 2024-09-11T06:24:09.348 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:09.348 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:09.348 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:09.348 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=87, tag_tid=17, entry_tid=13115], [object_number=86, tag_tid=17, entry_tid=13114], [object_number=85, tag_tid=17, entry_tid=13113], [object_number=84, tag_tid=17, entry_tid=13112]]], state=connected] 2024-09-11T06:24:09.351 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:09.352 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:09.353 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:24:09.353 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:09.354 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:09.354 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:09.354 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:24:09.354 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:09.354 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:09.354 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:09.354 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:24:09.354 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:09.354 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:09.354 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:09.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:09.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:09.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:09.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 21 2024-09-11T06:24:09.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:09.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:09.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:09.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=87, tag_tid=17, entry_tid=13115], [object_number=86, tag_tid=17, entry_tid=13114], [object_number=85, tag_tid=17, entry_tid=13113], [object_number=84, tag_tid=17, entry_tid=13112]]], state=connected] 2024-09-11T06:24:09.404 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:09.406 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=87, tag_tid=17, entry_tid=13115' 2024-09-11T06:24:09.406 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=87, tag_tid=17, entry_tid=13115' 2024-09-11T06:24:09.406 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=87, tag_tid=17, entry_tid=13115' '!=' 'object_number=84, tag_tid=17, entry_tid=12992' 2024-09-11T06:24:09.406 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:24:09.406 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=87, tag_tid=17, entry_tid=13115' = 'object_number=84, tag_tid=17, entry_tid=12992' 2024-09-11T06:24:09.406 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=87, tag_tid=17, entry_tid=13115' 2024-09-11T06:24:09.407 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:09.407 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:09.407 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:09.407 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:24:09.408 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=87, tag_tid=17, entry_tid=13115' 2024-09-11T06:24:09.408 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:09.408 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:09.409 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:24:09.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:09.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:09.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:09.411 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:24:09.412 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=87, tag_tid=17, entry_tid=13115' 2024-09-11T06:24:09.412 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:09.412 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:09.413 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=13115 2024-09-11T06:24:09.413 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 13115 2024-09-11T06:24:09.413 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:24:09.413 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:24:09.413 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:24:09.615 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:24:09.615 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:09.616 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:24:09.616 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:24:09.616 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:24:09.616 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:24:09.616 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:24:09.616 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:24:09.616 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:24:09.616 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:09.616 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:09.616 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:24:09.616 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:24:09.616 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:24:09.617 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:24:09.617 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:24:09.617 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:24:09.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:24:09.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:24:09.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:24:09.621 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:09.621 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:24:09.621 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:24:09.621 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:24:09.621 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:24:09.621 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:24:09.621 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:24:09.621 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:24:09.621 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:09.621 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:24:09.621 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:09.621 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:24:09.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:09.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:24:09.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:24:09.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:24:09.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:24:09.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:24:09.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:24:09.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:24:09.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:24:09.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:09.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:09.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:09.623 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:24:09.623 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:24:09.623 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:24:09.623 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:24:09.623 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:24:09.645 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:09.646 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:09.646 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:09.646 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:24:09.774 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:24:09.775 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:24:09.775 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:09.776 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:09.776 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:09.776 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:24:09.776 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:09.776 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:09.776 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:09.776 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:24:09.777 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:09.777 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:09.777 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:09.779 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:09.780 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:09.780 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:09.830 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 21 2024-09-11T06:24:09.830 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:09.830 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:09.830 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:09.830 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=84, tag_tid=17, entry_tid=13200], [object_number=87, tag_tid=17, entry_tid=13199], [object_number=86, tag_tid=17, entry_tid=13198], [object_number=85, tag_tid=17, entry_tid=13197]]], state=connected] 2024-09-11T06:24:09.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:09.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:09.836 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:24:09.836 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:09.836 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:09.836 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:09.836 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:24:09.836 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:09.836 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:09.836 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:09.836 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:24:09.836 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:09.836 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:09.836 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:09.837 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:09.837 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:09.837 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:10.091 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 21 2024-09-11T06:24:10.091 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:10.091 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:10.091 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:10.091 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=84, tag_tid=17, entry_tid=13200], [object_number=87, tag_tid=17, entry_tid=13199], [object_number=86, tag_tid=17, entry_tid=13198], [object_number=85, tag_tid=17, entry_tid=13197]]], state=connected] 2024-09-11T06:24:10.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:10.143 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=84, tag_tid=17, entry_tid=13200' 2024-09-11T06:24:10.143 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=84, tag_tid=17, entry_tid=13200' 2024-09-11T06:24:10.143 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=84, tag_tid=17, entry_tid=13200' '!=' 'object_number=87, tag_tid=17, entry_tid=13115' 2024-09-11T06:24:10.143 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:24:10.143 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=84, tag_tid=17, entry_tid=13200' = 'object_number=87, tag_tid=17, entry_tid=13115' 2024-09-11T06:24:10.143 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=84, tag_tid=17, entry_tid=13200' 2024-09-11T06:24:10.144 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:10.144 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:10.144 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:10.145 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:24:10.146 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=84, tag_tid=17, entry_tid=13200' 2024-09-11T06:24:10.146 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:10.146 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:10.147 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:24:10.148 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:10.148 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:10.148 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:10.150 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:24:10.150 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=84, tag_tid=17, entry_tid=13200' 2024-09-11T06:24:10.151 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:10.151 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:10.152 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=13200 2024-09-11T06:24:10.152 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 13200 2024-09-11T06:24:10.153 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:24:10.153 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:24:10.153 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:24:10.354 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:24:10.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:10.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:24:10.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:24:10.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:24:10.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:24:10.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:24:10.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:24:10.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:24:10.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:10.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:10.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:24:10.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:24:10.356 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:24:10.356 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:24:10.356 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:24:10.356 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:24:10.359 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:24:10.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:24:10.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:24:10.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:10.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:24:10.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:24:10.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:24:10.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:24:10.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:24:10.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:24:10.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:24:10.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:10.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:24:10.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:10.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:24:10.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:10.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:24:10.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:24:10.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:24:10.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:24:10.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:24:10.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:24:10.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:24:10.362 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:24:10.362 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:10.362 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:10.362 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:10.362 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:24:10.362 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:24:10.362 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:24:10.362 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:24:10.362 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:24:10.385 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:10.385 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:10.385 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:10.385 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:24:10.522 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:24:10.522 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:24:10.523 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:10.523 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:10.523 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:10.523 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:24:10.523 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:10.523 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:10.523 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:10.523 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:24:10.524 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:10.524 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:10.524 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:10.526 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:10.527 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:10.527 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:10.583 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 21 2024-09-11T06:24:10.583 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:10.583 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:10.583 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:10.583 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=86, tag_tid=17, entry_tid=13334], [object_number=85, tag_tid=17, entry_tid=13333], [object_number=84, tag_tid=17, entry_tid=13332], [object_number=87, tag_tid=17, entry_tid=13331]]], state=connected] 2024-09-11T06:24:10.586 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:10.587 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:10.589 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:24:10.589 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:10.589 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:10.589 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:10.589 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:24:10.589 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:10.589 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:10.589 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:10.589 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:24:10.589 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:10.589 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:10.589 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:10.590 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:10.590 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:10.590 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:10.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 21 2024-09-11T06:24:10.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:10.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:10.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:10.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=86, tag_tid=17, entry_tid=13334], [object_number=85, tag_tid=17, entry_tid=13333], [object_number=84, tag_tid=17, entry_tid=13332], [object_number=87, tag_tid=17, entry_tid=13331]]], state=connected] 2024-09-11T06:24:10.855 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:10.857 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=86, tag_tid=17, entry_tid=13334' 2024-09-11T06:24:10.857 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=86, tag_tid=17, entry_tid=13334' 2024-09-11T06:24:10.857 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=86, tag_tid=17, entry_tid=13334' '!=' 'object_number=84, tag_tid=17, entry_tid=13200' 2024-09-11T06:24:10.857 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:24:10.857 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=86, tag_tid=17, entry_tid=13334' = 'object_number=84, tag_tid=17, entry_tid=13200' 2024-09-11T06:24:10.857 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=86, tag_tid=17, entry_tid=13334' 2024-09-11T06:24:10.858 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:10.858 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:10.858 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:10.859 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:24:10.860 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=86, tag_tid=17, entry_tid=13334' 2024-09-11T06:24:10.860 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:10.860 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:10.861 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:24:10.862 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:10.862 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:10.862 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:10.863 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:24:10.864 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=86, tag_tid=17, entry_tid=13334' 2024-09-11T06:24:10.864 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:10.864 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:10.866 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=13334 2024-09-11T06:24:10.866 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 13334 2024-09-11T06:24:10.866 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:24:10.866 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:24:10.866 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:24:11.067 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:24:11.068 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:11.068 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:24:11.068 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:24:11.068 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:24:11.068 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:24:11.068 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:24:11.068 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:24:11.068 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:24:11.068 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:11.068 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:11.068 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:24:11.069 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:24:11.069 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:24:11.069 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:24:11.069 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:24:11.069 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:24:11.072 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:24:11.072 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:24:11.072 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:24:11.072 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:11.072 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:24:11.072 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:24:11.072 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:24:11.072 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:24:11.072 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:24:11.072 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:24:11.072 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:24:11.072 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:11.073 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:24:11.073 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:11.073 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:24:11.073 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:11.074 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:24:11.074 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:24:11.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:24:11.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:24:11.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:24:11.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:24:11.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:24:11.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:24:11.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:11.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:11.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:11.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:24:11.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:24:11.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:24:11.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:24:11.076 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:24:11.099 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:11.099 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:11.099 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:11.099 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:24:11.228 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:24:11.229 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:24:11.229 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:11.229 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:11.230 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:11.230 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:24:11.230 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:11.230 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:11.230 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:11.230 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:24:11.230 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:11.231 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:11.231 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:11.232 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:11.232 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:11.232 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:11.293 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 21 2024-09-11T06:24:11.293 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:11.293 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:11.293 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:11.293 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=84, tag_tid=17, entry_tid=13456], [object_number=87, tag_tid=17, entry_tid=13455], [object_number=86, tag_tid=17, entry_tid=13454], [object_number=85, tag_tid=17, entry_tid=13453]]], state=connected] 2024-09-11T06:24:11.296 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:11.297 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:11.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:24:11.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:11.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:11.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:11.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:24:11.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:11.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:11.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:11.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:24:11.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:11.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:11.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:11.300 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:11.300 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:11.300 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:11.353 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 21 2024-09-11T06:24:11.354 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:11.354 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:11.354 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:11.354 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=84, tag_tid=17, entry_tid=13456], [object_number=87, tag_tid=17, entry_tid=13455], [object_number=86, tag_tid=17, entry_tid=13454], [object_number=85, tag_tid=17, entry_tid=13453]]], state=connected] 2024-09-11T06:24:11.357 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:11.358 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=84, tag_tid=17, entry_tid=13456' 2024-09-11T06:24:11.359 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=84, tag_tid=17, entry_tid=13456' 2024-09-11T06:24:11.359 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=84, tag_tid=17, entry_tid=13456' '!=' 'object_number=86, tag_tid=17, entry_tid=13334' 2024-09-11T06:24:11.359 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:24:11.359 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=84, tag_tid=17, entry_tid=13456' = 'object_number=86, tag_tid=17, entry_tid=13334' 2024-09-11T06:24:11.359 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=84, tag_tid=17, entry_tid=13456' 2024-09-11T06:24:11.359 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:11.359 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:11.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:11.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:24:11.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=84, tag_tid=17, entry_tid=13456' 2024-09-11T06:24:11.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:11.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:11.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:24:11.362 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:11.362 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:11.362 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:11.363 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:24:11.364 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=84, tag_tid=17, entry_tid=13456' 2024-09-11T06:24:11.364 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:11.364 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:11.365 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=13456 2024-09-11T06:24:11.365 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 13456 2024-09-11T06:24:11.365 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:24:11.365 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:24:11.365 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:24:11.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:24:11.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:11.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:24:11.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:24:11.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:24:11.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:24:11.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:24:11.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:24:11.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:24:11.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:11.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:11.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:24:11.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:24:11.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:24:11.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:24:11.569 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:24:11.569 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:24:11.570 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:24:11.570 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:24:11.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:24:11.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:11.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:24:11.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:24:11.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:24:11.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:24:11.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:24:11.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:24:11.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:24:11.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:11.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:24:11.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:11.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:24:11.572 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:11.573 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:24:11.573 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:24:11.573 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:24:11.573 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:24:11.573 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:24:11.573 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:24:11.573 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:24:11.573 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:24:11.573 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:11.574 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:11.574 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:11.574 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:24:11.574 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:24:11.574 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:24:11.574 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:24:11.574 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:24:11.597 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:11.598 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:11.598 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:11.598 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:24:11.727 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:24:11.727 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:24:11.727 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:11.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:11.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:11.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:24:11.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:11.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:11.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:11.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:24:11.729 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:11.729 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:11.729 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:11.731 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:11.732 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:11.732 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:11.782 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 21 2024-09-11T06:24:11.782 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:11.782 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:11.782 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:11.782 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=89, tag_tid=17, entry_tid=13477], [object_number=88, tag_tid=17, entry_tid=13476], [object_number=91, tag_tid=17, entry_tid=13475], [object_number=86, tag_tid=17, entry_tid=13474]]], state=connected] 2024-09-11T06:24:11.785 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:11.787 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:11.788 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:24:11.788 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:11.788 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:11.788 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:11.788 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:24:11.788 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:11.788 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:11.788 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:11.788 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:24:11.789 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:11.789 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:11.789 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:11.789 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:11.791 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:11.791 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:11.838 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 21 2024-09-11T06:24:11.839 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:11.839 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:11.839 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:11.839 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=89, tag_tid=17, entry_tid=13477], [object_number=88, tag_tid=17, entry_tid=13476], [object_number=91, tag_tid=17, entry_tid=13475], [object_number=86, tag_tid=17, entry_tid=13474]]], state=connected] 2024-09-11T06:24:11.842 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:11.844 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=89, tag_tid=17, entry_tid=13477' 2024-09-11T06:24:11.844 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=89, tag_tid=17, entry_tid=13477' 2024-09-11T06:24:11.844 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=89, tag_tid=17, entry_tid=13477' '!=' 'object_number=84, tag_tid=17, entry_tid=13456' 2024-09-11T06:24:11.844 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:24:11.844 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=89, tag_tid=17, entry_tid=13477' = 'object_number=84, tag_tid=17, entry_tid=13456' 2024-09-11T06:24:11.844 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=89, tag_tid=17, entry_tid=13477' 2024-09-11T06:24:11.845 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:11.845 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:11.845 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:11.845 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:24:11.846 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=89, tag_tid=17, entry_tid=13477' 2024-09-11T06:24:11.846 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:11.846 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:11.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:24:11.848 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:11.848 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:11.848 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:11.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:24:11.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=89, tag_tid=17, entry_tid=13477' 2024-09-11T06:24:11.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:11.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:11.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=13477 2024-09-11T06:24:11.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 13477 2024-09-11T06:24:11.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:24:11.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:24:11.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:24:12.053 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:24:12.053 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:12.053 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:24:12.053 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:24:12.053 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:24:12.053 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:24:12.053 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:24:12.054 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:24:12.054 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:24:12.054 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:12.054 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:12.054 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:24:12.054 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:24:12.054 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:24:12.054 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:24:12.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:24:12.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:24:12.058 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:24:12.058 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:24:12.058 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:24:12.058 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:12.058 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:24:12.058 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:24:12.059 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:24:12.059 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:24:12.059 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:24:12.059 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:24:12.059 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:24:12.059 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:12.059 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:24:12.059 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:12.059 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:24:12.059 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:12.059 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:24:12.059 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:24:12.059 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:24:12.059 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:24:12.060 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:24:12.060 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:24:12.060 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:24:12.060 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:24:12.060 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:12.060 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:12.060 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:12.060 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:24:12.060 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:24:12.060 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:24:12.060 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:24:12.060 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:24:12.084 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:12.084 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:12.084 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:12.084 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:24:12.232 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:24:12.233 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:24:12.233 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:12.233 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:12.233 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:12.233 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:24:12.233 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:12.233 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:12.233 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:12.233 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:24:12.233 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:12.234 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:12.234 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:12.235 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:12.235 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:12.235 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:12.294 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 22 2024-09-11T06:24:12.294 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:12.294 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:12.294 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:12.294 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=89, tag_tid=17, entry_tid=13561], [object_number=88, tag_tid=17, entry_tid=13560], [object_number=91, tag_tid=17, entry_tid=13559], [object_number=90, tag_tid=17, entry_tid=13558]]], state=connected] 2024-09-11T06:24:12.297 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:12.298 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:12.300 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:24:12.300 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:12.300 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:12.300 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:12.300 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:24:12.300 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:12.300 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:12.300 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:12.300 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:24:12.300 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:12.300 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:12.300 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:12.301 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:12.301 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:12.301 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:12.347 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 22 2024-09-11T06:24:12.347 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:12.347 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:12.348 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:12.348 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=89, tag_tid=17, entry_tid=13561], [object_number=88, tag_tid=17, entry_tid=13560], [object_number=91, tag_tid=17, entry_tid=13559], [object_number=90, tag_tid=17, entry_tid=13558]]], state=connected] 2024-09-11T06:24:12.351 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:12.352 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=89, tag_tid=17, entry_tid=13561' 2024-09-11T06:24:12.353 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=89, tag_tid=17, entry_tid=13561' 2024-09-11T06:24:12.353 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=89, tag_tid=17, entry_tid=13561' '!=' 'object_number=89, tag_tid=17, entry_tid=13477' 2024-09-11T06:24:12.353 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:24:12.353 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=89, tag_tid=17, entry_tid=13561' = 'object_number=89, tag_tid=17, entry_tid=13477' 2024-09-11T06:24:12.353 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=89, tag_tid=17, entry_tid=13561' 2024-09-11T06:24:12.353 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:12.354 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:12.354 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:12.354 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:24:12.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=89, tag_tid=17, entry_tid=13561' 2024-09-11T06:24:12.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:12.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:12.356 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:24:12.356 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:12.356 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:12.357 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:12.357 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:24:12.358 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=89, tag_tid=17, entry_tid=13561' 2024-09-11T06:24:12.358 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:12.358 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:12.359 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=13561 2024-09-11T06:24:12.359 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 13561 2024-09-11T06:24:12.359 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:24:12.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:24:12.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:24:12.561 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:24:12.561 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:12.562 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:24:12.562 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:24:12.562 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:24:12.562 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:24:12.562 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:24:12.562 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:24:12.562 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:24:12.562 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:12.562 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:12.562 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:24:12.562 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:24:12.562 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:24:12.562 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:24:12.563 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:24:12.563 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:24:12.565 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:24:12.565 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:24:12.565 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:24:12.565 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:12.565 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:24:12.566 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:24:12.566 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:24:12.566 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:24:12.566 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:24:12.566 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:24:12.566 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:24:12.566 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:12.566 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:24:12.566 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:12.566 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:24:12.566 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:12.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:24:12.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:24:12.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:24:12.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:24:12.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:24:12.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:24:12.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:24:12.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:24:12.569 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:12.569 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:12.569 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:12.569 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:24:12.569 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:24:12.569 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:24:12.569 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:24:12.569 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:24:12.593 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:12.593 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:12.593 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:12.593 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:24:12.722 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:24:12.724 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:24:12.724 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:12.724 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:12.724 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:12.724 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:24:12.724 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:12.724 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:12.724 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:12.724 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:24:12.725 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:12.725 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:12.725 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:12.726 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:12.726 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:12.726 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:12.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 22 2024-09-11T06:24:12.785 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:12.785 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:12.785 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:12.785 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=88, tag_tid=17, entry_tid=13648], [object_number=91, tag_tid=17, entry_tid=13647], [object_number=90, tag_tid=17, entry_tid=13646], [object_number=89, tag_tid=17, entry_tid=13645]]], state=connected] 2024-09-11T06:24:12.788 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:12.789 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:12.791 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:24:12.791 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:12.791 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:12.791 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:12.791 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:24:12.791 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:12.791 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:12.791 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:12.791 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:24:12.791 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:12.791 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:12.792 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:12.792 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:12.792 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:12.792 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:12.840 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 22 2024-09-11T06:24:12.840 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:12.840 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:12.840 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:12.840 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=88, tag_tid=17, entry_tid=13648], [object_number=91, tag_tid=17, entry_tid=13647], [object_number=90, tag_tid=17, entry_tid=13646], [object_number=89, tag_tid=17, entry_tid=13645]]], state=connected] 2024-09-11T06:24:12.843 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:12.845 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=88, tag_tid=17, entry_tid=13648' 2024-09-11T06:24:12.845 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=88, tag_tid=17, entry_tid=13648' 2024-09-11T06:24:12.845 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=88, tag_tid=17, entry_tid=13648' '!=' 'object_number=89, tag_tid=17, entry_tid=13561' 2024-09-11T06:24:12.845 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:24:12.845 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=88, tag_tid=17, entry_tid=13648' = 'object_number=89, tag_tid=17, entry_tid=13561' 2024-09-11T06:24:12.846 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=88, tag_tid=17, entry_tid=13648' 2024-09-11T06:24:12.846 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:12.846 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:12.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:12.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:24:12.848 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=88, tag_tid=17, entry_tid=13648' 2024-09-11T06:24:12.848 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:12.848 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:12.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:24:12.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:12.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:12.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:12.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:24:12.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=88, tag_tid=17, entry_tid=13648' 2024-09-11T06:24:12.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:12.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:12.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=13648 2024-09-11T06:24:12.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 13648 2024-09-11T06:24:12.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:24:12.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:24:12.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:24:13.054 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:24:13.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:13.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:24:13.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:24:13.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:24:13.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:24:13.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:24:13.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:24:13.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:24:13.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:13.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:13.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:24:13.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:24:13.056 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:24:13.056 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:24:13.056 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:24:13.056 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:24:13.058 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:24:13.058 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:24:13.058 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:24:13.058 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:13.059 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:24:13.059 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:24:13.059 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:24:13.059 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:24:13.059 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:24:13.059 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:24:13.059 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:24:13.060 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:13.060 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:24:13.060 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:13.060 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:24:13.060 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:13.063 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:24:13.063 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:24:13.063 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:24:13.063 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:24:13.063 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:24:13.063 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:24:13.063 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:24:13.064 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:24:13.064 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:13.064 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:13.064 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:13.064 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:24:13.064 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:24:13.065 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:24:13.065 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:24:13.065 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:24:13.085 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:13.086 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:13.086 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:13.086 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:24:13.214 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:24:13.215 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:24:13.215 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:13.215 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:13.215 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:13.216 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:24:13.216 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:13.216 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:13.216 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:13.216 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:24:13.216 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:13.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:13.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:13.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:13.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:13.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:13.274 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 22 2024-09-11T06:24:13.274 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:13.274 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:13.274 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:13.275 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=90, tag_tid=17, entry_tid=13730], [object_number=89, tag_tid=17, entry_tid=13729], [object_number=88, tag_tid=17, entry_tid=13728], [object_number=91, tag_tid=17, entry_tid=13727]]], state=connected] 2024-09-11T06:24:13.277 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:13.278 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:13.280 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:24:13.280 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:13.280 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:13.280 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:13.280 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:24:13.280 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:13.281 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:13.281 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:13.281 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:24:13.281 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:13.281 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:13.281 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:13.282 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:13.282 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:13.282 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:13.330 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 22 2024-09-11T06:24:13.330 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:13.330 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:13.330 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:13.330 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=90, tag_tid=17, entry_tid=13730], [object_number=89, tag_tid=17, entry_tid=13729], [object_number=88, tag_tid=17, entry_tid=13728], [object_number=91, tag_tid=17, entry_tid=13727]]], state=connected] 2024-09-11T06:24:13.333 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:13.335 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=90, tag_tid=17, entry_tid=13730' 2024-09-11T06:24:13.335 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=90, tag_tid=17, entry_tid=13730' 2024-09-11T06:24:13.335 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=90, tag_tid=17, entry_tid=13730' '!=' 'object_number=88, tag_tid=17, entry_tid=13648' 2024-09-11T06:24:13.335 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:24:13.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=90, tag_tid=17, entry_tid=13730' = 'object_number=88, tag_tid=17, entry_tid=13648' 2024-09-11T06:24:13.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=90, tag_tid=17, entry_tid=13730' 2024-09-11T06:24:13.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:13.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:13.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:13.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:24:13.337 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=90, tag_tid=17, entry_tid=13730' 2024-09-11T06:24:13.337 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:13.337 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:13.338 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:24:13.339 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:13.339 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:13.339 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:13.340 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:24:13.340 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=90, tag_tid=17, entry_tid=13730' 2024-09-11T06:24:13.341 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:13.341 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:13.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=13730 2024-09-11T06:24:13.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 13730 2024-09-11T06:24:13.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:24:13.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:24:13.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:24:13.544 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:24:13.544 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:13.544 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:24:13.544 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:24:13.544 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:24:13.544 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:24:13.544 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:24:13.545 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:24:13.545 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:24:13.545 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:13.545 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:13.545 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:24:13.545 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:24:13.545 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:24:13.545 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:24:13.546 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:24:13.546 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:24:13.550 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:24:13.550 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:24:13.550 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:24:13.550 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:13.550 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:24:13.550 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:24:13.550 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:24:13.550 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:24:13.551 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:24:13.551 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:24:13.551 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:24:13.551 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:13.551 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:24:13.551 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:13.551 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:24:13.551 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:13.551 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:24:13.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:24:13.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:24:13.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:24:13.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:24:13.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:24:13.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:24:13.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:24:13.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:13.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:13.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:13.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:24:13.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:24:13.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:24:13.553 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:24:13.553 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:24:13.576 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:13.576 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:13.576 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:13.577 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:24:13.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:24:13.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:24:13.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:13.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:13.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:13.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:24:13.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:13.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:13.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:13.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:24:13.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:13.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:13.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:13.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:13.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:13.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:13.772 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 22 2024-09-11T06:24:13.772 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:13.773 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:13.773 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:13.773 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=89, tag_tid=17, entry_tid=13813], [object_number=88, tag_tid=17, entry_tid=13812], [object_number=91, tag_tid=17, entry_tid=13811], [object_number=90, tag_tid=17, entry_tid=13810]]], state=connected] 2024-09-11T06:24:13.776 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:13.777 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:13.778 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:24:13.778 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:13.778 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:13.779 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:13.779 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:24:13.779 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:13.779 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:13.779 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:13.779 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:24:13.779 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:13.779 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:13.779 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:13.780 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:13.780 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:13.780 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:13.826 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 22 2024-09-11T06:24:13.826 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:13.826 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:13.827 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:13.827 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=89, tag_tid=17, entry_tid=13813], [object_number=88, tag_tid=17, entry_tid=13812], [object_number=91, tag_tid=17, entry_tid=13811], [object_number=90, tag_tid=17, entry_tid=13810]]], state=connected] 2024-09-11T06:24:13.830 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:13.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=89, tag_tid=17, entry_tid=13813' 2024-09-11T06:24:13.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=89, tag_tid=17, entry_tid=13813' 2024-09-11T06:24:13.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=89, tag_tid=17, entry_tid=13813' '!=' 'object_number=90, tag_tid=17, entry_tid=13730' 2024-09-11T06:24:13.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:24:13.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=89, tag_tid=17, entry_tid=13813' = 'object_number=90, tag_tid=17, entry_tid=13730' 2024-09-11T06:24:13.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=89, tag_tid=17, entry_tid=13813' 2024-09-11T06:24:13.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:13.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:13.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:13.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:24:13.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=89, tag_tid=17, entry_tid=13813' 2024-09-11T06:24:13.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:13.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:13.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:24:13.835 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:13.835 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:13.835 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:13.836 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:24:13.837 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=89, tag_tid=17, entry_tid=13813' 2024-09-11T06:24:13.837 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:13.837 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:13.839 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=13813 2024-09-11T06:24:13.839 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 13813 2024-09-11T06:24:13.839 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:24:13.839 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:24:13.839 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:24:14.040 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:24:14.040 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:14.040 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:24:14.040 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:24:14.040 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:24:14.041 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:24:14.041 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:24:14.041 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:24:14.041 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:24:14.041 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:14.041 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:14.041 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:24:14.041 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:24:14.041 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:24:14.041 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:24:14.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:24:14.045 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:24:14.045 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:24:14.046 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:24:14.046 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:24:14.046 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:14.046 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:24:14.046 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:24:14.046 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:24:14.046 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:24:14.046 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:24:14.046 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:24:14.046 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:24:14.046 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:14.046 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:24:14.046 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:14.046 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:24:14.047 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:14.047 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:24:14.047 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:24:14.047 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:24:14.047 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:24:14.047 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:24:14.047 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:24:14.047 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:24:14.047 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:24:14.047 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:14.047 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:14.047 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:14.047 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:24:14.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:24:14.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:24:14.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:24:14.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:24:14.072 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:14.073 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:14.073 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:14.073 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:24:14.206 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:24:14.207 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:24:14.207 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:14.207 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:14.208 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:14.208 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:24:14.208 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:14.208 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:14.208 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:14.208 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:24:14.208 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:14.209 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:14.209 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:14.210 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:14.210 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:14.210 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:14.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 22 2024-09-11T06:24:14.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:14.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:14.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:14.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=89, tag_tid=17, entry_tid=13897], [object_number=88, tag_tid=17, entry_tid=13896], [object_number=91, tag_tid=17, entry_tid=13895], [object_number=90, tag_tid=17, entry_tid=13894]]], state=connected] 2024-09-11T06:24:14.270 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:14.271 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:14.273 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:24:14.273 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:14.273 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:14.273 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:14.273 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:24:14.273 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:14.273 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:14.273 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:14.273 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:24:14.274 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:14.274 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:14.274 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:14.274 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:14.275 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:14.275 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:14.321 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 22 2024-09-11T06:24:14.321 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:14.322 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:14.322 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:14.322 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=89, tag_tid=17, entry_tid=13897], [object_number=88, tag_tid=17, entry_tid=13896], [object_number=91, tag_tid=17, entry_tid=13895], [object_number=90, tag_tid=17, entry_tid=13894]]], state=connected] 2024-09-11T06:24:14.325 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:14.327 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=89, tag_tid=17, entry_tid=13897' 2024-09-11T06:24:14.327 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=89, tag_tid=17, entry_tid=13897' 2024-09-11T06:24:14.327 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=89, tag_tid=17, entry_tid=13897' '!=' 'object_number=89, tag_tid=17, entry_tid=13813' 2024-09-11T06:24:14.327 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:24:14.327 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=89, tag_tid=17, entry_tid=13897' = 'object_number=89, tag_tid=17, entry_tid=13813' 2024-09-11T06:24:14.327 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=89, tag_tid=17, entry_tid=13897' 2024-09-11T06:24:14.327 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:14.327 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:14.328 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:14.328 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:24:14.329 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=89, tag_tid=17, entry_tid=13897' 2024-09-11T06:24:14.329 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:14.329 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:14.330 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:24:14.330 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:14.331 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:14.331 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:14.332 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:24:14.332 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=89, tag_tid=17, entry_tid=13897' 2024-09-11T06:24:14.332 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:14.333 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:14.334 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=13897 2024-09-11T06:24:14.334 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 13897 2024-09-11T06:24:14.334 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:24:14.334 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:24:14.334 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:24:14.535 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:24:14.536 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:14.536 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:24:14.536 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:24:14.536 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:24:14.536 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:24:14.536 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:24:14.536 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:24:14.536 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:24:14.536 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:14.536 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:14.536 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:24:14.536 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:24:14.537 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:24:14.537 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:24:14.537 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:24:14.537 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:24:14.540 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:24:14.540 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:24:14.541 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:24:14.541 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:14.541 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:24:14.541 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:24:14.541 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:24:14.541 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:24:14.541 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:24:14.541 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:24:14.541 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:24:14.541 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:14.541 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:24:14.541 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:14.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:24:14.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:14.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:24:14.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:24:14.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:24:14.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:24:14.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:24:14.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:24:14.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:24:14.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:24:14.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:14.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:14.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:14.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:24:14.543 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:24:14.543 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:24:14.543 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:24:14.543 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:24:14.566 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:14.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:14.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:14.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:24:14.697 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:24:14.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:24:14.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:14.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:14.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:14.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:24:14.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:14.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:14.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:14.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:24:14.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:14.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:14.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:14.700 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:14.701 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:14.701 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:14.749 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 22 2024-09-11T06:24:14.749 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:14.749 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:14.749 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:14.749 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=91, tag_tid=17, entry_tid=13983], [object_number=90, tag_tid=17, entry_tid=13982], [object_number=89, tag_tid=17, entry_tid=13981], [object_number=88, tag_tid=17, entry_tid=13980]]], state=connected] 2024-09-11T06:24:14.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:14.754 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:14.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:24:14.756 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:14.756 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:14.756 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:14.756 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:24:14.756 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:14.756 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:14.756 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:14.756 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:24:14.756 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:14.756 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:14.756 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:14.757 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:14.757 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:14.757 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:14.803 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 22 2024-09-11T06:24:14.803 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:14.804 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:14.804 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:14.804 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=91, tag_tid=17, entry_tid=13983], [object_number=90, tag_tid=17, entry_tid=13982], [object_number=89, tag_tid=17, entry_tid=13981], [object_number=88, tag_tid=17, entry_tid=13980]]], state=connected] 2024-09-11T06:24:14.807 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:14.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=91, tag_tid=17, entry_tid=13983' 2024-09-11T06:24:14.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=91, tag_tid=17, entry_tid=13983' 2024-09-11T06:24:14.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=91, tag_tid=17, entry_tid=13983' '!=' 'object_number=89, tag_tid=17, entry_tid=13897' 2024-09-11T06:24:14.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:24:14.810 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=91, tag_tid=17, entry_tid=13983' = 'object_number=89, tag_tid=17, entry_tid=13897' 2024-09-11T06:24:14.810 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=91, tag_tid=17, entry_tid=13983' 2024-09-11T06:24:14.810 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:14.810 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:14.810 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:14.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:24:14.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=91, tag_tid=17, entry_tid=13983' 2024-09-11T06:24:14.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:14.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:14.812 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:24:14.813 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:14.813 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:14.813 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:14.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:24:14.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=91, tag_tid=17, entry_tid=13983' 2024-09-11T06:24:14.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:14.815 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:14.816 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=13983 2024-09-11T06:24:14.816 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 13983 2024-09-11T06:24:14.816 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:24:14.816 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:24:14.816 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:24:15.018 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:24:15.018 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:15.018 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:24:15.018 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:24:15.018 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:24:15.019 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:24:15.019 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:24:15.019 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:24:15.019 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:24:15.019 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:15.019 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:15.020 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:24:15.020 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:24:15.020 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:24:15.020 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:24:15.024 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:24:15.024 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:24:15.024 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:24:15.024 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:24:15.024 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:24:15.024 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:15.024 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:24:15.025 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:24:15.025 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:24:15.025 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:24:15.025 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:24:15.025 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:24:15.025 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:24:15.025 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:15.025 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:24:15.025 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:15.025 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:24:15.025 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:15.025 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:24:15.025 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:24:15.026 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:24:15.026 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:24:15.026 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:24:15.026 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:24:15.026 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:24:15.026 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:24:15.026 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:15.026 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:15.026 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:15.026 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:24:15.026 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:24:15.026 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:24:15.026 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:24:15.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:24:15.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:15.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:15.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:15.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:24:15.183 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:24:15.186 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:24:15.186 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:15.186 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:15.187 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:15.187 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:24:15.187 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:15.187 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:15.187 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:15.187 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:24:15.187 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:15.187 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:15.188 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:15.188 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:15.188 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:15.188 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:15.189 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:24:15.186+0000 7fc41084f640 -1 reset not still connected to 0x55eb15565d40 2024-09-11T06:24:15.190 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:24:15.186+0000 7fc41084f640 -1 reset not still connected to 0x55eb15806dd0 2024-09-11T06:24:15.190 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:24:15.186+0000 7fc41084f640 -1 reset not still connected to 0x55eb16561450 2024-09-11T06:24:15.236 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 22 2024-09-11T06:24:15.236 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:15.236 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:15.237 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:15.237 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=90, tag_tid=17, entry_tid=14066], [object_number=89, tag_tid=17, entry_tid=14065], [object_number=88, tag_tid=17, entry_tid=14064], [object_number=91, tag_tid=17, entry_tid=14063]]], state=connected] 2024-09-11T06:24:15.239 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:15.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:15.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:24:15.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:15.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:15.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:15.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:24:15.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:15.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:15.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:15.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:24:15.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:15.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:15.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:15.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:15.244 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:15.244 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:15.290 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 22 2024-09-11T06:24:15.291 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:15.291 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:15.291 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:15.291 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=90, tag_tid=17, entry_tid=14066], [object_number=89, tag_tid=17, entry_tid=14065], [object_number=88, tag_tid=17, entry_tid=14064], [object_number=91, tag_tid=17, entry_tid=14063]]], state=connected] 2024-09-11T06:24:15.294 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:15.296 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=90, tag_tid=17, entry_tid=14066' 2024-09-11T06:24:15.296 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=90, tag_tid=17, entry_tid=14066' 2024-09-11T06:24:15.296 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=90, tag_tid=17, entry_tid=14066' '!=' 'object_number=91, tag_tid=17, entry_tid=13983' 2024-09-11T06:24:15.296 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:24:15.296 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=90, tag_tid=17, entry_tid=14066' = 'object_number=91, tag_tid=17, entry_tid=13983' 2024-09-11T06:24:15.297 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=90, tag_tid=17, entry_tid=14066' 2024-09-11T06:24:15.297 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:15.297 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:15.297 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:15.297 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:24:15.298 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=90, tag_tid=17, entry_tid=14066' 2024-09-11T06:24:15.298 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:15.298 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:15.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:24:15.300 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:15.300 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:15.300 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:15.301 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:24:15.301 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=90, tag_tid=17, entry_tid=14066' 2024-09-11T06:24:15.301 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:15.301 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:15.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=14066 2024-09-11T06:24:15.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 14066 2024-09-11T06:24:15.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:24:15.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:24:15.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:24:15.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:24:15.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:15.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:24:15.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:24:15.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:24:15.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:24:15.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:24:15.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:24:15.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:24:15.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:15.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:15.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:24:15.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:24:15.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:24:15.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:24:15.507 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:24:15.507 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:24:15.510 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:24:15.510 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:24:15.510 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:24:15.510 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:15.510 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:24:15.510 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:24:15.511 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:24:15.511 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:24:15.511 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:24:15.511 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:24:15.511 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:24:15.511 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:15.511 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:24:15.511 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:15.511 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:24:15.511 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:15.511 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:24:15.511 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:24:15.511 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:24:15.511 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:24:15.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:24:15.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:24:15.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:24:15.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:24:15.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:15.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:15.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:15.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:24:15.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:24:15.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:24:15.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:24:15.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:24:15.535 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:15.535 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:15.536 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:15.536 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:24:15.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:24:15.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:24:15.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:15.668 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:15.668 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:15.668 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:24:15.668 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:15.668 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:15.668 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:15.668 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:24:15.668 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:15.669 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:15.669 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:15.671 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:15.671 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:15.671 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:15.724 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 22 2024-09-11T06:24:15.724 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:15.724 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:15.724 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:15.724 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=88, tag_tid=17, entry_tid=14156], [object_number=91, tag_tid=17, entry_tid=14155], [object_number=90, tag_tid=17, entry_tid=14154], [object_number=89, tag_tid=17, entry_tid=14153]]], state=connected] 2024-09-11T06:24:15.727 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:15.729 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:15.730 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:24:15.730 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:15.730 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:15.730 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:15.730 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:24:15.730 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:15.731 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:15.731 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:15.731 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:24:15.731 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:15.731 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:15.731 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:15.732 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:15.732 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:15.732 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:15.778 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 22 2024-09-11T06:24:15.779 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:15.779 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:15.779 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:15.779 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=88, tag_tid=17, entry_tid=14156], [object_number=91, tag_tid=17, entry_tid=14155], [object_number=90, tag_tid=17, entry_tid=14154], [object_number=89, tag_tid=17, entry_tid=14153]]], state=connected] 2024-09-11T06:24:15.782 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:15.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=88, tag_tid=17, entry_tid=14156' 2024-09-11T06:24:15.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=88, tag_tid=17, entry_tid=14156' 2024-09-11T06:24:15.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=88, tag_tid=17, entry_tid=14156' '!=' 'object_number=90, tag_tid=17, entry_tid=14066' 2024-09-11T06:24:15.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:24:15.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=88, tag_tid=17, entry_tid=14156' = 'object_number=90, tag_tid=17, entry_tid=14066' 2024-09-11T06:24:15.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=88, tag_tid=17, entry_tid=14156' 2024-09-11T06:24:15.785 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:15.785 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:15.785 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:15.785 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:24:15.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=88, tag_tid=17, entry_tid=14156' 2024-09-11T06:24:15.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:15.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:15.787 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:24:15.787 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:15.788 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:15.788 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:15.788 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:24:15.789 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=88, tag_tid=17, entry_tid=14156' 2024-09-11T06:24:15.789 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:15.789 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:15.790 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=14156 2024-09-11T06:24:15.791 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 14156 2024-09-11T06:24:15.791 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:24:15.791 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:24:15.791 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:24:15.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:24:15.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:15.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:24:15.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:24:15.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:24:15.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:24:15.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:24:15.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:24:15.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:24:15.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:15.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:15.994 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:24:15.994 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:24:15.994 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:24:15.994 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:24:15.994 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:24:15.996 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:24:15.996 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:24:15.996 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:24:15.996 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:24:15.996 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:15.996 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:24:15.997 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:24:15.997 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:24:15.997 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:24:15.997 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:24:15.997 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:24:15.997 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:24:15.997 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:15.997 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:24:15.997 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:15.997 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:24:15.997 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:15.997 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:24:15.997 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:24:15.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:24:15.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:24:15.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:24:15.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:24:15.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:24:15.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:24:15.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:15.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:15.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:15.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:24:15.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:24:15.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:24:15.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:24:15.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:24:16.021 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:16.022 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:16.022 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:16.022 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:24:16.149 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:24:16.150 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:24:16.150 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:16.150 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:16.150 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:16.150 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:24:16.150 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:16.150 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:16.151 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:16.151 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:24:16.151 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:16.151 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:16.151 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:16.152 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:16.153 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:16.153 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:16.200 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 22 2024-09-11T06:24:16.200 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:16.200 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:16.200 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:16.200 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=89, tag_tid=17, entry_tid=14245], [object_number=88, tag_tid=17, entry_tid=14244], [object_number=91, tag_tid=17, entry_tid=14243], [object_number=90, tag_tid=17, entry_tid=14242]]], state=connected] 2024-09-11T06:24:16.203 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:16.205 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:16.206 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:24:16.206 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:16.206 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:16.206 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:16.206 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:24:16.206 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:16.206 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:16.206 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:16.206 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:24:16.206 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:16.206 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:16.206 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:16.207 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:16.207 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:16.207 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:16.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 22 2024-09-11T06:24:16.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:16.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:16.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:16.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=89, tag_tid=17, entry_tid=14245], [object_number=88, tag_tid=17, entry_tid=14244], [object_number=91, tag_tid=17, entry_tid=14243], [object_number=90, tag_tid=17, entry_tid=14242]]], state=connected] 2024-09-11T06:24:16.256 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:16.258 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=89, tag_tid=17, entry_tid=14245' 2024-09-11T06:24:16.259 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=89, tag_tid=17, entry_tid=14245' 2024-09-11T06:24:16.259 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=89, tag_tid=17, entry_tid=14245' '!=' 'object_number=88, tag_tid=17, entry_tid=14156' 2024-09-11T06:24:16.259 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:24:16.259 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=89, tag_tid=17, entry_tid=14245' = 'object_number=88, tag_tid=17, entry_tid=14156' 2024-09-11T06:24:16.259 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=89, tag_tid=17, entry_tid=14245' 2024-09-11T06:24:16.259 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:16.259 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:16.259 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:16.260 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:24:16.261 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=89, tag_tid=17, entry_tid=14245' 2024-09-11T06:24:16.261 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:16.261 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:16.261 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:24:16.262 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:16.262 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:16.262 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:16.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:24:16.264 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=89, tag_tid=17, entry_tid=14245' 2024-09-11T06:24:16.264 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:16.264 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:16.265 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=14245 2024-09-11T06:24:16.265 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 14245 2024-09-11T06:24:16.265 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:24:16.265 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:24:16.266 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:24:16.467 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:24:16.467 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:16.467 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:24:16.467 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:24:16.467 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:24:16.467 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:24:16.467 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:24:16.467 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:24:16.468 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:24:16.468 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:16.468 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:16.468 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:24:16.468 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:24:16.468 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:24:16.468 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:24:16.468 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:24:16.469 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:24:16.473 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:24:16.473 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:24:16.473 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:24:16.473 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:16.473 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:24:16.473 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:24:16.473 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:24:16.473 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:24:16.473 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:24:16.473 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:24:16.473 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:24:16.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:16.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:24:16.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:16.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:24:16.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:16.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:24:16.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:24:16.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:24:16.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:24:16.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:24:16.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:24:16.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:24:16.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:24:16.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:16.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:16.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:16.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:24:16.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:24:16.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:24:16.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:24:16.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:24:16.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:16.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:16.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:16.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:24:16.631 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:24:16.632 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:24:16.632 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:16.632 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:16.632 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:16.633 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:24:16.633 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:16.633 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:16.633 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:16.633 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:24:16.633 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:16.633 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:16.634 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:16.634 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:16.635 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:16.635 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:16.682 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 22 2024-09-11T06:24:16.682 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:16.682 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:16.682 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:16.682 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=91, tag_tid=17, entry_tid=14331], [object_number=90, tag_tid=17, entry_tid=14330], [object_number=89, tag_tid=17, entry_tid=14329], [object_number=88, tag_tid=17, entry_tid=14328]]], state=connected] 2024-09-11T06:24:16.685 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:16.686 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:16.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:24:16.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:16.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:16.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:16.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:24:16.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:16.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:16.689 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:16.689 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:24:16.689 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:16.689 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:16.689 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:16.690 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:16.690 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:16.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:16.735 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 22 2024-09-11T06:24:16.736 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:16.736 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:16.736 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:16.736 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=91, tag_tid=17, entry_tid=14331], [object_number=90, tag_tid=17, entry_tid=14330], [object_number=89, tag_tid=17, entry_tid=14329], [object_number=88, tag_tid=17, entry_tid=14328]]], state=connected] 2024-09-11T06:24:16.739 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:16.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=91, tag_tid=17, entry_tid=14331' 2024-09-11T06:24:16.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=91, tag_tid=17, entry_tid=14331' 2024-09-11T06:24:16.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=91, tag_tid=17, entry_tid=14331' '!=' 'object_number=89, tag_tid=17, entry_tid=14245' 2024-09-11T06:24:16.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:24:16.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=91, tag_tid=17, entry_tid=14331' = 'object_number=89, tag_tid=17, entry_tid=14245' 2024-09-11T06:24:16.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=91, tag_tid=17, entry_tid=14331' 2024-09-11T06:24:16.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:16.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:16.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:16.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:24:16.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=91, tag_tid=17, entry_tid=14331' 2024-09-11T06:24:16.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:16.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:16.745 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:24:16.745 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:16.745 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:16.745 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:16.745 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:24:16.746 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=91, tag_tid=17, entry_tid=14331' 2024-09-11T06:24:16.746 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:16.746 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:16.747 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=14331 2024-09-11T06:24:16.747 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 14331 2024-09-11T06:24:16.747 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:24:16.747 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:24:16.747 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:24:16.949 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:24:16.949 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:16.949 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:24:16.949 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:24:16.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:24:16.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:24:16.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:24:16.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:24:16.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:24:16.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:16.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:16.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:24:16.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:24:16.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:24:16.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:24:16.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:24:16.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:24:16.952 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:24:16.952 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:24:16.952 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:24:16.953 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:16.953 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:24:16.953 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:24:16.953 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:24:16.953 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:24:16.953 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:24:16.953 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:24:16.953 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:24:16.953 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:16.953 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:24:16.953 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:16.953 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:24:16.953 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:16.955 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:24:16.955 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:24:16.955 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:24:16.955 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:24:16.955 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:24:16.955 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:24:16.955 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:24:16.955 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:24:16.956 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:16.956 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:16.956 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:16.956 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:24:16.956 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:24:16.956 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:24:16.956 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:24:16.956 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:24:16.981 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:16.981 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:16.981 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:16.981 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:24:17.110 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:24:17.113 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:24:17.113 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:17.114 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:17.114 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:17.114 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:24:17.114 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:17.114 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:17.114 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:17.114 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:24:17.114 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:17.114 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:17.114 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:17.114 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:17.114 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:17.114 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:17.369 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 22 2024-09-11T06:24:17.369 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:17.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:17.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:17.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=88, tag_tid=17, entry_tid=14412], [object_number=91, tag_tid=17, entry_tid=14411], [object_number=90, tag_tid=17, entry_tid=14410], [object_number=89, tag_tid=17, entry_tid=14409]]], state=connected] 2024-09-11T06:24:17.373 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:17.374 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:17.376 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:24:17.376 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:17.376 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:17.376 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:17.376 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:24:17.376 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:17.376 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:17.376 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:17.376 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:24:17.376 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:17.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:17.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:17.378 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:17.378 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:17.379 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:17.431 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 22 2024-09-11T06:24:17.431 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:17.431 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:17.431 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:17.431 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=88, tag_tid=17, entry_tid=14412], [object_number=91, tag_tid=17, entry_tid=14411], [object_number=90, tag_tid=17, entry_tid=14410], [object_number=89, tag_tid=17, entry_tid=14409]]], state=connected] 2024-09-11T06:24:17.435 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:17.437 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=88, tag_tid=17, entry_tid=14412' 2024-09-11T06:24:17.437 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=88, tag_tid=17, entry_tid=14412' 2024-09-11T06:24:17.437 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=88, tag_tid=17, entry_tid=14412' '!=' 'object_number=91, tag_tid=17, entry_tid=14331' 2024-09-11T06:24:17.437 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:24:17.437 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=88, tag_tid=17, entry_tid=14412' = 'object_number=91, tag_tid=17, entry_tid=14331' 2024-09-11T06:24:17.438 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=88, tag_tid=17, entry_tid=14412' 2024-09-11T06:24:17.438 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:17.438 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:17.438 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:17.438 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:24:17.439 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=88, tag_tid=17, entry_tid=14412' 2024-09-11T06:24:17.439 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:17.439 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:17.440 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:24:17.441 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:17.441 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:17.441 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:17.442 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:24:17.443 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=88, tag_tid=17, entry_tid=14412' 2024-09-11T06:24:17.443 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:17.443 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:17.444 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=14412 2024-09-11T06:24:17.444 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 14412 2024-09-11T06:24:17.444 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:24:17.445 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:24:17.445 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:24:17.646 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:24:17.646 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:17.646 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:24:17.646 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:24:17.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:24:17.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:24:17.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:24:17.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:24:17.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:24:17.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:17.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:17.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:24:17.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:24:17.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:24:17.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:24:17.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:24:17.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:24:17.655 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:24:17.656 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:24:17.656 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:24:17.656 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:17.656 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:24:17.656 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:24:17.656 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:24:17.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:24:17.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:24:17.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:24:17.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:24:17.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:17.658 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:24:17.658 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:17.658 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:24:17.658 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:17.658 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:24:17.658 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:24:17.659 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:24:17.659 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:24:17.659 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:24:17.659 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:24:17.659 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:24:17.660 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:24:17.660 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:17.660 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:17.660 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:17.660 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:24:17.661 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:24:17.661 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:24:17.661 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:24:17.661 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:24:17.680 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:17.680 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:17.681 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:17.681 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:24:17.810 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:24:17.812 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:24:17.812 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:17.812 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:17.812 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:17.812 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:24:17.812 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:17.812 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:17.812 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:17.813 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:24:17.813 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:17.813 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:17.813 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:17.815 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:17.815 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:17.815 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:17.862 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 22 2024-09-11T06:24:17.862 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:17.862 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:17.862 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:17.862 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=91, tag_tid=17, entry_tid=14531], [object_number=90, tag_tid=17, entry_tid=14530], [object_number=89, tag_tid=17, entry_tid=14529], [object_number=88, tag_tid=17, entry_tid=14528]]], state=connected] 2024-09-11T06:24:17.865 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:17.866 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:17.867 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:24:17.868 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:17.868 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:17.868 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:17.868 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:24:17.868 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:17.868 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:17.868 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:17.868 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:24:17.868 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:17.868 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:17.868 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:17.869 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:17.869 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:17.869 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:17.915 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 22 2024-09-11T06:24:17.916 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:17.916 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:17.916 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:17.916 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=91, tag_tid=17, entry_tid=14531], [object_number=90, tag_tid=17, entry_tid=14530], [object_number=89, tag_tid=17, entry_tid=14529], [object_number=88, tag_tid=17, entry_tid=14528]]], state=connected] 2024-09-11T06:24:17.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:17.921 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=91, tag_tid=17, entry_tid=14531' 2024-09-11T06:24:17.921 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=91, tag_tid=17, entry_tid=14531' 2024-09-11T06:24:17.921 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=91, tag_tid=17, entry_tid=14531' '!=' 'object_number=88, tag_tid=17, entry_tid=14412' 2024-09-11T06:24:17.921 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:24:17.921 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=91, tag_tid=17, entry_tid=14531' = 'object_number=88, tag_tid=17, entry_tid=14412' 2024-09-11T06:24:17.921 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=91, tag_tid=17, entry_tid=14531' 2024-09-11T06:24:17.921 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:17.922 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:17.922 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:17.922 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:24:17.923 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=91, tag_tid=17, entry_tid=14531' 2024-09-11T06:24:17.923 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:17.923 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:17.924 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:24:17.925 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:17.925 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:17.925 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:17.926 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:24:17.926 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=91, tag_tid=17, entry_tid=14531' 2024-09-11T06:24:17.926 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:17.926 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:17.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=14531 2024-09-11T06:24:17.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 14531 2024-09-11T06:24:17.928 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:24:17.928 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:24:17.928 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:24:18.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:24:18.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:18.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:24:18.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:24:18.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:24:18.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:24:18.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:24:18.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:24:18.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:24:18.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:18.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:18.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:24:18.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:24:18.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:24:18.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:24:18.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:24:18.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:24:18.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:24:18.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:24:18.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:24:18.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:18.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:24:18.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:24:18.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:24:18.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:24:18.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:24:18.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:24:18.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:24:18.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:18.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:24:18.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:18.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:24:18.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:18.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:24:18.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:24:18.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:24:18.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:24:18.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:24:18.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:24:18.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:24:18.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:24:18.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:18.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:18.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:18.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:24:18.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:24:18.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:24:18.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:24:18.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:24:18.160 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:18.160 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:18.161 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:18.161 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:24:18.291 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:24:18.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:24:18.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:18.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:18.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:18.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:24:18.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:18.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:18.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:18.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:24:18.293 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:18.293 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:18.293 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:18.294 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:18.294 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:18.294 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:18.349 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 22 2024-09-11T06:24:18.349 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:18.349 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:18.349 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:18.349 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=90, tag_tid=17, entry_tid=14618], [object_number=89, tag_tid=17, entry_tid=14617], [object_number=88, tag_tid=17, entry_tid=14616], [object_number=91, tag_tid=17, entry_tid=14615]]], state=connected] 2024-09-11T06:24:18.352 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:18.354 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:18.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:24:18.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:18.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:18.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:18.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:24:18.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:18.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:18.356 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:18.356 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:24:18.356 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:18.356 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:18.356 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:18.356 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:18.357 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:18.357 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:18.606 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 22 2024-09-11T06:24:18.606 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:18.606 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:18.606 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:18.606 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=90, tag_tid=17, entry_tid=14618], [object_number=89, tag_tid=17, entry_tid=14617], [object_number=88, tag_tid=17, entry_tid=14616], [object_number=91, tag_tid=17, entry_tid=14615]]], state=connected] 2024-09-11T06:24:18.611 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:18.614 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=90, tag_tid=17, entry_tid=14618' 2024-09-11T06:24:18.614 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=90, tag_tid=17, entry_tid=14618' 2024-09-11T06:24:18.614 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=90, tag_tid=17, entry_tid=14618' '!=' 'object_number=91, tag_tid=17, entry_tid=14531' 2024-09-11T06:24:18.614 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:24:18.614 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=90, tag_tid=17, entry_tid=14618' = 'object_number=91, tag_tid=17, entry_tid=14531' 2024-09-11T06:24:18.614 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=90, tag_tid=17, entry_tid=14618' 2024-09-11T06:24:18.614 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:18.615 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:18.615 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:18.616 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:24:18.617 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=90, tag_tid=17, entry_tid=14618' 2024-09-11T06:24:18.617 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:18.617 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:18.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:24:18.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:18.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:18.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:18.621 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:24:18.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=90, tag_tid=17, entry_tid=14618' 2024-09-11T06:24:18.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:18.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:18.624 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=14618 2024-09-11T06:24:18.624 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 14618 2024-09-11T06:24:18.624 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:24:18.624 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:24:18.624 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:24:18.826 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:24:18.826 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:18.826 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:24:18.827 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:24:18.827 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:24:18.827 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:24:18.827 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:24:18.827 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:24:18.827 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:24:18.827 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:18.827 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:18.827 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:24:18.827 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:24:18.827 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:24:18.827 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:24:18.830 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:24:18.830 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:24:18.830 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:24:18.830 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:24:18.830 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:24:18.830 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:18.830 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:24:18.830 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:24:18.830 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:24:18.830 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:24:18.830 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:24:18.830 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:24:18.831 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:24:18.831 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:18.831 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:24:18.831 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:18.831 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:24:18.831 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:18.831 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:24:18.831 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:24:18.831 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:24:18.831 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:24:18.831 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:24:18.831 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:24:18.831 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:24:18.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:24:18.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:18.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:18.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:18.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:24:18.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:24:18.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:24:18.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:24:18.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:24:18.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:18.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:18.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:18.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:24:18.990 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:24:18.991 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:24:18.991 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:18.991 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:18.991 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:18.991 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:24:18.991 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:18.991 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:18.992 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:18.992 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:24:18.992 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:18.992 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:18.992 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:18.994 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:18.994 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:18.994 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:19.046 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 22 2024-09-11T06:24:19.046 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:19.046 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:19.046 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:19.046 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=88, tag_tid=17, entry_tid=14756], [object_number=91, tag_tid=17, entry_tid=14755], [object_number=90, tag_tid=17, entry_tid=14754], [object_number=89, tag_tid=17, entry_tid=14753]]], state=connected] 2024-09-11T06:24:19.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:19.051 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:19.052 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:24:19.052 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:19.052 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:19.052 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:19.052 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:24:19.052 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:19.052 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:19.053 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:19.053 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:24:19.053 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:19.053 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:19.053 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:19.053 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:19.053 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:19.054 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:19.102 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 22 2024-09-11T06:24:19.102 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:19.102 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:19.102 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:19.103 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=88, tag_tid=17, entry_tid=14756], [object_number=91, tag_tid=17, entry_tid=14755], [object_number=90, tag_tid=17, entry_tid=14754], [object_number=89, tag_tid=17, entry_tid=14753]]], state=connected] 2024-09-11T06:24:19.106 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:19.107 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=88, tag_tid=17, entry_tid=14756' 2024-09-11T06:24:19.107 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=88, tag_tid=17, entry_tid=14756' 2024-09-11T06:24:19.108 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=88, tag_tid=17, entry_tid=14756' '!=' 'object_number=90, tag_tid=17, entry_tid=14618' 2024-09-11T06:24:19.108 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:24:19.108 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=88, tag_tid=17, entry_tid=14756' = 'object_number=90, tag_tid=17, entry_tid=14618' 2024-09-11T06:24:19.108 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=88, tag_tid=17, entry_tid=14756' 2024-09-11T06:24:19.108 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:19.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:19.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:19.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:24:19.110 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=88, tag_tid=17, entry_tid=14756' 2024-09-11T06:24:19.110 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:19.110 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:19.110 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:24:19.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:19.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:19.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:19.112 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:24:19.113 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=88, tag_tid=17, entry_tid=14756' 2024-09-11T06:24:19.113 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:19.113 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:19.114 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=14756 2024-09-11T06:24:19.114 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 14756 2024-09-11T06:24:19.114 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:24:19.114 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:24:19.115 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:24:19.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:24:19.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:19.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:24:19.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:24:19.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:24:19.317 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:24:19.317 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:24:19.317 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:24:19.317 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:24:19.317 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:19.317 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:19.317 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:24:19.317 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:24:19.317 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:24:19.317 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:24:19.318 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:24:19.318 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:24:19.321 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:24:19.321 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:24:19.321 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:24:19.321 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:19.321 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:24:19.321 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:24:19.321 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:24:19.321 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:24:19.321 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:24:19.321 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:24:19.322 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:24:19.322 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:19.322 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:24:19.322 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:19.322 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:24:19.322 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:19.323 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:24:19.323 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:24:19.324 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:24:19.324 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:24:19.324 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:24:19.324 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:24:19.324 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:24:19.324 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:24:19.324 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:19.324 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:19.324 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:19.324 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:24:19.325 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:24:19.325 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:24:19.325 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:24:19.325 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:24:19.348 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:19.349 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:19.349 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:19.349 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:24:19.477 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:24:19.479 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:24:19.479 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:19.479 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:19.479 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:19.479 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:24:19.479 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:19.479 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:19.479 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:19.479 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:24:19.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:19.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:19.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:19.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:19.483 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:19.483 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:19.530 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 22 2024-09-11T06:24:19.530 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:19.530 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:19.530 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:19.530 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=88, tag_tid=17, entry_tid=14848], [object_number=91, tag_tid=17, entry_tid=14847], [object_number=90, tag_tid=17, entry_tid=14846], [object_number=89, tag_tid=17, entry_tid=14845]]], state=connected] 2024-09-11T06:24:19.533 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:19.534 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:19.535 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:24:19.536 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:19.536 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:19.536 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:19.536 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:24:19.536 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:19.536 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:19.536 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:19.536 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:24:19.536 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:19.536 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:19.536 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:19.537 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:19.537 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:19.537 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:19.584 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 22 2024-09-11T06:24:19.584 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:19.584 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:19.584 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:19.584 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=88, tag_tid=17, entry_tid=14848], [object_number=91, tag_tid=17, entry_tid=14847], [object_number=90, tag_tid=17, entry_tid=14846], [object_number=89, tag_tid=17, entry_tid=14845]]], state=connected] 2024-09-11T06:24:19.587 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:19.589 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=88, tag_tid=17, entry_tid=14848' 2024-09-11T06:24:19.589 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=88, tag_tid=17, entry_tid=14848' 2024-09-11T06:24:19.589 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=88, tag_tid=17, entry_tid=14848' '!=' 'object_number=88, tag_tid=17, entry_tid=14756' 2024-09-11T06:24:19.589 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:24:19.589 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=88, tag_tid=17, entry_tid=14848' = 'object_number=88, tag_tid=17, entry_tid=14756' 2024-09-11T06:24:19.589 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=88, tag_tid=17, entry_tid=14848' 2024-09-11T06:24:19.590 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:19.590 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:19.590 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:19.590 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:24:19.591 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=88, tag_tid=17, entry_tid=14848' 2024-09-11T06:24:19.591 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:19.591 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:19.592 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:24:19.593 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:19.593 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:19.593 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:19.594 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:24:19.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=88, tag_tid=17, entry_tid=14848' 2024-09-11T06:24:19.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:19.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:19.596 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=14848 2024-09-11T06:24:19.596 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 14848 2024-09-11T06:24:19.596 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:24:19.596 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:24:19.596 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:24:19.798 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:24:19.798 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:19.798 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:24:19.798 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:24:19.798 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:24:19.798 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:24:19.798 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:24:19.798 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:24:19.799 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:24:19.799 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:19.799 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:19.799 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:24:19.799 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:24:19.799 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:24:19.799 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:24:19.799 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:24:19.800 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:24:19.803 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:24:19.803 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:24:19.804 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:24:19.804 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:19.804 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:24:19.804 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:24:19.804 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:24:19.804 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:24:19.804 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:24:19.804 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:24:19.804 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:24:19.804 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:19.804 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:24:19.804 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:19.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:24:19.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:19.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:24:19.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:24:19.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:24:19.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:24:19.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:24:19.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:24:19.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:24:19.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:24:19.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:19.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:19.806 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:19.806 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:24:19.806 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:24:19.806 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:24:19.806 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:24:19.806 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:24:19.831 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:19.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:19.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:19.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:24:19.963 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:24:19.963 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:24:19.964 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:19.964 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:19.964 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:19.964 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:24:19.964 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:19.964 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:19.964 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:19.964 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:24:19.965 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:19.965 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:19.965 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:19.967 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:19.968 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:19.968 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:20.019 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 22 2024-09-11T06:24:20.020 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:20.020 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:20.020 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:20.020 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=91, tag_tid=17, entry_tid=14947], [object_number=90, tag_tid=17, entry_tid=14946], [object_number=89, tag_tid=17, entry_tid=14945], [object_number=88, tag_tid=17, entry_tid=14944]]], state=connected] 2024-09-11T06:24:20.023 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:20.024 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:20.026 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:24:20.026 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:20.026 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:20.026 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:20.026 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:24:20.026 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:20.026 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:20.026 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:20.026 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:24:20.026 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:20.026 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:20.026 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:20.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:20.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:20.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:20.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 22 2024-09-11T06:24:20.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:20.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:20.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:20.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=91, tag_tid=17, entry_tid=14947], [object_number=90, tag_tid=17, entry_tid=14946], [object_number=89, tag_tid=17, entry_tid=14945], [object_number=88, tag_tid=17, entry_tid=14944]]], state=connected] 2024-09-11T06:24:20.078 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:20.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=91, tag_tid=17, entry_tid=14947' 2024-09-11T06:24:20.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=91, tag_tid=17, entry_tid=14947' 2024-09-11T06:24:20.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=91, tag_tid=17, entry_tid=14947' '!=' 'object_number=88, tag_tid=17, entry_tid=14848' 2024-09-11T06:24:20.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:24:20.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=91, tag_tid=17, entry_tid=14947' = 'object_number=88, tag_tid=17, entry_tid=14848' 2024-09-11T06:24:20.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=91, tag_tid=17, entry_tid=14947' 2024-09-11T06:24:20.081 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:20.081 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:20.081 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:20.081 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:24:20.082 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=91, tag_tid=17, entry_tid=14947' 2024-09-11T06:24:20.082 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:20.082 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:20.083 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:24:20.084 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:20.084 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:20.084 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:20.085 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:24:20.085 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=91, tag_tid=17, entry_tid=14947' 2024-09-11T06:24:20.085 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:20.086 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:20.087 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=14947 2024-09-11T06:24:20.087 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 14947 2024-09-11T06:24:20.087 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:24:20.087 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:24:20.087 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:24:20.289 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:24:20.289 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:20.289 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:24:20.289 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:24:20.289 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:24:20.289 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:24:20.289 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:24:20.289 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:24:20.289 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:24:20.289 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:20.290 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:20.290 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:24:20.290 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:24:20.290 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:24:20.290 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:24:20.290 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:24:20.291 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:24:20.293 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:24:20.293 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:24:20.293 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:24:20.293 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:20.293 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:24:20.293 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:24:20.293 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:24:20.293 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:24:20.293 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:24:20.293 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:24:20.293 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:24:20.293 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:20.294 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:24:20.294 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:20.294 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:24:20.294 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:20.295 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:24:20.295 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:24:20.295 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:24:20.295 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:24:20.295 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:24:20.295 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:24:20.295 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:24:20.295 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:24:20.295 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:20.295 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:20.295 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:20.295 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:24:20.296 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:24:20.296 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:24:20.296 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:24:20.296 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:24:20.320 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:20.320 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:20.320 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:20.320 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:24:20.451 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:24:20.452 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:24:20.452 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:20.452 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:20.452 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:20.452 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:24:20.452 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:20.452 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:20.452 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:20.452 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:24:20.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:20.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:20.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:20.456 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:20.456 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:20.456 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:20.510 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 22 2024-09-11T06:24:20.510 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:20.510 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:20.510 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:20.510 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=89, tag_tid=17, entry_tid=15033], [object_number=88, tag_tid=17, entry_tid=15032], [object_number=91, tag_tid=17, entry_tid=15031], [object_number=90, tag_tid=17, entry_tid=15030]]], state=connected] 2024-09-11T06:24:20.514 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:20.515 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:20.516 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:24:20.516 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:20.516 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:20.517 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:20.517 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:24:20.517 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:20.517 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:20.517 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:20.517 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:24:20.517 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:20.517 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:20.517 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:20.517 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:20.518 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:20.518 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:20.566 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 22 2024-09-11T06:24:20.566 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:20.566 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:20.566 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:20.566 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=89, tag_tid=17, entry_tid=15033], [object_number=88, tag_tid=17, entry_tid=15032], [object_number=91, tag_tid=17, entry_tid=15031], [object_number=90, tag_tid=17, entry_tid=15030]]], state=connected] 2024-09-11T06:24:20.569 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:20.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=89, tag_tid=17, entry_tid=15033' 2024-09-11T06:24:20.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=89, tag_tid=17, entry_tid=15033' 2024-09-11T06:24:20.572 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=89, tag_tid=17, entry_tid=15033' '!=' 'object_number=91, tag_tid=17, entry_tid=14947' 2024-09-11T06:24:20.572 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:24:20.572 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=89, tag_tid=17, entry_tid=15033' = 'object_number=91, tag_tid=17, entry_tid=14947' 2024-09-11T06:24:20.572 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=89, tag_tid=17, entry_tid=15033' 2024-09-11T06:24:20.572 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:20.572 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:20.572 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:20.572 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:24:20.573 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=89, tag_tid=17, entry_tid=15033' 2024-09-11T06:24:20.573 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:20.573 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:20.574 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:24:20.575 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:20.575 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:20.575 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:20.576 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:24:20.577 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=89, tag_tid=17, entry_tid=15033' 2024-09-11T06:24:20.577 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:20.577 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:20.578 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=15033 2024-09-11T06:24:20.578 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 15033 2024-09-11T06:24:20.578 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:24:20.578 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:24:20.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:24:20.780 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:24:20.780 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:20.780 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:24:20.780 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:24:20.780 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:24:20.781 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:24:20.781 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:24:20.781 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:24:20.781 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:24:20.781 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:20.781 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:20.781 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:24:20.781 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:24:20.781 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:24:20.781 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:24:20.782 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:24:20.782 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:24:20.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:24:20.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:24:20.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:24:20.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:20.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:24:20.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:24:20.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:24:20.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:24:20.787 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:24:20.787 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:24:20.787 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:24:20.787 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:20.787 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:24:20.787 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:20.787 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:24:20.787 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:20.787 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:24:20.787 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:24:20.787 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:24:20.787 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:24:20.787 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:24:20.788 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:24:20.788 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:24:20.788 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:24:20.788 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:20.788 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:20.788 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:20.788 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:24:20.788 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:24:20.788 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:24:20.788 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:24:20.788 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:24:20.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:20.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:20.812 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:20.812 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:24:20.942 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:24:20.943 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:24:20.943 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:20.943 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:20.943 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:20.943 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:24:20.943 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:20.943 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:20.943 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:20.943 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:24:20.943 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:20.944 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:20.944 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:20.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:20.946 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:20.946 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:20.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 22 2024-09-11T06:24:20.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:20.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:20.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:20.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=90, tag_tid=17, entry_tid=15122], [object_number=89, tag_tid=17, entry_tid=15121], [object_number=88, tag_tid=17, entry_tid=15120], [object_number=91, tag_tid=17, entry_tid=15119]]], state=connected] 2024-09-11T06:24:21.004 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:21.005 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:21.005 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:24:21.005 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:21.005 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:21.005 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:21.005 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:24:21.005 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:21.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:21.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:21.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:24:21.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:21.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:21.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:21.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:21.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:21.007 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:21.052 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 22 2024-09-11T06:24:21.053 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:21.053 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:21.053 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:21.053 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=90, tag_tid=17, entry_tid=15122], [object_number=89, tag_tid=17, entry_tid=15121], [object_number=88, tag_tid=17, entry_tid=15120], [object_number=91, tag_tid=17, entry_tid=15119]]], state=connected] 2024-09-11T06:24:21.056 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:21.057 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=90, tag_tid=17, entry_tid=15122' 2024-09-11T06:24:21.058 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=90, tag_tid=17, entry_tid=15122' 2024-09-11T06:24:21.058 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=90, tag_tid=17, entry_tid=15122' '!=' 'object_number=89, tag_tid=17, entry_tid=15033' 2024-09-11T06:24:21.058 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:24:21.058 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=90, tag_tid=17, entry_tid=15122' = 'object_number=89, tag_tid=17, entry_tid=15033' 2024-09-11T06:24:21.058 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=90, tag_tid=17, entry_tid=15122' 2024-09-11T06:24:21.059 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:21.059 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:21.059 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:21.059 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:24:21.060 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=90, tag_tid=17, entry_tid=15122' 2024-09-11T06:24:21.060 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:21.060 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:21.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:24:21.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:21.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:21.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:21.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:24:21.063 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=90, tag_tid=17, entry_tid=15122' 2024-09-11T06:24:21.063 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:21.063 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:21.064 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=15122 2024-09-11T06:24:21.065 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 15122 2024-09-11T06:24:21.065 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:24:21.065 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:24:21.065 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:24:21.266 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:24:21.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:21.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:24:21.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:24:21.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:24:21.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:24:21.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:24:21.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:24:21.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:24:21.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:21.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:21.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:24:21.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:24:21.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:24:21.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:24:21.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:24:21.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:24:21.271 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:24:21.271 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:24:21.271 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:24:21.271 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:21.271 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:24:21.271 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:24:21.271 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:24:21.271 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:24:21.271 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:24:21.272 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:24:21.272 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:24:21.272 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:21.272 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:24:21.272 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:21.272 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:24:21.272 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:21.273 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:24:21.273 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:24:21.273 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:24:21.273 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:24:21.274 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:24:21.274 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:24:21.274 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:24:21.274 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:24:21.274 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:21.274 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:21.274 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:21.274 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:24:21.274 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:24:21.274 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:24:21.274 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:24:21.274 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:24:21.297 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:21.298 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:21.298 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:21.298 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:24:21.429 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:24:21.430 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:24:21.430 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:21.430 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:21.430 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:21.431 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:24:21.431 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:21.431 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:21.431 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:21.431 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:24:21.432 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:21.432 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:21.432 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:21.433 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:21.433 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:21.433 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:21.485 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 22 2024-09-11T06:24:21.485 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:21.485 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:21.486 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:21.486 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=90, tag_tid=17, entry_tid=15206], [object_number=89, tag_tid=17, entry_tid=15205], [object_number=88, tag_tid=17, entry_tid=15204], [object_number=91, tag_tid=17, entry_tid=15203]]], state=connected] 2024-09-11T06:24:21.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:21.490 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:21.491 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:24:21.491 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:21.491 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:21.492 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:21.492 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:24:21.492 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:21.492 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:21.492 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:21.492 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:24:21.492 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:21.492 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:21.492 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:21.493 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:21.493 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:21.493 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:21.541 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 22 2024-09-11T06:24:21.541 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:21.541 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:21.541 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:21.541 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=90, tag_tid=17, entry_tid=15206], [object_number=89, tag_tid=17, entry_tid=15205], [object_number=88, tag_tid=17, entry_tid=15204], [object_number=91, tag_tid=17, entry_tid=15203]]], state=connected] 2024-09-11T06:24:21.545 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:21.547 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=90, tag_tid=17, entry_tid=15206' 2024-09-11T06:24:21.547 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=90, tag_tid=17, entry_tid=15206' 2024-09-11T06:24:21.547 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=90, tag_tid=17, entry_tid=15206' '!=' 'object_number=90, tag_tid=17, entry_tid=15122' 2024-09-11T06:24:21.547 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:24:21.547 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=90, tag_tid=17, entry_tid=15206' = 'object_number=90, tag_tid=17, entry_tid=15122' 2024-09-11T06:24:21.547 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=90, tag_tid=17, entry_tid=15206' 2024-09-11T06:24:21.547 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:21.547 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:21.547 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:21.548 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:24:21.549 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=90, tag_tid=17, entry_tid=15206' 2024-09-11T06:24:21.549 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:21.549 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:21.550 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:24:21.550 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:21.550 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:21.551 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:21.551 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:24:21.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=90, tag_tid=17, entry_tid=15206' 2024-09-11T06:24:21.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:21.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:21.553 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=15206 2024-09-11T06:24:21.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 15206 2024-09-11T06:24:21.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:24:21.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:24:21.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:24:21.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:24:21.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:21.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:24:21.756 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:24:21.756 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:24:21.756 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:24:21.756 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:24:21.756 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:24:21.756 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:24:21.756 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:21.756 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:21.756 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:24:21.756 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:24:21.756 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:24:21.756 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:24:21.757 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:24:21.757 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:24:21.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:24:21.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:24:21.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:24:21.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:21.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:24:21.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:24:21.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:24:21.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:24:21.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:24:21.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:24:21.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:24:21.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:21.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:24:21.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:21.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:24:21.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:21.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:24:21.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:24:21.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:24:21.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:24:21.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:24:21.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:24:21.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:24:21.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:24:21.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:21.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:21.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:21.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:24:21.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:24:21.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:24:21.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:24:21.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:24:21.785 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:21.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:21.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:21.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:24:21.923 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:24:21.924 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:24:21.924 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:21.924 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:21.924 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:21.924 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:24:21.924 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:21.924 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:21.924 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:21.924 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:24:21.925 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:21.925 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:21.925 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:21.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:21.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:21.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:21.979 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 22 2024-09-11T06:24:21.979 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:21.979 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:21.979 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:21.979 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=90, tag_tid=17, entry_tid=15286], [object_number=89, tag_tid=17, entry_tid=15285], [object_number=88, tag_tid=17, entry_tid=15284], [object_number=91, tag_tid=17, entry_tid=15283]]], state=connected] 2024-09-11T06:24:21.983 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:21.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:21.985 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:24:21.986 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:21.986 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:21.986 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:21.986 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:24:21.986 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:21.986 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:21.986 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:21.986 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:24:21.986 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:21.986 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:21.986 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:21.987 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:21.987 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:21.987 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:22.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 22 2024-09-11T06:24:22.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:22.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:22.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:22.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=90, tag_tid=17, entry_tid=15286], [object_number=89, tag_tid=17, entry_tid=15285], [object_number=88, tag_tid=17, entry_tid=15284], [object_number=91, tag_tid=17, entry_tid=15283]]], state=connected] 2024-09-11T06:24:22.037 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:22.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=90, tag_tid=17, entry_tid=15286' 2024-09-11T06:24:22.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=90, tag_tid=17, entry_tid=15286' 2024-09-11T06:24:22.040 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=90, tag_tid=17, entry_tid=15286' '!=' 'object_number=90, tag_tid=17, entry_tid=15206' 2024-09-11T06:24:22.040 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:24:22.040 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=90, tag_tid=17, entry_tid=15286' = 'object_number=90, tag_tid=17, entry_tid=15206' 2024-09-11T06:24:22.040 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=90, tag_tid=17, entry_tid=15286' 2024-09-11T06:24:22.040 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:22.040 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:22.040 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:22.040 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:24:22.041 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=90, tag_tid=17, entry_tid=15286' 2024-09-11T06:24:22.041 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:22.041 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:22.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:24:22.043 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:22.043 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:22.043 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:22.044 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:24:22.045 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=90, tag_tid=17, entry_tid=15286' 2024-09-11T06:24:22.045 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:22.045 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:22.046 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=15286 2024-09-11T06:24:22.046 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 15286 2024-09-11T06:24:22.046 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:24:22.046 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:24:22.046 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:24:22.248 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:24:22.248 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:22.248 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:24:22.248 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:24:22.248 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:24:22.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:24:22.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:24:22.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:24:22.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:24:22.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:22.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:22.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:24:22.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:24:22.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:24:22.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:24:22.250 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:24:22.250 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:24:22.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:24:22.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:24:22.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:24:22.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:22.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:24:22.252 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:24:22.252 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:24:22.252 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:24:22.252 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:24:22.252 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:24:22.252 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:24:22.252 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:22.252 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:24:22.252 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:22.252 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:24:22.252 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:22.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:24:22.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:24:22.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:24:22.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:24:22.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:24:22.254 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:24:22.254 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:24:22.254 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:24:22.254 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:22.254 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:22.254 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:22.254 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:24:22.254 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:24:22.254 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:24:22.254 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:24:22.254 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:24:22.277 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:22.278 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:22.278 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:22.278 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:24:22.411 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:24:22.412 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:24:22.412 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:22.412 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:22.412 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:22.412 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:24:22.412 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:22.413 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:22.413 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:22.413 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:24:22.413 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:22.413 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:22.413 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:22.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:22.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:22.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:22.467 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 22 2024-09-11T06:24:22.467 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:22.467 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:22.467 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:22.467 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=91, tag_tid=17, entry_tid=15371], [object_number=90, tag_tid=17, entry_tid=15370], [object_number=89, tag_tid=17, entry_tid=15369], [object_number=88, tag_tid=17, entry_tid=15368]]], state=connected] 2024-09-11T06:24:22.470 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:22.471 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:22.473 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:24:22.473 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:22.473 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:22.473 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:22.473 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:24:22.473 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:22.473 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:22.473 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:22.473 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:24:22.473 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:22.473 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:22.473 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:22.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:22.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:22.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:22.521 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 22 2024-09-11T06:24:22.521 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:22.521 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:22.521 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:22.521 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=91, tag_tid=17, entry_tid=15371], [object_number=90, tag_tid=17, entry_tid=15370], [object_number=89, tag_tid=17, entry_tid=15369], [object_number=88, tag_tid=17, entry_tid=15368]]], state=connected] 2024-09-11T06:24:22.524 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:22.526 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=91, tag_tid=17, entry_tid=15371' 2024-09-11T06:24:22.527 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=91, tag_tid=17, entry_tid=15371' 2024-09-11T06:24:22.527 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=91, tag_tid=17, entry_tid=15371' '!=' 'object_number=90, tag_tid=17, entry_tid=15286' 2024-09-11T06:24:22.527 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:24:22.527 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=91, tag_tid=17, entry_tid=15371' = 'object_number=90, tag_tid=17, entry_tid=15286' 2024-09-11T06:24:22.527 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=91, tag_tid=17, entry_tid=15371' 2024-09-11T06:24:22.527 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:22.527 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:22.527 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:22.528 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:24:22.528 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=91, tag_tid=17, entry_tid=15371' 2024-09-11T06:24:22.528 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:22.529 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:22.529 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:24:22.530 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:22.530 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:22.530 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:22.531 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:24:22.532 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=91, tag_tid=17, entry_tid=15371' 2024-09-11T06:24:22.532 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:22.532 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:22.533 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=15371 2024-09-11T06:24:22.533 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 15371 2024-09-11T06:24:22.533 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:24:22.533 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:24:22.533 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:24:22.735 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:24:22.735 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:22.735 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:24:22.735 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:24:22.735 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:24:22.735 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:24:22.735 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:24:22.735 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:24:22.735 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:24:22.736 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:22.736 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:22.736 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:24:22.736 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:24:22.736 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:24:22.736 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:24:22.736 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:24:22.737 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:24:22.740 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:24:22.740 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:24:22.741 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:24:22.741 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:22.741 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:24:22.741 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:24:22.741 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:24:22.741 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:24:22.741 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:24:22.741 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:24:22.741 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:24:22.741 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:22.741 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:24:22.741 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:22.741 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:24:22.741 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:22.741 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:24:22.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:24:22.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:24:22.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:24:22.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:24:22.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:24:22.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:24:22.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:24:22.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:22.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:22.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:22.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:24:22.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:24:22.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:24:22.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:24:22.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:24:22.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:22.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:22.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:22.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:24:22.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:24:22.905 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:24:22.906 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:22.906 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:22.906 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:22.906 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:24:22.906 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:22.906 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:22.906 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:22.906 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:24:22.907 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:22.907 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:22.907 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:22.908 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:22.908 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:22.909 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:22.954 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 22 2024-09-11T06:24:22.954 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:22.954 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:22.954 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:22.954 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=91, tag_tid=17, entry_tid=15459], [object_number=90, tag_tid=17, entry_tid=15458], [object_number=89, tag_tid=17, entry_tid=15457], [object_number=88, tag_tid=17, entry_tid=15456]]], state=connected] 2024-09-11T06:24:22.957 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:22.959 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:22.960 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:24:22.960 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:22.960 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:22.960 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:22.960 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:24:22.960 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:22.960 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:22.960 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:22.961 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:24:22.961 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:22.961 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:22.961 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:22.961 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:22.962 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:22.962 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:23.007 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 22 2024-09-11T06:24:23.007 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:23.008 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:23.008 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:23.008 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=91, tag_tid=17, entry_tid=15459], [object_number=90, tag_tid=17, entry_tid=15458], [object_number=89, tag_tid=17, entry_tid=15457], [object_number=88, tag_tid=17, entry_tid=15456]]], state=connected] 2024-09-11T06:24:23.011 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:23.012 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=91, tag_tid=17, entry_tid=15459' 2024-09-11T06:24:23.012 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=91, tag_tid=17, entry_tid=15459' 2024-09-11T06:24:23.013 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=91, tag_tid=17, entry_tid=15459' '!=' 'object_number=91, tag_tid=17, entry_tid=15371' 2024-09-11T06:24:23.013 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:24:23.013 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=91, tag_tid=17, entry_tid=15459' = 'object_number=91, tag_tid=17, entry_tid=15371' 2024-09-11T06:24:23.013 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=91, tag_tid=17, entry_tid=15459' 2024-09-11T06:24:23.013 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:23.013 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:23.014 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:23.014 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:24:23.014 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=91, tag_tid=17, entry_tid=15459' 2024-09-11T06:24:23.015 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:23.015 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:23.016 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:24:23.016 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:23.016 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:23.016 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:23.017 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:24:23.018 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=91, tag_tid=17, entry_tid=15459' 2024-09-11T06:24:23.018 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:23.018 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:23.019 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=15459 2024-09-11T06:24:23.019 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 15459 2024-09-11T06:24:23.020 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:24:23.020 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:24:23.020 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:24:23.221 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:24:23.222 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:23.222 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:24:23.222 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:24:23.222 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:24:23.222 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:24:23.222 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:24:23.222 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:24:23.222 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:24:23.222 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:23.222 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:23.222 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:24:23.222 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:24:23.222 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:24:23.223 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:24:23.223 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:24:23.223 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:24:23.225 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:24:23.225 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:24:23.225 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:24:23.225 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:23.225 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:24:23.226 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:24:23.226 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:24:23.226 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:24:23.226 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:24:23.226 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:24:23.226 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:24:23.226 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:23.226 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:24:23.226 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:23.226 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:24:23.226 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:23.227 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:24:23.227 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:24:23.227 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:24:23.227 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:24:23.227 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:24:23.227 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:24:23.227 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:24:23.228 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:24:23.228 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:23.228 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:23.228 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:23.228 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:24:23.228 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:24:23.228 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:24:23.228 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:24:23.228 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:24:23.252 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:23.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:23.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:23.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:24:23.383 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:24:23.383 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:24:23.383 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:23.384 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:23.384 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:23.384 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:24:23.384 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:23.384 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:23.384 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:23.384 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:24:23.385 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:23.385 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:23.385 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:23.387 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:23.387 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:23.388 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:23.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 22 2024-09-11T06:24:23.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:23.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:23.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:23.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=90, tag_tid=17, entry_tid=15558], [object_number=89, tag_tid=17, entry_tid=15557], [object_number=88, tag_tid=17, entry_tid=15556], [object_number=91, tag_tid=17, entry_tid=15555]]], state=connected] 2024-09-11T06:24:23.651 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:23.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:23.654 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:24:23.654 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:23.654 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:23.654 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:23.654 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:24:23.654 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:23.654 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:23.654 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:23.654 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:24:23.655 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:23.655 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:23.655 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:23.655 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:23.656 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:23.656 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:23.707 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 22 2024-09-11T06:24:23.707 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:23.707 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:23.707 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:23.707 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=90, tag_tid=17, entry_tid=15558], [object_number=89, tag_tid=17, entry_tid=15557], [object_number=88, tag_tid=17, entry_tid=15556], [object_number=91, tag_tid=17, entry_tid=15555]]], state=connected] 2024-09-11T06:24:23.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:23.712 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=90, tag_tid=17, entry_tid=15558' 2024-09-11T06:24:23.712 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=90, tag_tid=17, entry_tid=15558' 2024-09-11T06:24:23.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=90, tag_tid=17, entry_tid=15558' '!=' 'object_number=91, tag_tid=17, entry_tid=15459' 2024-09-11T06:24:23.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:24:23.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=90, tag_tid=17, entry_tid=15558' = 'object_number=91, tag_tid=17, entry_tid=15459' 2024-09-11T06:24:23.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=90, tag_tid=17, entry_tid=15558' 2024-09-11T06:24:23.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:23.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:23.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:23.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:24:23.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=90, tag_tid=17, entry_tid=15558' 2024-09-11T06:24:23.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:23.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:23.716 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:24:23.716 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:23.716 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:23.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:23.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:24:23.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=90, tag_tid=17, entry_tid=15558' 2024-09-11T06:24:23.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:23.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:23.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=15558 2024-09-11T06:24:23.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 15558 2024-09-11T06:24:23.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:24:23.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:24:23.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:24:23.922 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:24:23.922 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:23.922 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:24:23.922 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:24:23.922 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:24:23.922 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:24:23.922 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:24:23.923 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:24:23.923 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:24:23.923 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:23.923 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:23.923 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:24:23.923 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:24:23.923 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:24:23.923 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:24:23.923 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:24:23.925 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:24:23.926 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:24:23.926 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:24:23.926 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:24:23.926 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:23.926 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:24:23.926 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:24:23.926 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:24:23.926 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:24:23.926 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:24:23.926 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:24:23.926 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:24:23.926 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:23.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:24:23.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:23.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:24:23.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:23.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:24:23.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:24:23.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:24:23.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:24:23.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:24:23.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:24:23.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:24:23.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:24:23.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:23.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:23.928 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:23.928 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:24:23.928 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:24:23.928 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:24:23.928 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:24:23.928 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:24:23.952 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:23.952 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:23.952 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:23.952 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:24:24.082 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:24:24.083 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:24:24.083 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:24.083 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:24.083 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:24.083 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:24:24.083 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:24.083 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:24.083 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:24.083 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:24:24.083 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:24.084 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:24.084 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:24.086 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:24.087 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:24.087 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:24.139 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 22 2024-09-11T06:24:24.139 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:24.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:24.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:24.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=91, tag_tid=17, entry_tid=15703], [object_number=90, tag_tid=17, entry_tid=15702], [object_number=89, tag_tid=17, entry_tid=15701], [object_number=88, tag_tid=17, entry_tid=15700]]], state=connected] 2024-09-11T06:24:24.143 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:24.144 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:24.147 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:24:24.147 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:24.147 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:24.147 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:24.147 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:24:24.148 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:24.148 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:24.148 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:24.148 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:24:24.148 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:24.149 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:24.149 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:24.149 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:24.149 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:24.149 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:24.196 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 22 2024-09-11T06:24:24.197 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:24.197 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:24.197 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:24.197 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=91, tag_tid=17, entry_tid=15703], [object_number=90, tag_tid=17, entry_tid=15702], [object_number=89, tag_tid=17, entry_tid=15701], [object_number=88, tag_tid=17, entry_tid=15700]]], state=connected] 2024-09-11T06:24:24.200 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:24.201 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=91, tag_tid=17, entry_tid=15703' 2024-09-11T06:24:24.202 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=91, tag_tid=17, entry_tid=15703' 2024-09-11T06:24:24.202 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=91, tag_tid=17, entry_tid=15703' '!=' 'object_number=90, tag_tid=17, entry_tid=15558' 2024-09-11T06:24:24.202 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:24:24.202 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=91, tag_tid=17, entry_tid=15703' = 'object_number=90, tag_tid=17, entry_tid=15558' 2024-09-11T06:24:24.202 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=91, tag_tid=17, entry_tid=15703' 2024-09-11T06:24:24.202 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:24.203 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:24.203 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:24.203 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:24:24.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=91, tag_tid=17, entry_tid=15703' 2024-09-11T06:24:24.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:24.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:24.205 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:24:24.206 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:24.206 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:24.206 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:24.207 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:24:24.207 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=91, tag_tid=17, entry_tid=15703' 2024-09-11T06:24:24.207 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:24.207 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:24.209 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=15703 2024-09-11T06:24:24.209 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 15703 2024-09-11T06:24:24.209 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:24:24.209 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:24:24.209 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:24:24.413 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:24:24.413 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:24.413 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:24:24.413 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:24:24.413 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:24:24.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:24:24.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:24:24.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:24:24.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:24:24.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:24.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:24.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:24:24.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:24:24.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:24:24.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:24:24.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:24:24.416 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:24:24.421 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:24:24.421 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:24:24.422 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:24:24.422 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:24.422 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:24:24.422 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:24:24.422 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:24:24.423 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:24:24.423 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:24:24.423 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:24:24.423 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:24:24.423 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:24.423 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:24:24.424 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:24.424 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:24:24.424 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:24.424 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:24:24.424 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:24:24.424 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:24:24.424 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:24:24.425 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:24:24.425 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:24:24.425 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:24:24.425 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:24:24.425 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:24.425 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:24.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:24.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:24:24.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:24:24.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:24:24.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:24:24.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:24:24.441 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:24.441 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:24.441 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:24.442 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:24:24.569 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:24:24.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:24:24.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:24.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:24.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:24.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:24:24.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:24.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:24.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:24.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:24:24.572 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:24.572 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:24.572 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:24.573 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:24.574 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:24.574 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:24.628 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 22 2024-09-11T06:24:24.628 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:24.628 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:24.629 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:24.629 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=89, tag_tid=17, entry_tid=15797], [object_number=88, tag_tid=17, entry_tid=15796], [object_number=91, tag_tid=17, entry_tid=15795], [object_number=90, tag_tid=17, entry_tid=15794]]], state=connected] 2024-09-11T06:24:24.632 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:24.633 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:24.634 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:24:24.634 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:24.634 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:24.634 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:24.635 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:24:24.635 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:24.635 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:24.635 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:24.635 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:24:24.635 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:24.635 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:24.635 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:24.636 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:24.636 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:24.636 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:24.683 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 22 2024-09-11T06:24:24.683 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:24.683 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:24.683 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:24.683 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=89, tag_tid=17, entry_tid=15797], [object_number=88, tag_tid=17, entry_tid=15796], [object_number=91, tag_tid=17, entry_tid=15795], [object_number=90, tag_tid=17, entry_tid=15794]]], state=connected] 2024-09-11T06:24:24.686 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:24.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=89, tag_tid=17, entry_tid=15797' 2024-09-11T06:24:24.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=89, tag_tid=17, entry_tid=15797' 2024-09-11T06:24:24.689 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=89, tag_tid=17, entry_tid=15797' '!=' 'object_number=91, tag_tid=17, entry_tid=15703' 2024-09-11T06:24:24.689 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:24:24.689 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=89, tag_tid=17, entry_tid=15797' = 'object_number=91, tag_tid=17, entry_tid=15703' 2024-09-11T06:24:24.689 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=89, tag_tid=17, entry_tid=15797' 2024-09-11T06:24:24.689 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:24.689 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:24.689 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:24.689 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:24:24.690 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=89, tag_tid=17, entry_tid=15797' 2024-09-11T06:24:24.690 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:24.690 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:24.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:24:24.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:24.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:24.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:24.693 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:24:24.694 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=89, tag_tid=17, entry_tid=15797' 2024-09-11T06:24:24.694 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:24.694 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:24.695 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=15797 2024-09-11T06:24:24.695 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 15797 2024-09-11T06:24:24.695 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:24:24.695 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:24:24.695 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:24:24.897 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:24:24.897 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:24.897 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:24:24.897 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:24:24.897 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:24:24.897 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:24:24.897 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:24:24.897 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:24:24.897 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:24:24.898 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:24.898 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:24.898 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:24:24.898 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:24:24.898 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:24:24.898 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:24:24.898 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:24:24.898 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:24:24.901 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:24:24.901 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:24:24.901 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:24:24.901 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:24.901 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:24:24.901 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:24:24.902 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:24:24.902 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:24:24.902 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:24:24.902 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:24:24.902 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:24:24.902 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:24.902 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:24:24.902 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:24.902 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:24:24.902 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:24.902 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:24:24.902 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:24:24.902 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:24:24.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:24:24.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:24:24.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:24:24.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:24:24.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:24:24.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:24.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:24.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:24.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:24:24.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:24:24.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:24:24.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:24:24.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:24:24.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:24.928 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:24.928 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:24.928 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:24:25.060 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:24:25.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:24:25.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:25.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:25.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:25.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:24:25.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:25.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:25.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:25.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:24:25.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:25.063 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:25.063 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:25.063 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:25.064 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:25.064 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:25.113 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 22 2024-09-11T06:24:25.113 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:25.113 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:25.113 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:25.113 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=90, tag_tid=17, entry_tid=15910], [object_number=89, tag_tid=17, entry_tid=15909], [object_number=88, tag_tid=17, entry_tid=15908], [object_number=91, tag_tid=17, entry_tid=15907]]], state=connected] 2024-09-11T06:24:25.116 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:25.118 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:25.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:24:25.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:25.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:25.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:25.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:24:25.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:25.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:25.120 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:25.120 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:24:25.120 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:25.120 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:25.120 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:25.120 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:25.121 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:25.121 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:25.167 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 22 2024-09-11T06:24:25.168 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:25.168 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:25.168 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:25.168 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=90, tag_tid=17, entry_tid=15910], [object_number=89, tag_tid=17, entry_tid=15909], [object_number=88, tag_tid=17, entry_tid=15908], [object_number=91, tag_tid=17, entry_tid=15907]]], state=connected] 2024-09-11T06:24:25.171 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:25.173 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=90, tag_tid=17, entry_tid=15910' 2024-09-11T06:24:25.173 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=90, tag_tid=17, entry_tid=15910' 2024-09-11T06:24:25.174 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=90, tag_tid=17, entry_tid=15910' '!=' 'object_number=89, tag_tid=17, entry_tid=15797' 2024-09-11T06:24:25.174 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:24:25.174 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=90, tag_tid=17, entry_tid=15910' = 'object_number=89, tag_tid=17, entry_tid=15797' 2024-09-11T06:24:25.174 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=90, tag_tid=17, entry_tid=15910' 2024-09-11T06:24:25.174 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:25.174 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:25.174 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:25.174 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:24:25.175 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=90, tag_tid=17, entry_tid=15910' 2024-09-11T06:24:25.175 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:25.175 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:25.176 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:24:25.177 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:25.177 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:25.177 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:25.177 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:24:25.178 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=90, tag_tid=17, entry_tid=15910' 2024-09-11T06:24:25.178 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:25.178 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:25.179 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=15910 2024-09-11T06:24:25.180 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 15910 2024-09-11T06:24:25.180 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:24:25.180 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:24:25.180 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:24:25.382 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:24:25.382 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:25.382 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:24:25.382 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:24:25.382 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:24:25.382 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:24:25.382 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:24:25.382 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:24:25.382 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:24:25.382 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:25.383 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:25.383 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:24:25.383 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:24:25.383 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:24:25.383 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:24:25.383 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:24:25.383 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:24:25.387 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:24:25.387 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:24:25.387 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:24:25.387 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:25.387 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:24:25.387 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:24:25.387 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:24:25.387 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:24:25.387 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:24:25.387 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:24:25.387 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:24:25.387 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:25.388 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:24:25.388 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:25.388 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:24:25.388 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:25.388 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:24:25.388 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:24:25.388 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:24:25.388 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:24:25.388 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:24:25.388 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:24:25.388 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:24:25.388 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:24:25.389 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:25.389 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:25.389 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:25.389 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:24:25.389 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:24:25.389 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:24:25.389 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:24:25.389 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:24:25.411 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:25.412 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:25.412 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:25.412 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:24:25.547 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:24:25.548 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:24:25.548 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:25.548 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:25.548 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:25.548 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:24:25.548 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:25.548 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:25.548 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:25.548 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:24:25.548 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:25.549 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:25.549 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:25.550 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:25.550 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:25.550 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:25.606 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 22 2024-09-11T06:24:25.607 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:25.607 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:25.607 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:25.607 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=91, tag_tid=17, entry_tid=16023], [object_number=90, tag_tid=17, entry_tid=16022], [object_number=89, tag_tid=17, entry_tid=16021], [object_number=88, tag_tid=17, entry_tid=16020]]], state=connected] 2024-09-11T06:24:25.610 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:25.611 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:25.612 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:24:25.612 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:25.612 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:25.613 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:25.613 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:24:25.613 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:25.613 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:25.613 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:25.613 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:24:25.613 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:25.613 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:25.613 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:25.614 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:25.614 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:25.614 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:25.661 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 22 2024-09-11T06:24:25.661 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:25.661 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:25.661 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:25.662 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=91, tag_tid=17, entry_tid=16023], [object_number=90, tag_tid=17, entry_tid=16022], [object_number=89, tag_tid=17, entry_tid=16021], [object_number=88, tag_tid=17, entry_tid=16020]]], state=connected] 2024-09-11T06:24:25.665 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:25.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=91, tag_tid=17, entry_tid=16023' 2024-09-11T06:24:25.668 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=91, tag_tid=17, entry_tid=16023' 2024-09-11T06:24:25.668 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=91, tag_tid=17, entry_tid=16023' '!=' 'object_number=90, tag_tid=17, entry_tid=15910' 2024-09-11T06:24:25.668 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:24:25.668 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=91, tag_tid=17, entry_tid=16023' = 'object_number=90, tag_tid=17, entry_tid=15910' 2024-09-11T06:24:25.668 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=91, tag_tid=17, entry_tid=16023' 2024-09-11T06:24:25.668 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:25.669 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:25.669 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:25.671 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:24:25.671 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=91, tag_tid=17, entry_tid=16023' 2024-09-11T06:24:25.671 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:25.671 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:25.672 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:24:25.672 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:25.672 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:25.672 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:25.675 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:24:25.675 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=91, tag_tid=17, entry_tid=16023' 2024-09-11T06:24:25.675 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:25.676 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:25.676 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=16023 2024-09-11T06:24:25.676 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 16023 2024-09-11T06:24:25.676 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:24:25.676 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:24:25.677 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:24:25.877 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:24:25.877 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:25.877 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:24:25.877 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:24:25.877 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:24:25.877 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:24:25.878 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:24:25.878 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:24:25.878 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:24:25.878 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:25.878 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:25.878 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:24:25.878 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:24:25.878 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:24:25.879 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:24:25.879 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:24:25.883 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:24:25.883 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:24:25.883 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:24:25.883 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:24:25.883 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:25.883 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:24:25.883 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:24:25.884 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:24:25.884 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:24:25.884 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:24:25.884 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:24:25.884 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:24:25.884 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:25.884 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:24:25.884 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:25.884 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:24:25.885 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:25.885 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:24:25.885 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:24:25.885 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:24:25.885 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:24:25.885 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:24:25.885 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:24:25.885 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:24:25.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:24:25.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:25.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:25.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:25.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:24:25.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:24:25.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:24:25.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:24:25.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:24:25.907 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:25.907 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:25.907 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:25.907 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:24:26.032 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:24:26.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:24:26.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:26.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:26.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:26.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:24:26.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:26.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:26.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:26.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:24:26.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:26.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:26.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:26.035 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:26.035 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:26.035 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:26.088 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 22 2024-09-11T06:24:26.088 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:26.088 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:26.088 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:26.088 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=89, tag_tid=17, entry_tid=16129], [object_number=88, tag_tid=17, entry_tid=16128], [object_number=91, tag_tid=17, entry_tid=16127], [object_number=90, tag_tid=17, entry_tid=16126]]], state=connected] 2024-09-11T06:24:26.091 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:26.093 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:26.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:24:26.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:26.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:26.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:26.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:24:26.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:26.095 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:26.095 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:26.095 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:24:26.095 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:26.095 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:26.095 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:26.096 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:26.096 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:26.096 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:26.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 22 2024-09-11T06:24:26.143 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:26.143 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:26.143 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:26.143 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=89, tag_tid=17, entry_tid=16129], [object_number=88, tag_tid=17, entry_tid=16128], [object_number=91, tag_tid=17, entry_tid=16127], [object_number=90, tag_tid=17, entry_tid=16126]]], state=connected] 2024-09-11T06:24:26.146 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:26.148 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=89, tag_tid=17, entry_tid=16129' 2024-09-11T06:24:26.148 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=89, tag_tid=17, entry_tid=16129' 2024-09-11T06:24:26.148 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=89, tag_tid=17, entry_tid=16129' '!=' 'object_number=91, tag_tid=17, entry_tid=16023' 2024-09-11T06:24:26.148 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:24:26.148 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=89, tag_tid=17, entry_tid=16129' = 'object_number=91, tag_tid=17, entry_tid=16023' 2024-09-11T06:24:26.148 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=89, tag_tid=17, entry_tid=16129' 2024-09-11T06:24:26.148 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:26.149 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:26.149 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:26.149 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:24:26.150 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=89, tag_tid=17, entry_tid=16129' 2024-09-11T06:24:26.150 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:26.150 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:26.151 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:24:26.151 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:26.151 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:26.151 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:26.152 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:24:26.153 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=89, tag_tid=17, entry_tid=16129' 2024-09-11T06:24:26.153 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:26.153 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:26.154 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=16129 2024-09-11T06:24:26.155 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 16129 2024-09-11T06:24:26.155 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:24:26.155 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:24:26.155 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:24:26.356 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:24:26.356 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:26.356 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:24:26.357 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:24:26.357 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:24:26.357 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:24:26.357 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:24:26.357 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:24:26.357 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:24:26.357 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:26.357 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:26.357 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:24:26.357 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:24:26.357 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:24:26.357 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:24:26.358 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:24:26.358 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:24:26.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:24:26.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:24:26.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:24:26.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:26.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:24:26.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:24:26.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:24:26.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:24:26.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:24:26.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:24:26.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:24:26.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:26.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:24:26.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:26.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:24:26.362 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:26.363 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:24:26.363 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:24:26.363 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:24:26.363 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:24:26.363 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:24:26.363 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:24:26.363 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:24:26.363 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:24:26.363 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:26.364 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:26.364 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:26.364 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:24:26.364 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:24:26.364 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:24:26.364 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:24:26.364 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:24:26.388 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:26.388 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:26.389 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:26.389 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:24:26.526 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:24:26.527 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:24:26.527 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:26.528 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:26.528 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:26.528 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:24:26.528 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:26.528 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:26.528 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:26.528 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:24:26.528 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:26.529 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:26.529 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:26.530 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:26.530 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:26.531 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:26.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 22 2024-09-11T06:24:26.580 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:26.580 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:26.580 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:26.580 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=88, tag_tid=17, entry_tid=16244], [object_number=91, tag_tid=17, entry_tid=16243], [object_number=90, tag_tid=17, entry_tid=16242], [object_number=89, tag_tid=17, entry_tid=16241]]], state=connected] 2024-09-11T06:24:26.583 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:26.585 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:26.586 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:24:26.586 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:26.586 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:26.586 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:26.586 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:24:26.586 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:26.586 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:26.586 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:26.586 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:24:26.586 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:26.586 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:26.587 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:26.587 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:26.587 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:26.587 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:26.634 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 22 2024-09-11T06:24:26.635 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:26.635 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:26.635 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:26.635 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=88, tag_tid=17, entry_tid=16244], [object_number=91, tag_tid=17, entry_tid=16243], [object_number=90, tag_tid=17, entry_tid=16242], [object_number=89, tag_tid=17, entry_tid=16241]]], state=connected] 2024-09-11T06:24:26.638 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:26.640 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=88, tag_tid=17, entry_tid=16244' 2024-09-11T06:24:26.640 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=88, tag_tid=17, entry_tid=16244' 2024-09-11T06:24:26.640 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=88, tag_tid=17, entry_tid=16244' '!=' 'object_number=89, tag_tid=17, entry_tid=16129' 2024-09-11T06:24:26.640 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:24:26.640 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=88, tag_tid=17, entry_tid=16244' = 'object_number=89, tag_tid=17, entry_tid=16129' 2024-09-11T06:24:26.640 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=88, tag_tid=17, entry_tid=16244' 2024-09-11T06:24:26.640 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:26.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:26.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:26.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:24:26.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=88, tag_tid=17, entry_tid=16244' 2024-09-11T06:24:26.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:26.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:26.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:24:26.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:26.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:26.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:26.645 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:24:26.645 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=88, tag_tid=17, entry_tid=16244' 2024-09-11T06:24:26.645 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:26.645 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:26.646 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=16244 2024-09-11T06:24:26.646 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 16244 2024-09-11T06:24:26.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:24:26.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:24:26.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:24:26.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:24:26.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:26.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:24:26.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:24:26.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:24:26.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:24:26.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:24:26.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:24:26.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:24:26.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:26.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:26.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:24:26.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:24:26.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:24:26.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:24:26.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:24:26.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:24:26.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:24:26.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:24:26.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:24:26.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:26.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:24:26.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:24:26.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:24:26.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:24:26.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:24:26.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:24:26.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:24:26.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:26.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:24:26.854 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:26.854 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:24:26.854 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:26.855 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:24:26.855 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:24:26.855 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:24:26.855 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:24:26.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:24:26.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:24:26.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:24:26.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:24:26.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:26.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:26.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:26.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:24:26.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:24:26.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:24:26.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:24:26.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:24:26.880 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:26.880 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:26.880 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:26.881 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:24:27.010 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:24:27.011 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:24:27.011 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:27.011 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:27.011 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:27.011 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:24:27.012 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:27.012 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:27.012 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:27.012 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:24:27.012 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:27.012 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:27.013 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:27.015 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:27.015 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:27.015 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:27.067 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 22 2024-09-11T06:24:27.067 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:27.067 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:27.067 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:27.068 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=88, tag_tid=17, entry_tid=16356], [object_number=91, tag_tid=17, entry_tid=16355], [object_number=90, tag_tid=17, entry_tid=16354], [object_number=89, tag_tid=17, entry_tid=16353]]], state=connected] 2024-09-11T06:24:27.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:27.072 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:27.073 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:24:27.073 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:27.073 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:27.073 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:27.073 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:24:27.073 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:27.073 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:27.074 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:27.074 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:24:27.074 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:27.074 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:27.074 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:27.074 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:27.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:27.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:27.122 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 22 2024-09-11T06:24:27.122 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:27.124 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:27.124 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:27.124 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=88, tag_tid=17, entry_tid=16356], [object_number=91, tag_tid=17, entry_tid=16355], [object_number=90, tag_tid=17, entry_tid=16354], [object_number=89, tag_tid=17, entry_tid=16353]]], state=connected] 2024-09-11T06:24:27.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:27.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=88, tag_tid=17, entry_tid=16356' 2024-09-11T06:24:27.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=88, tag_tid=17, entry_tid=16356' 2024-09-11T06:24:27.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=88, tag_tid=17, entry_tid=16356' '!=' 'object_number=88, tag_tid=17, entry_tid=16244' 2024-09-11T06:24:27.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:24:27.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=88, tag_tid=17, entry_tid=16356' = 'object_number=88, tag_tid=17, entry_tid=16244' 2024-09-11T06:24:27.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=88, tag_tid=17, entry_tid=16356' 2024-09-11T06:24:27.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:27.128 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:27.128 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:27.128 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:24:27.129 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=88, tag_tid=17, entry_tid=16356' 2024-09-11T06:24:27.129 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:27.129 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:27.130 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:24:27.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:27.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:27.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:27.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:24:27.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=88, tag_tid=17, entry_tid=16356' 2024-09-11T06:24:27.133 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:27.133 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:27.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=16356 2024-09-11T06:24:27.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 16356 2024-09-11T06:24:27.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:24:27.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:24:27.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:24:27.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:24:27.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:27.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:24:27.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:24:27.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:24:27.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:24:27.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:24:27.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:24:27.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:24:27.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:27.337 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:27.337 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:24:27.337 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:24:27.337 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:24:27.337 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:24:27.337 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:24:27.337 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:24:27.341 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:24:27.341 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:24:27.341 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:24:27.341 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:27.341 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:24:27.341 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:24:27.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:24:27.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:24:27.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:24:27.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:24:27.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:24:27.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:27.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:24:27.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:27.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:24:27.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:27.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:24:27.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:24:27.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:24:27.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:24:27.343 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:24:27.343 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:24:27.343 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:24:27.343 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:24:27.343 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:27.343 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:27.343 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:27.343 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:24:27.343 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:24:27.343 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:24:27.343 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:24:27.343 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:24:27.367 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:27.367 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:27.368 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:27.368 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:24:27.491 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:24:27.492 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:24:27.492 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:27.492 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:27.492 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:27.492 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:24:27.492 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:27.492 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:27.492 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:27.493 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:24:27.493 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:27.493 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:27.493 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:27.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:27.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:27.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:27.546 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 22 2024-09-11T06:24:27.547 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:27.547 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:27.547 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:27.547 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=88, tag_tid=17, entry_tid=16456], [object_number=91, tag_tid=17, entry_tid=16455], [object_number=90, tag_tid=17, entry_tid=16454], [object_number=89, tag_tid=17, entry_tid=16453]]], state=connected] 2024-09-11T06:24:27.550 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:27.551 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:27.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:24:27.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:27.553 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:27.553 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:27.553 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:24:27.553 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:27.553 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:27.553 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:27.553 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:24:27.553 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:27.553 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:27.553 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:27.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:27.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:27.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:27.600 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 22 2024-09-11T06:24:27.601 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:27.601 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:27.601 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:27.601 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=88, tag_tid=17, entry_tid=16456], [object_number=91, tag_tid=17, entry_tid=16455], [object_number=90, tag_tid=17, entry_tid=16454], [object_number=89, tag_tid=17, entry_tid=16453]]], state=connected] 2024-09-11T06:24:27.604 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:27.606 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=88, tag_tid=17, entry_tid=16456' 2024-09-11T06:24:27.606 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=88, tag_tid=17, entry_tid=16456' 2024-09-11T06:24:27.606 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=88, tag_tid=17, entry_tid=16456' '!=' 'object_number=88, tag_tid=17, entry_tid=16356' 2024-09-11T06:24:27.606 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:24:27.606 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=88, tag_tid=17, entry_tid=16456' = 'object_number=88, tag_tid=17, entry_tid=16356' 2024-09-11T06:24:27.606 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=88, tag_tid=17, entry_tid=16456' 2024-09-11T06:24:27.606 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:27.606 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:27.607 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:27.607 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:24:27.608 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=88, tag_tid=17, entry_tid=16456' 2024-09-11T06:24:27.608 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:27.608 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:27.609 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:24:27.609 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:27.610 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:27.610 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:27.610 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:24:27.611 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=88, tag_tid=17, entry_tid=16456' 2024-09-11T06:24:27.611 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:27.611 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:27.612 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=16456 2024-09-11T06:24:27.613 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 16456 2024-09-11T06:24:27.613 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:24:27.613 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:24:27.613 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:24:27.815 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:24:27.815 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:27.815 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:24:27.815 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:24:27.815 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:24:27.815 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:24:27.815 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:24:27.815 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:24:27.815 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:24:27.815 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:27.816 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:27.816 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:24:27.816 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:24:27.816 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:24:27.816 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:24:27.819 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:24:27.819 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:24:27.819 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:24:27.819 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:24:27.819 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:24:27.820 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:27.820 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:24:27.820 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:24:27.820 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:24:27.820 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:24:27.820 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:24:27.820 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:24:27.820 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:24:27.820 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:27.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:24:27.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:27.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:24:27.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:27.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:24:27.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:24:27.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:24:27.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:24:27.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:24:27.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:24:27.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:24:27.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:24:27.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:27.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:27.822 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:27.822 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:24:27.822 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:24:27.822 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:24:27.822 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:24:27.822 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:24:27.844 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:27.845 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:27.845 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:27.845 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:24:27.996 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:24:27.997 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:24:27.997 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:27.997 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:27.997 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:27.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:24:27.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:27.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:27.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:27.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:24:27.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:27.999 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:27.999 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:28.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:28.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:28.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:28.047 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 22 2024-09-11T06:24:28.047 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:28.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:28.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:28.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=88, tag_tid=17, entry_tid=16552], [object_number=91, tag_tid=17, entry_tid=16551], [object_number=90, tag_tid=17, entry_tid=16550], [object_number=89, tag_tid=17, entry_tid=16549]]], state=connected] 2024-09-11T06:24:28.051 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:28.052 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:28.053 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:24:28.054 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:28.054 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:28.054 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:28.054 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:24:28.054 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:28.054 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:28.054 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:28.054 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:24:28.054 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:28.054 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:28.054 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:28.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:28.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:28.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:28.101 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 22 2024-09-11T06:24:28.101 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:28.101 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:28.101 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:28.101 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=88, tag_tid=17, entry_tid=16552], [object_number=91, tag_tid=17, entry_tid=16551], [object_number=90, tag_tid=17, entry_tid=16550], [object_number=89, tag_tid=17, entry_tid=16549]]], state=connected] 2024-09-11T06:24:28.105 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:28.107 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=88, tag_tid=17, entry_tid=16552' 2024-09-11T06:24:28.107 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=88, tag_tid=17, entry_tid=16552' 2024-09-11T06:24:28.107 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=88, tag_tid=17, entry_tid=16552' '!=' 'object_number=88, tag_tid=17, entry_tid=16456' 2024-09-11T06:24:28.107 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:24:28.107 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=88, tag_tid=17, entry_tid=16552' = 'object_number=88, tag_tid=17, entry_tid=16456' 2024-09-11T06:24:28.107 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=88, tag_tid=17, entry_tid=16552' 2024-09-11T06:24:28.107 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:28.107 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:28.107 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:28.108 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:24:28.108 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=88, tag_tid=17, entry_tid=16552' 2024-09-11T06:24:28.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:28.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:28.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:24:28.110 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:28.110 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:28.110 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:28.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:24:28.112 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=88, tag_tid=17, entry_tid=16552' 2024-09-11T06:24:28.112 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:28.112 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:28.114 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=16552 2024-09-11T06:24:28.114 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 16552 2024-09-11T06:24:28.114 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:24:28.114 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:24:28.114 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:24:28.315 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:24:28.315 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:28.315 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:24:28.315 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:24:28.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:24:28.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:24:28.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:24:28.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:24:28.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:24:28.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:28.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:28.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:24:28.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:24:28.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:24:28.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:24:28.319 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:24:28.319 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:24:28.319 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:24:28.320 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:24:28.320 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:24:28.320 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:28.320 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:24:28.320 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:24:28.320 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:24:28.320 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:24:28.320 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:24:28.320 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:24:28.320 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:24:28.320 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:28.320 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:24:28.320 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:28.321 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:24:28.321 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:28.321 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:24:28.321 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:24:28.321 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:24:28.321 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:24:28.321 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:24:28.321 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:24:28.321 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:24:28.321 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:24:28.321 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:28.321 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:28.322 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:28.322 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:24:28.322 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:24:28.322 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:24:28.322 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:24:28.322 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:24:28.345 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:28.345 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:28.346 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:28.346 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:24:28.473 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:24:28.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:24:28.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:28.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:28.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:28.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:24:28.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:28.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:28.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:28.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:24:28.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:28.476 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:28.476 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:28.478 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:28.478 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:28.478 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:28.536 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 22 2024-09-11T06:24:28.536 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:28.536 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:28.536 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:28.536 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=89, tag_tid=17, entry_tid=16637], [object_number=88, tag_tid=17, entry_tid=16636], [object_number=91, tag_tid=17, entry_tid=16635], [object_number=90, tag_tid=17, entry_tid=16634]]], state=connected] 2024-09-11T06:24:28.539 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:28.540 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:28.541 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:24:28.541 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:28.541 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:28.541 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:28.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:24:28.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:28.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:28.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:28.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:24:28.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:28.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:28.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:28.543 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:28.543 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:28.544 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:28.589 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 22 2024-09-11T06:24:28.590 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:28.590 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:28.590 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:28.590 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=89, tag_tid=17, entry_tid=16637], [object_number=88, tag_tid=17, entry_tid=16636], [object_number=91, tag_tid=17, entry_tid=16635], [object_number=90, tag_tid=17, entry_tid=16634]]], state=connected] 2024-09-11T06:24:28.593 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:28.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=89, tag_tid=17, entry_tid=16637' 2024-09-11T06:24:28.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=89, tag_tid=17, entry_tid=16637' 2024-09-11T06:24:28.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=89, tag_tid=17, entry_tid=16637' '!=' 'object_number=88, tag_tid=17, entry_tid=16552' 2024-09-11T06:24:28.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:24:28.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=89, tag_tid=17, entry_tid=16637' = 'object_number=88, tag_tid=17, entry_tid=16552' 2024-09-11T06:24:28.596 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=89, tag_tid=17, entry_tid=16637' 2024-09-11T06:24:28.596 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:28.596 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:28.596 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:28.596 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:24:28.597 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=89, tag_tid=17, entry_tid=16637' 2024-09-11T06:24:28.597 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:28.597 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:28.598 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:24:28.599 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:28.599 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:28.599 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:28.600 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:24:28.600 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=89, tag_tid=17, entry_tid=16637' 2024-09-11T06:24:28.600 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:28.600 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:28.602 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=16637 2024-09-11T06:24:28.602 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 16637 2024-09-11T06:24:28.602 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:24:28.602 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:24:28.602 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:24:28.804 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:24:28.804 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:28.804 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:24:28.804 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:24:28.804 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:24:28.804 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:24:28.804 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:24:28.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:24:28.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:24:28.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:28.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:28.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:24:28.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:24:28.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:24:28.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:24:28.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:24:28.806 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:24:28.808 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:24:28.808 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:24:28.808 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:24:28.808 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:28.808 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:24:28.808 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:24:28.808 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:24:28.808 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:24:28.808 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:24:28.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:24:28.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:24:28.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:28.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:24:28.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:28.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:24:28.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:28.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:24:28.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:24:28.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:24:28.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:24:28.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:24:28.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:24:28.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:24:28.810 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:24:28.810 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:28.810 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:28.810 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:28.810 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:24:28.810 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:24:28.810 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:24:28.810 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:24:28.810 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:24:28.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:28.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:28.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:28.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:24:28.964 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:24:28.965 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:24:28.965 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:28.965 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:28.965 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:28.965 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:24:28.965 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:28.965 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:28.966 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:28.966 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:24:28.966 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:28.966 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:28.966 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:28.968 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:28.968 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:28.968 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:29.225 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 22 2024-09-11T06:24:29.225 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:29.225 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:29.225 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:29.225 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=90, tag_tid=17, entry_tid=16722], [object_number=89, tag_tid=17, entry_tid=16721], [object_number=88, tag_tid=17, entry_tid=16720], [object_number=91, tag_tid=17, entry_tid=16719]]], state=connected] 2024-09-11T06:24:29.230 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:29.232 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:29.233 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:24:29.233 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:29.233 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:29.233 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:29.234 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:24:29.234 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:29.234 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:29.234 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:29.234 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:24:29.234 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:29.234 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:29.235 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:29.236 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:29.236 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:29.236 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:29.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 22 2024-09-11T06:24:29.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:29.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:29.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:29.293 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=90, tag_tid=17, entry_tid=16722], [object_number=89, tag_tid=17, entry_tid=16721], [object_number=88, tag_tid=17, entry_tid=16720], [object_number=91, tag_tid=17, entry_tid=16719]]], state=connected] 2024-09-11T06:24:29.296 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:29.298 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=90, tag_tid=17, entry_tid=16722' 2024-09-11T06:24:29.298 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=90, tag_tid=17, entry_tid=16722' 2024-09-11T06:24:29.298 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=90, tag_tid=17, entry_tid=16722' '!=' 'object_number=89, tag_tid=17, entry_tid=16637' 2024-09-11T06:24:29.298 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:24:29.298 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=90, tag_tid=17, entry_tid=16722' = 'object_number=89, tag_tid=17, entry_tid=16637' 2024-09-11T06:24:29.298 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=90, tag_tid=17, entry_tid=16722' 2024-09-11T06:24:29.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:29.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:29.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:29.300 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:24:29.300 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=90, tag_tid=17, entry_tid=16722' 2024-09-11T06:24:29.300 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:29.301 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:29.301 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:24:29.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:29.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:29.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:29.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:24:29.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=90, tag_tid=17, entry_tid=16722' 2024-09-11T06:24:29.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:29.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:29.305 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=16722 2024-09-11T06:24:29.306 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 16722 2024-09-11T06:24:29.306 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:24:29.306 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:24:29.306 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:24:29.508 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:24:29.508 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:29.508 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:24:29.508 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:24:29.508 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:24:29.508 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:24:29.508 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:24:29.508 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:24:29.508 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:24:29.508 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:29.508 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:29.509 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:24:29.509 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:24:29.509 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:24:29.509 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:24:29.509 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:24:29.509 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:24:29.511 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:24:29.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:24:29.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:24:29.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:29.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:24:29.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:24:29.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:24:29.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:24:29.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:24:29.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:24:29.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:24:29.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:29.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:24:29.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:29.513 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:24:29.513 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:29.513 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:24:29.513 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:24:29.513 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:24:29.513 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:24:29.513 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:24:29.513 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:24:29.513 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:24:29.513 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:24:29.513 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:29.513 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:29.513 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:29.514 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:24:29.514 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:24:29.514 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:24:29.514 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:24:29.514 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:24:29.539 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:29.539 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:29.539 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:29.540 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:24:29.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:24:29.674 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:24:29.675 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:29.675 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:29.675 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:29.675 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:24:29.675 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:29.675 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:29.675 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:29.675 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:24:29.676 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:29.676 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:29.676 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:29.678 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:29.678 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:29.678 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:29.725 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 22 2024-09-11T06:24:29.725 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:29.725 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:29.725 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:29.725 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=89, tag_tid=17, entry_tid=16845], [object_number=88, tag_tid=17, entry_tid=16844], [object_number=91, tag_tid=17, entry_tid=16843], [object_number=90, tag_tid=17, entry_tid=16842]]], state=connected] 2024-09-11T06:24:29.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:29.729 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:29.730 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:24:29.731 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:29.731 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:29.731 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:29.731 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:24:29.731 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:29.731 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:29.731 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:29.731 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:24:29.731 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:29.731 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:29.731 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:29.732 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:29.732 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:29.732 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:29.778 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 22 2024-09-11T06:24:29.778 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:29.778 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:29.779 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:29.779 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=89, tag_tid=17, entry_tid=16845], [object_number=88, tag_tid=17, entry_tid=16844], [object_number=91, tag_tid=17, entry_tid=16843], [object_number=90, tag_tid=17, entry_tid=16842]]], state=connected] 2024-09-11T06:24:29.782 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:29.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=89, tag_tid=17, entry_tid=16845' 2024-09-11T06:24:29.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=89, tag_tid=17, entry_tid=16845' 2024-09-11T06:24:29.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=89, tag_tid=17, entry_tid=16845' '!=' 'object_number=90, tag_tid=17, entry_tid=16722' 2024-09-11T06:24:29.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:24:29.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=89, tag_tid=17, entry_tid=16845' = 'object_number=90, tag_tid=17, entry_tid=16722' 2024-09-11T06:24:29.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=89, tag_tid=17, entry_tid=16845' 2024-09-11T06:24:29.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:29.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:29.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:29.785 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:24:29.785 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=89, tag_tid=17, entry_tid=16845' 2024-09-11T06:24:29.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:29.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:29.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:24:29.787 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:29.787 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:29.787 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:29.788 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:24:29.789 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=89, tag_tid=17, entry_tid=16845' 2024-09-11T06:24:29.789 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:29.789 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:29.791 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=16845 2024-09-11T06:24:29.791 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 16845 2024-09-11T06:24:29.791 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:24:29.791 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:24:29.791 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:24:29.992 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:24:29.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:29.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:24:29.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:24:29.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:24:29.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:24:29.994 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:24:29.994 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:24:29.994 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:24:29.994 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:29.994 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:29.994 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:24:29.994 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:24:29.994 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:24:29.994 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:24:29.995 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:24:29.995 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:24:29.996 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:24:29.996 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:24:29.996 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:24:29.997 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:29.997 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:24:29.997 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:24:29.997 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:24:29.997 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:24:29.997 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:24:29.997 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:24:29.997 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:24:29.997 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:29.997 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:24:29.997 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:29.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:24:29.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:30.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:24:30.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:24:30.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:24:30.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:24:30.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:24:30.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:24:30.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:24:30.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:24:30.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:30.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:30.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:30.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:24:30.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:24:30.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:24:30.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:24:30.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:24:30.025 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:30.025 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:30.025 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:30.025 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:24:30.155 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:24:30.156 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:24:30.156 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:30.156 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:30.156 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:30.156 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:24:30.156 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:30.156 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:30.156 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:30.156 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:24:30.156 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:30.157 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:30.157 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:30.159 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:30.159 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:30.159 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:30.206 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 22 2024-09-11T06:24:30.206 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:30.206 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:30.206 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:30.206 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=89, tag_tid=17, entry_tid=16929], [object_number=88, tag_tid=17, entry_tid=16928], [object_number=91, tag_tid=17, entry_tid=16927], [object_number=90, tag_tid=17, entry_tid=16926]]], state=connected] 2024-09-11T06:24:30.209 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:30.211 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:30.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:24:30.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:30.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:30.213 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:30.213 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:24:30.213 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:30.213 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:30.213 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:30.213 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:24:30.213 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:30.213 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:30.213 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:30.214 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:30.214 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:30.214 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:30.260 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 22 2024-09-11T06:24:30.260 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:30.260 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:30.260 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:30.260 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=89, tag_tid=17, entry_tid=16929], [object_number=88, tag_tid=17, entry_tid=16928], [object_number=91, tag_tid=17, entry_tid=16927], [object_number=90, tag_tid=17, entry_tid=16926]]], state=connected] 2024-09-11T06:24:30.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:30.266 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=89, tag_tid=17, entry_tid=16929' 2024-09-11T06:24:30.266 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=89, tag_tid=17, entry_tid=16929' 2024-09-11T06:24:30.266 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=89, tag_tid=17, entry_tid=16929' '!=' 'object_number=89, tag_tid=17, entry_tid=16845' 2024-09-11T06:24:30.266 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:24:30.266 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=89, tag_tid=17, entry_tid=16929' = 'object_number=89, tag_tid=17, entry_tid=16845' 2024-09-11T06:24:30.266 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=89, tag_tid=17, entry_tid=16929' 2024-09-11T06:24:30.266 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:30.266 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:30.266 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:30.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:24:30.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=89, tag_tid=17, entry_tid=16929' 2024-09-11T06:24:30.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:30.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:30.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:24:30.269 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:30.269 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:30.269 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:30.270 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:24:30.271 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=89, tag_tid=17, entry_tid=16929' 2024-09-11T06:24:30.271 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:30.271 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:30.272 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=16929 2024-09-11T06:24:30.272 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 16929 2024-09-11T06:24:30.272 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:24:30.272 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:24:30.273 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:24:30.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:24:30.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:30.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:24:30.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:24:30.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:24:30.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:24:30.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:24:30.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:24:30.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:24:30.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:30.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:30.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:24:30.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:24:30.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:24:30.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:24:30.476 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:24:30.476 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:24:30.479 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:24:30.479 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:24:30.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:24:30.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:30.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:24:30.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:24:30.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:24:30.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:24:30.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:24:30.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:24:30.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:24:30.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:30.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:24:30.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:30.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:24:30.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:30.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:24:30.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:24:30.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:24:30.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:24:30.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:24:30.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:24:30.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:24:30.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:24:30.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:30.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:30.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:30.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:24:30.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:24:30.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:24:30.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:24:30.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:24:30.507 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:30.507 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:30.507 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:30.508 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:24:30.635 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:24:30.636 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:24:30.636 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:30.636 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:30.637 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:30.637 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:24:30.637 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:30.637 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:30.637 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:30.637 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:24:30.637 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:30.637 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:30.637 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:30.638 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:30.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:30.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:30.694 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 22 2024-09-11T06:24:30.694 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:30.694 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:30.695 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:30.695 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=88, tag_tid=17, entry_tid=17016], [object_number=91, tag_tid=17, entry_tid=17015], [object_number=90, tag_tid=17, entry_tid=17014], [object_number=89, tag_tid=17, entry_tid=17013]]], state=connected] 2024-09-11T06:24:30.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:30.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:30.701 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:24:30.701 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:30.701 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:30.701 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:30.701 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:24:30.701 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:30.701 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:30.701 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:30.701 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:24:30.701 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:30.701 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:30.702 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:30.702 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:30.702 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:30.702 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:30.749 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 22 2024-09-11T06:24:30.749 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:30.749 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:30.749 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:30.749 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=88, tag_tid=17, entry_tid=17016], [object_number=91, tag_tid=17, entry_tid=17015], [object_number=90, tag_tid=17, entry_tid=17014], [object_number=89, tag_tid=17, entry_tid=17013]]], state=connected] 2024-09-11T06:24:30.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:30.754 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=88, tag_tid=17, entry_tid=17016' 2024-09-11T06:24:30.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=88, tag_tid=17, entry_tid=17016' 2024-09-11T06:24:30.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=88, tag_tid=17, entry_tid=17016' '!=' 'object_number=89, tag_tid=17, entry_tid=16929' 2024-09-11T06:24:30.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:24:30.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=88, tag_tid=17, entry_tid=17016' = 'object_number=89, tag_tid=17, entry_tid=16929' 2024-09-11T06:24:30.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=88, tag_tid=17, entry_tid=17016' 2024-09-11T06:24:30.756 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:30.756 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:30.756 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:30.756 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:24:30.757 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=88, tag_tid=17, entry_tid=17016' 2024-09-11T06:24:30.757 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:30.757 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:30.757 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:24:30.758 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:30.758 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:30.758 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:30.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:24:30.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=88, tag_tid=17, entry_tid=17016' 2024-09-11T06:24:30.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:30.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:30.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=17016 2024-09-11T06:24:30.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 17016 2024-09-11T06:24:30.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:24:30.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:24:30.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:24:30.963 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:24:30.964 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:30.964 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:24:30.964 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:24:30.964 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:24:30.964 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:24:30.964 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:24:30.964 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:24:30.964 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:24:30.964 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:30.964 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:30.964 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:24:30.965 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:24:30.965 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:24:30.965 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:24:30.965 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:24:30.965 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:24:30.968 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:24:30.968 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:24:30.968 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:24:30.968 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:30.969 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:24:30.969 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:24:30.969 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:24:30.969 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:24:30.969 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:24:30.969 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:24:30.969 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:24:30.969 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:30.969 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:24:30.969 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:30.969 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:24:30.969 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:30.969 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:24:30.969 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:24:30.970 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:24:30.970 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:24:30.970 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:24:30.970 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:24:30.970 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:24:30.970 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:24:30.970 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:30.970 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:30.970 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:30.970 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:24:30.970 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:24:30.970 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:24:30.970 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:24:30.971 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:24:30.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:30.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:30.994 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:30.994 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:24:31.124 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:24:31.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:24:31.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:31.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:31.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:31.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:24:31.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:31.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:31.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:31.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:24:31.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:31.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:31.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:31.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:31.128 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:31.128 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:31.181 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 22 2024-09-11T06:24:31.181 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:31.181 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:31.181 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:31.181 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=88, tag_tid=17, entry_tid=17096], [object_number=91, tag_tid=17, entry_tid=17095], [object_number=90, tag_tid=17, entry_tid=17094], [object_number=89, tag_tid=17, entry_tid=17093]]], state=connected] 2024-09-11T06:24:31.184 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:31.186 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:31.187 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:24:31.187 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:31.187 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:31.187 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:31.188 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:24:31.188 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:31.188 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:31.188 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:31.188 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:24:31.188 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:31.188 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:31.188 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:31.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:31.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:31.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:31.236 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 22 2024-09-11T06:24:31.237 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:31.237 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:31.237 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:31.237 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=88, tag_tid=17, entry_tid=17096], [object_number=91, tag_tid=17, entry_tid=17095], [object_number=90, tag_tid=17, entry_tid=17094], [object_number=89, tag_tid=17, entry_tid=17093]]], state=connected] 2024-09-11T06:24:31.240 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:31.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=88, tag_tid=17, entry_tid=17096' 2024-09-11T06:24:31.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=88, tag_tid=17, entry_tid=17096' 2024-09-11T06:24:31.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=88, tag_tid=17, entry_tid=17096' '!=' 'object_number=88, tag_tid=17, entry_tid=17016' 2024-09-11T06:24:31.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:24:31.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=88, tag_tid=17, entry_tid=17096' = 'object_number=88, tag_tid=17, entry_tid=17016' 2024-09-11T06:24:31.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=88, tag_tid=17, entry_tid=17096' 2024-09-11T06:24:31.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:31.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:31.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:31.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:24:31.244 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=88, tag_tid=17, entry_tid=17096' 2024-09-11T06:24:31.244 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:31.244 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:31.245 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:24:31.245 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:31.245 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:31.245 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:31.246 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:24:31.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=88, tag_tid=17, entry_tid=17096' 2024-09-11T06:24:31.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:31.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:31.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=17096 2024-09-11T06:24:31.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 17096 2024-09-11T06:24:31.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:24:31.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:24:31.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:24:31.451 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:24:31.451 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:31.451 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:24:31.451 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:24:31.451 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:24:31.451 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:24:31.451 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:24:31.451 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:24:31.451 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:24:31.451 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:31.451 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:31.452 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:24:31.452 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:24:31.452 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:24:31.452 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:24:31.452 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:24:31.452 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:24:31.455 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:24:31.455 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:24:31.455 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:24:31.455 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:31.455 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:24:31.455 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:24:31.455 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:24:31.455 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:24:31.455 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:24:31.455 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:24:31.455 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:24:31.456 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:31.456 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:24:31.456 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:31.456 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:24:31.456 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:31.456 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:24:31.456 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:24:31.456 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:24:31.456 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:24:31.456 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:24:31.456 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:24:31.456 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:24:31.456 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:24:31.457 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:31.457 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:31.457 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:31.457 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:24:31.457 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:24:31.457 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:24:31.457 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:24:31.457 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:24:31.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:31.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:31.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:31.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:24:31.618 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:24:31.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:24:31.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:31.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:31.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:31.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:24:31.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:31.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:31.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:31.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:24:31.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:31.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:31.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:31.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:31.621 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:31.621 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:31.683 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 22 2024-09-11T06:24:31.683 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:31.683 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:31.683 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:31.683 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=93, tag_tid=17, entry_tid=17125], [object_number=92, tag_tid=17, entry_tid=17124], [object_number=95, tag_tid=17, entry_tid=17123], [object_number=90, tag_tid=17, entry_tid=17122]]], state=connected] 2024-09-11T06:24:31.686 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:31.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:31.689 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:24:31.689 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:31.689 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:31.689 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:31.689 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:24:31.689 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:31.690 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:31.690 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:31.690 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:24:31.690 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:31.690 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:31.690 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:31.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:31.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:31.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:31.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 22 2024-09-11T06:24:31.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:31.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:31.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:31.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=93, tag_tid=17, entry_tid=17125], [object_number=92, tag_tid=17, entry_tid=17124], [object_number=95, tag_tid=17, entry_tid=17123], [object_number=90, tag_tid=17, entry_tid=17122]]], state=connected] 2024-09-11T06:24:31.741 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:31.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=93, tag_tid=17, entry_tid=17125' 2024-09-11T06:24:31.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=93, tag_tid=17, entry_tid=17125' 2024-09-11T06:24:31.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=93, tag_tid=17, entry_tid=17125' '!=' 'object_number=88, tag_tid=17, entry_tid=17096' 2024-09-11T06:24:31.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:24:31.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=93, tag_tid=17, entry_tid=17125' = 'object_number=88, tag_tid=17, entry_tid=17096' 2024-09-11T06:24:31.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=93, tag_tid=17, entry_tid=17125' 2024-09-11T06:24:31.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:31.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:31.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:31.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:24:31.745 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=17, entry_tid=17125' 2024-09-11T06:24:31.745 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:31.745 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:31.746 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:24:31.747 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:31.747 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:31.747 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:31.748 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:24:31.748 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=17, entry_tid=17125' 2024-09-11T06:24:31.749 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:31.749 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:31.750 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=17125 2024-09-11T06:24:31.750 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 17125 2024-09-11T06:24:31.750 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:24:31.750 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:24:31.750 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:24:31.952 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:24:31.952 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:31.952 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:24:31.952 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:24:31.952 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:24:31.952 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:24:31.952 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:24:31.953 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:24:31.953 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:24:31.953 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:31.953 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:31.953 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:24:31.953 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:24:31.953 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:24:31.953 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:24:31.953 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:24:31.954 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:24:31.957 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:24:31.957 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:24:31.957 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:24:31.957 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:31.957 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:24:31.957 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:24:31.957 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:24:31.957 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:24:31.957 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:24:31.957 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:24:31.957 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:24:31.958 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:31.958 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:24:31.958 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:31.958 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:24:31.958 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:31.958 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:24:31.958 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:24:31.958 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:24:31.958 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:24:31.958 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:24:31.958 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:24:31.958 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:24:31.958 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:24:31.958 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:31.959 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:31.959 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:31.959 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:24:31.959 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:24:31.959 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:24:31.959 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:24:31.959 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:24:31.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:31.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:31.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:31.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:24:32.123 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:24:32.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:24:32.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:32.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:32.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:32.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:24:32.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:32.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:32.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:32.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:24:32.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:32.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:32.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:32.128 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:32.128 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:32.128 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:32.187 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 23 2024-09-11T06:24:32.188 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:32.188 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:32.188 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:32.188 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=95, tag_tid=17, entry_tid=17211], [object_number=94, tag_tid=17, entry_tid=17210], [object_number=93, tag_tid=17, entry_tid=17209], [object_number=92, tag_tid=17, entry_tid=17208]]], state=connected] 2024-09-11T06:24:32.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:32.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:32.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:24:32.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:32.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:32.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:32.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:24:32.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:32.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:32.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:32.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:24:32.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:32.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:32.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:32.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:32.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:32.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:32.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 23 2024-09-11T06:24:32.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:32.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:32.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:32.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=95, tag_tid=17, entry_tid=17211], [object_number=94, tag_tid=17, entry_tid=17210], [object_number=93, tag_tid=17, entry_tid=17209], [object_number=92, tag_tid=17, entry_tid=17208]]], state=connected] 2024-09-11T06:24:32.245 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:32.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=95, tag_tid=17, entry_tid=17211' 2024-09-11T06:24:32.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=95, tag_tid=17, entry_tid=17211' 2024-09-11T06:24:32.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=95, tag_tid=17, entry_tid=17211' '!=' 'object_number=93, tag_tid=17, entry_tid=17125' 2024-09-11T06:24:32.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:24:32.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=95, tag_tid=17, entry_tid=17211' = 'object_number=93, tag_tid=17, entry_tid=17125' 2024-09-11T06:24:32.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=95, tag_tid=17, entry_tid=17211' 2024-09-11T06:24:32.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:32.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:32.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:32.248 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:24:32.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=95, tag_tid=17, entry_tid=17211' 2024-09-11T06:24:32.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:32.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:32.250 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:24:32.250 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:32.250 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:32.250 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:32.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:24:32.252 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=95, tag_tid=17, entry_tid=17211' 2024-09-11T06:24:32.252 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:32.252 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:32.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=17211 2024-09-11T06:24:32.254 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 17211 2024-09-11T06:24:32.254 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:24:32.254 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:24:32.254 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:24:32.455 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:24:32.456 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:32.456 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:24:32.456 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:24:32.456 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:24:32.456 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:24:32.456 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:24:32.456 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:24:32.456 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:24:32.456 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:32.456 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:32.456 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:24:32.456 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:24:32.456 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:24:32.457 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:24:32.457 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:24:32.457 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:24:32.460 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:24:32.460 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:24:32.461 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:24:32.461 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:32.461 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:24:32.461 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:24:32.461 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:24:32.461 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:24:32.461 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:24:32.461 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:24:32.461 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:24:32.461 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:32.461 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:24:32.461 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:32.461 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:24:32.462 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:32.462 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:24:32.462 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:24:32.462 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:24:32.462 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:24:32.462 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:24:32.462 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:24:32.462 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:24:32.462 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:24:32.462 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:32.462 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:32.462 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:32.462 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:24:32.462 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:24:32.463 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:24:32.463 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:24:32.463 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:24:32.485 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:32.486 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:32.486 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:32.486 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:24:32.611 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:24:32.612 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:24:32.612 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:32.613 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:32.613 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:32.613 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:24:32.613 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:32.613 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:32.613 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:32.613 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:24:32.613 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:32.613 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:32.613 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:32.615 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:32.616 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:32.616 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:32.668 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 23 2024-09-11T06:24:32.668 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:32.668 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:32.668 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:32.668 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=92, tag_tid=17, entry_tid=17308], [object_number=95, tag_tid=17, entry_tid=17307], [object_number=94, tag_tid=17, entry_tid=17306], [object_number=93, tag_tid=17, entry_tid=17305]]], state=connected] 2024-09-11T06:24:32.671 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:32.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:32.674 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:24:32.674 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:32.674 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:32.674 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:32.674 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:24:32.675 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:32.675 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:32.675 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:32.675 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:24:32.675 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:32.675 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:32.675 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:32.675 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:32.676 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:32.676 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:32.722 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 23 2024-09-11T06:24:32.723 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:32.723 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:32.723 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:32.723 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=92, tag_tid=17, entry_tid=17308], [object_number=95, tag_tid=17, entry_tid=17307], [object_number=94, tag_tid=17, entry_tid=17306], [object_number=93, tag_tid=17, entry_tid=17305]]], state=connected] 2024-09-11T06:24:32.726 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:32.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=92, tag_tid=17, entry_tid=17308' 2024-09-11T06:24:32.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=92, tag_tid=17, entry_tid=17308' 2024-09-11T06:24:32.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=92, tag_tid=17, entry_tid=17308' '!=' 'object_number=95, tag_tid=17, entry_tid=17211' 2024-09-11T06:24:32.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:24:32.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=92, tag_tid=17, entry_tid=17308' = 'object_number=95, tag_tid=17, entry_tid=17211' 2024-09-11T06:24:32.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=92, tag_tid=17, entry_tid=17308' 2024-09-11T06:24:32.729 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:32.729 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:32.729 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:32.729 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:24:32.730 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=92, tag_tid=17, entry_tid=17308' 2024-09-11T06:24:32.730 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:32.730 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:32.731 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:24:32.731 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:32.732 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:32.732 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:32.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:24:32.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=92, tag_tid=17, entry_tid=17308' 2024-09-11T06:24:32.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:32.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:32.735 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=17308 2024-09-11T06:24:32.735 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 17308 2024-09-11T06:24:32.735 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:24:32.735 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:24:32.735 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:24:32.936 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:24:32.936 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:32.937 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:24:32.937 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:24:32.937 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:24:32.937 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:24:32.937 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:24:32.937 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:24:32.937 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:24:32.937 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:32.937 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:32.937 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:24:32.937 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:24:32.937 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:24:32.937 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:24:32.938 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:24:32.938 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:24:32.941 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:24:32.941 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:24:32.941 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:24:32.941 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:32.942 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:24:32.942 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:24:32.942 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:24:32.942 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:24:32.942 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:24:32.942 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:24:32.942 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:24:32.942 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:32.942 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:24:32.942 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:32.942 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:24:32.942 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:32.942 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:24:32.942 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:24:32.943 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:24:32.943 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:24:32.943 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:24:32.943 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:24:32.943 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:24:32.943 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:24:32.943 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:32.943 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:32.943 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:32.943 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:24:32.943 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:24:32.943 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:24:32.943 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:24:32.944 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:24:32.970 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:32.970 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:32.970 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:32.970 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:24:33.103 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:24:33.104 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:24:33.104 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:33.105 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:33.105 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:33.105 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:24:33.105 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:33.105 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:33.105 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:33.105 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:24:33.105 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:33.106 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:33.106 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:33.107 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:33.108 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:33.108 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:33.162 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 23 2024-09-11T06:24:33.162 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:33.162 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:33.162 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:33.162 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=93, tag_tid=17, entry_tid=17401], [object_number=92, tag_tid=17, entry_tid=17400], [object_number=95, tag_tid=17, entry_tid=17399], [object_number=94, tag_tid=17, entry_tid=17398]]], state=connected] 2024-09-11T06:24:33.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:33.167 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:33.168 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:24:33.168 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:33.168 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:33.168 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:33.168 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:24:33.168 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:33.168 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:33.168 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:33.168 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:24:33.169 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:33.169 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:33.169 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:33.169 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:33.169 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:33.169 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:33.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 23 2024-09-11T06:24:33.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:33.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:33.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:33.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=93, tag_tid=17, entry_tid=17401], [object_number=92, tag_tid=17, entry_tid=17400], [object_number=95, tag_tid=17, entry_tid=17399], [object_number=94, tag_tid=17, entry_tid=17398]]], state=connected] 2024-09-11T06:24:33.221 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:33.223 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=93, tag_tid=17, entry_tid=17401' 2024-09-11T06:24:33.223 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=93, tag_tid=17, entry_tid=17401' 2024-09-11T06:24:33.223 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=93, tag_tid=17, entry_tid=17401' '!=' 'object_number=92, tag_tid=17, entry_tid=17308' 2024-09-11T06:24:33.223 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:24:33.223 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=93, tag_tid=17, entry_tid=17401' = 'object_number=92, tag_tid=17, entry_tid=17308' 2024-09-11T06:24:33.223 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=93, tag_tid=17, entry_tid=17401' 2024-09-11T06:24:33.224 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:33.224 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:33.224 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:33.225 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:24:33.225 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=17, entry_tid=17401' 2024-09-11T06:24:33.225 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:33.225 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:33.226 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:24:33.227 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:33.227 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:33.227 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:33.228 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:24:33.228 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=17, entry_tid=17401' 2024-09-11T06:24:33.229 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:33.229 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:33.230 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=17401 2024-09-11T06:24:33.230 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 17401 2024-09-11T06:24:33.230 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:24:33.230 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:24:33.230 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:24:33.432 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:24:33.432 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:33.432 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:24:33.432 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:24:33.432 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:24:33.433 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:24:33.433 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:24:33.433 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:24:33.433 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:24:33.433 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:33.433 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:33.433 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:24:33.433 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:24:33.433 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:24:33.433 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:24:33.435 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:24:33.435 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:24:33.435 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:24:33.435 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:24:33.435 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:24:33.436 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:33.436 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:24:33.436 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:24:33.436 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:24:33.436 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:24:33.436 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:24:33.436 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:24:33.436 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:24:33.436 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:33.436 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:24:33.436 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:33.436 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:24:33.436 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:33.437 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:24:33.437 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:24:33.437 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:24:33.437 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:24:33.437 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:24:33.437 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:24:33.437 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:24:33.437 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:24:33.437 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:33.437 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:33.437 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:33.437 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:24:33.438 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:24:33.438 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:24:33.438 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:24:33.438 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:24:33.463 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:33.463 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:33.463 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:33.463 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:24:33.593 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:24:33.594 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:24:33.594 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:33.594 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:33.594 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:33.594 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:24:33.594 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:33.594 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:33.594 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:33.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:24:33.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:33.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:33.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:33.596 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:33.597 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:33.597 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:33.645 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 23 2024-09-11T06:24:33.645 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:33.645 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:33.645 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:33.645 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=95, tag_tid=17, entry_tid=17495], [object_number=94, tag_tid=17, entry_tid=17494], [object_number=93, tag_tid=17, entry_tid=17493], [object_number=92, tag_tid=17, entry_tid=17492]]], state=connected] 2024-09-11T06:24:33.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:33.649 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:33.651 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:24:33.651 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:33.651 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:33.651 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:33.651 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:24:33.651 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:33.652 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:33.652 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:33.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:24:33.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:33.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:33.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:33.654 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:33.654 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:33.654 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:33.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 23 2024-09-11T06:24:33.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:33.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:33.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:33.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=95, tag_tid=17, entry_tid=17495], [object_number=94, tag_tid=17, entry_tid=17494], [object_number=93, tag_tid=17, entry_tid=17493], [object_number=92, tag_tid=17, entry_tid=17492]]], state=connected] 2024-09-11T06:24:33.701 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:33.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=95, tag_tid=17, entry_tid=17495' 2024-09-11T06:24:33.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=95, tag_tid=17, entry_tid=17495' 2024-09-11T06:24:33.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=95, tag_tid=17, entry_tid=17495' '!=' 'object_number=93, tag_tid=17, entry_tid=17401' 2024-09-11T06:24:33.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:24:33.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=95, tag_tid=17, entry_tid=17495' = 'object_number=93, tag_tid=17, entry_tid=17401' 2024-09-11T06:24:33.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=95, tag_tid=17, entry_tid=17495' 2024-09-11T06:24:33.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:33.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:33.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:33.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:24:33.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=95, tag_tid=17, entry_tid=17495' 2024-09-11T06:24:33.706 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:33.706 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:33.706 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:24:33.707 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:33.707 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:33.707 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:33.708 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:24:33.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=95, tag_tid=17, entry_tid=17495' 2024-09-11T06:24:33.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:33.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:33.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=17495 2024-09-11T06:24:33.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 17495 2024-09-11T06:24:33.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:24:33.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:24:33.711 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:24:33.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:24:33.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:33.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:24:33.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:24:33.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:24:33.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:24:33.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:24:33.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:24:33.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:24:33.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:33.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:33.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:24:33.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:24:33.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:24:33.915 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:24:33.915 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:24:33.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:24:33.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:24:33.921 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:24:33.921 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:24:33.921 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:33.921 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:24:33.921 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:24:33.921 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:24:33.921 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:24:33.921 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:24:33.921 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:24:33.921 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:24:33.922 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:33.922 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:24:33.922 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:33.922 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:24:33.922 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:33.922 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:24:33.922 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:24:33.922 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:24:33.922 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:24:33.922 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:24:33.922 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:24:33.922 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:24:33.922 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:24:33.923 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:33.923 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:33.923 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:33.923 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:24:33.923 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:24:33.923 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:24:33.923 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:24:33.923 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:24:33.942 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:33.942 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:33.942 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:33.943 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:24:34.069 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:24:34.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:24:34.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:34.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:34.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:34.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:24:34.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:34.071 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:34.071 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:34.071 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:24:34.071 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:34.072 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:34.072 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:34.073 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:34.073 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:34.073 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:34.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 23 2024-09-11T06:24:34.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:34.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:34.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:34.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=92, tag_tid=17, entry_tid=17584], [object_number=95, tag_tid=17, entry_tid=17583], [object_number=94, tag_tid=17, entry_tid=17582], [object_number=93, tag_tid=17, entry_tid=17581]]], state=connected] 2024-09-11T06:24:34.129 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:34.130 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:34.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:24:34.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:34.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:34.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:34.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:24:34.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:34.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:34.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:34.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:24:34.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:34.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:34.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:34.133 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:34.133 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:34.133 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:34.179 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 23 2024-09-11T06:24:34.179 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:34.179 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:34.179 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:34.180 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=92, tag_tid=17, entry_tid=17584], [object_number=95, tag_tid=17, entry_tid=17583], [object_number=94, tag_tid=17, entry_tid=17582], [object_number=93, tag_tid=17, entry_tid=17581]]], state=connected] 2024-09-11T06:24:34.182 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:34.184 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=92, tag_tid=17, entry_tid=17584' 2024-09-11T06:24:34.184 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=92, tag_tid=17, entry_tid=17584' 2024-09-11T06:24:34.184 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=92, tag_tid=17, entry_tid=17584' '!=' 'object_number=95, tag_tid=17, entry_tid=17495' 2024-09-11T06:24:34.185 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:24:34.185 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=92, tag_tid=17, entry_tid=17584' = 'object_number=95, tag_tid=17, entry_tid=17495' 2024-09-11T06:24:34.185 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=92, tag_tid=17, entry_tid=17584' 2024-09-11T06:24:34.185 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:34.185 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:34.186 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:34.186 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:24:34.187 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=92, tag_tid=17, entry_tid=17584' 2024-09-11T06:24:34.187 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:34.187 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:34.188 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:24:34.188 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:34.188 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:34.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:34.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:24:34.190 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=92, tag_tid=17, entry_tid=17584' 2024-09-11T06:24:34.190 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:34.190 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:34.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=17584 2024-09-11T06:24:34.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 17584 2024-09-11T06:24:34.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:24:34.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:24:34.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:24:34.393 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:24:34.394 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:34.394 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:24:34.394 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:24:34.394 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:24:34.394 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:24:34.394 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:24:34.394 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:24:34.394 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:24:34.394 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:34.394 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:34.394 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:24:34.395 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:24:34.395 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:24:34.395 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:24:34.395 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:24:34.395 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:24:34.398 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:24:34.399 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:24:34.399 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:24:34.399 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:34.399 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:24:34.399 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:24:34.399 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:24:34.399 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:24:34.399 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:24:34.399 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:24:34.399 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:24:34.399 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:34.399 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:24:34.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:34.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:24:34.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:34.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:24:34.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:24:34.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:24:34.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:24:34.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:24:34.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:24:34.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:24:34.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:24:34.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:34.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:34.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:34.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:24:34.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:24:34.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:24:34.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:24:34.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:24:34.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:34.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:34.427 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:34.427 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:24:34.559 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:24:34.560 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:24:34.560 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:34.561 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:34.561 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:34.561 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:24:34.561 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:34.561 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:34.561 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:34.561 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:24:34.561 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:34.562 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:34.562 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:34.563 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:34.563 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:34.563 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:34.615 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 23 2024-09-11T06:24:34.616 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:34.616 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:34.616 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:34.616 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=92, tag_tid=17, entry_tid=17668], [object_number=95, tag_tid=17, entry_tid=17667], [object_number=94, tag_tid=17, entry_tid=17666], [object_number=93, tag_tid=17, entry_tid=17665]]], state=connected] 2024-09-11T06:24:34.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:34.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:34.621 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:24:34.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:34.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:34.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:34.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:24:34.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:34.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:34.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:34.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:24:34.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:34.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:34.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:34.623 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:34.623 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:34.623 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:34.669 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 23 2024-09-11T06:24:34.669 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:34.669 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:34.670 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:34.670 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=92, tag_tid=17, entry_tid=17668], [object_number=95, tag_tid=17, entry_tid=17667], [object_number=94, tag_tid=17, entry_tid=17666], [object_number=93, tag_tid=17, entry_tid=17665]]], state=connected] 2024-09-11T06:24:34.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:34.674 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=92, tag_tid=17, entry_tid=17668' 2024-09-11T06:24:34.675 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=92, tag_tid=17, entry_tid=17668' 2024-09-11T06:24:34.675 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=92, tag_tid=17, entry_tid=17668' '!=' 'object_number=92, tag_tid=17, entry_tid=17584' 2024-09-11T06:24:34.675 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:24:34.675 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=92, tag_tid=17, entry_tid=17668' = 'object_number=92, tag_tid=17, entry_tid=17584' 2024-09-11T06:24:34.675 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=92, tag_tid=17, entry_tid=17668' 2024-09-11T06:24:34.676 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:34.676 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:34.676 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:34.676 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:24:34.677 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=92, tag_tid=17, entry_tid=17668' 2024-09-11T06:24:34.677 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:34.677 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:34.678 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:24:34.679 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:34.679 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:34.679 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:34.679 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:24:34.680 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=92, tag_tid=17, entry_tid=17668' 2024-09-11T06:24:34.680 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:34.680 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:34.682 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=17668 2024-09-11T06:24:34.682 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 17668 2024-09-11T06:24:34.682 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:24:34.682 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:24:34.682 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:24:34.883 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:24:34.884 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:34.884 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:24:34.884 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:24:34.884 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:24:34.884 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:24:34.884 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:24:34.884 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:24:34.884 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:24:34.884 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:34.884 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:34.884 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:24:34.885 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:24:34.885 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:24:34.885 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:24:34.885 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:24:34.885 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:24:34.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:24:34.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:24:34.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:24:34.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:34.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:24:34.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:24:34.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:24:34.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:24:34.889 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:24:34.889 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:24:34.889 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:24:34.889 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:34.889 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:24:34.889 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:34.889 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:24:34.889 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:34.890 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:24:34.890 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:24:34.890 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:24:34.890 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:24:34.890 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:24:34.890 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:24:34.890 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:24:34.890 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:24:34.890 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:34.890 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:34.891 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:34.891 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:24:34.891 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:24:34.891 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:24:34.891 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:24:34.891 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:24:34.915 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:34.916 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:34.916 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:34.916 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:24:35.045 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:24:35.046 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:24:35.046 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:35.047 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:35.047 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:35.047 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:24:35.047 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:35.047 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:35.047 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:35.047 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:24:35.047 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:35.047 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:35.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:35.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:35.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:35.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:35.102 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 23 2024-09-11T06:24:35.102 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:35.102 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:35.103 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:35.103 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=95, tag_tid=17, entry_tid=17763], [object_number=94, tag_tid=17, entry_tid=17762], [object_number=93, tag_tid=17, entry_tid=17761], [object_number=92, tag_tid=17, entry_tid=17760]]], state=connected] 2024-09-11T06:24:35.105 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:35.108 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:35.108 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:24:35.108 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:35.108 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:35.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:35.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:24:35.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:35.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:35.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:35.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:24:35.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:35.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:35.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:35.110 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:35.110 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:35.110 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:35.157 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 23 2024-09-11T06:24:35.157 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:35.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:35.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:35.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=95, tag_tid=17, entry_tid=17763], [object_number=94, tag_tid=17, entry_tid=17762], [object_number=93, tag_tid=17, entry_tid=17761], [object_number=92, tag_tid=17, entry_tid=17760]]], state=connected] 2024-09-11T06:24:35.161 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:35.163 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=95, tag_tid=17, entry_tid=17763' 2024-09-11T06:24:35.163 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=95, tag_tid=17, entry_tid=17763' 2024-09-11T06:24:35.163 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=95, tag_tid=17, entry_tid=17763' '!=' 'object_number=92, tag_tid=17, entry_tid=17668' 2024-09-11T06:24:35.163 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:24:35.164 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=95, tag_tid=17, entry_tid=17763' = 'object_number=92, tag_tid=17, entry_tid=17668' 2024-09-11T06:24:35.164 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=95, tag_tid=17, entry_tid=17763' 2024-09-11T06:24:35.164 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:35.164 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:35.164 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:35.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:24:35.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=95, tag_tid=17, entry_tid=17763' 2024-09-11T06:24:35.166 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:35.166 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:35.166 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:24:35.167 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:35.167 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:35.167 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:35.168 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:24:35.169 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=95, tag_tid=17, entry_tid=17763' 2024-09-11T06:24:35.169 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:35.169 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:35.170 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=17763 2024-09-11T06:24:35.170 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 17763 2024-09-11T06:24:35.170 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:24:35.171 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:24:35.171 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:24:35.372 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:24:35.372 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:35.373 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:24:35.373 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:24:35.373 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:24:35.373 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:24:35.373 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:24:35.373 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:24:35.373 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:24:35.373 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:35.373 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:35.373 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:24:35.373 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:24:35.373 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:24:35.373 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:24:35.374 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:24:35.374 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:24:35.376 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:24:35.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:24:35.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:24:35.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:35.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:24:35.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:24:35.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:24:35.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:24:35.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:24:35.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:24:35.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:24:35.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:35.378 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:24:35.378 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:35.378 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:24:35.378 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:35.379 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:24:35.379 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:24:35.379 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:24:35.379 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:24:35.379 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:24:35.380 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:24:35.380 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:24:35.380 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:24:35.380 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:35.380 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:35.380 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:35.380 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:24:35.380 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:24:35.380 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:24:35.380 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:24:35.380 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:24:35.403 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:35.404 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:35.404 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:35.404 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:24:35.536 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:24:35.537 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:24:35.537 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:35.537 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:35.537 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:35.538 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:24:35.538 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:35.538 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:35.538 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:35.538 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:24:35.539 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:35.539 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:35.539 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:35.540 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:35.541 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:35.541 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:35.589 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 23 2024-09-11T06:24:35.589 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:35.589 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:35.590 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:35.590 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=93, tag_tid=17, entry_tid=17865], [object_number=92, tag_tid=17, entry_tid=17864], [object_number=95, tag_tid=17, entry_tid=17863], [object_number=94, tag_tid=17, entry_tid=17862]]], state=connected] 2024-09-11T06:24:35.593 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:35.594 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:35.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:24:35.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:35.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:35.596 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:35.596 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:24:35.596 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:35.596 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:35.596 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:35.596 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:24:35.596 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:35.596 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:35.596 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:35.597 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:35.597 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:35.597 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:35.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 23 2024-09-11T06:24:35.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:35.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:35.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:35.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=93, tag_tid=17, entry_tid=17865], [object_number=92, tag_tid=17, entry_tid=17864], [object_number=95, tag_tid=17, entry_tid=17863], [object_number=94, tag_tid=17, entry_tid=17862]]], state=connected] 2024-09-11T06:24:35.646 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:35.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=93, tag_tid=17, entry_tid=17865' 2024-09-11T06:24:35.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=93, tag_tid=17, entry_tid=17865' 2024-09-11T06:24:35.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=93, tag_tid=17, entry_tid=17865' '!=' 'object_number=95, tag_tid=17, entry_tid=17763' 2024-09-11T06:24:35.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:24:35.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=93, tag_tid=17, entry_tid=17865' = 'object_number=95, tag_tid=17, entry_tid=17763' 2024-09-11T06:24:35.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=93, tag_tid=17, entry_tid=17865' 2024-09-11T06:24:35.649 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:35.649 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:35.649 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:35.649 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:24:35.650 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=17, entry_tid=17865' 2024-09-11T06:24:35.650 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:35.650 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:35.651 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:24:35.652 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:35.652 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:35.652 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:35.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:24:35.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=17, entry_tid=17865' 2024-09-11T06:24:35.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:35.654 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:35.655 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=17865 2024-09-11T06:24:35.655 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 17865 2024-09-11T06:24:35.655 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:24:35.655 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:24:35.655 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:24:35.857 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:24:35.857 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:35.857 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:24:35.857 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:24:35.857 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:24:35.857 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:24:35.857 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:24:35.857 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:24:35.857 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:24:35.857 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:35.858 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:35.858 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:24:35.858 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:24:35.858 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:24:35.858 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:24:35.858 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:24:35.859 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:24:35.861 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:24:35.861 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:24:35.861 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:24:35.862 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:35.862 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:24:35.862 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:24:35.862 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:24:35.862 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:24:35.862 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:24:35.862 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:24:35.862 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:24:35.862 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:35.862 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:24:35.862 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:35.862 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:24:35.863 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:35.863 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:24:35.863 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:24:35.863 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:24:35.863 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:24:35.863 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:24:35.863 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:24:35.863 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:24:35.863 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:24:35.863 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:35.863 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:35.863 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:35.863 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:24:35.863 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:24:35.864 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:24:35.864 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:24:35.864 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:24:35.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:35.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:35.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:35.889 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:24:36.019 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:24:36.021 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:24:36.021 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:36.021 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:36.021 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:36.021 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:24:36.021 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:36.021 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:36.021 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:36.021 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:24:36.022 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:36.022 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:36.022 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:36.023 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:36.024 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:36.024 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:36.074 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 23 2024-09-11T06:24:36.074 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:36.074 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:36.074 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:36.074 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=93, tag_tid=17, entry_tid=17957], [object_number=92, tag_tid=17, entry_tid=17956], [object_number=95, tag_tid=17, entry_tid=17955], [object_number=94, tag_tid=17, entry_tid=17954]]], state=connected] 2024-09-11T06:24:36.077 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:36.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:36.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:24:36.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:36.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:36.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:36.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:24:36.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:36.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:36.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:36.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:24:36.081 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:36.081 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:36.081 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:36.081 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:36.081 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:36.082 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:36.128 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 23 2024-09-11T06:24:36.128 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:36.128 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:36.128 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:36.128 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=93, tag_tid=17, entry_tid=17957], [object_number=92, tag_tid=17, entry_tid=17956], [object_number=95, tag_tid=17, entry_tid=17955], [object_number=94, tag_tid=17, entry_tid=17954]]], state=connected] 2024-09-11T06:24:36.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:36.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=93, tag_tid=17, entry_tid=17957' 2024-09-11T06:24:36.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=93, tag_tid=17, entry_tid=17957' 2024-09-11T06:24:36.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=93, tag_tid=17, entry_tid=17957' '!=' 'object_number=93, tag_tid=17, entry_tid=17865' 2024-09-11T06:24:36.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:24:36.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=93, tag_tid=17, entry_tid=17957' = 'object_number=93, tag_tid=17, entry_tid=17865' 2024-09-11T06:24:36.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=93, tag_tid=17, entry_tid=17957' 2024-09-11T06:24:36.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:36.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:36.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:36.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:24:36.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=17, entry_tid=17957' 2024-09-11T06:24:36.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:36.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:36.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:24:36.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:36.139 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:36.139 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:36.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:24:36.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=17, entry_tid=17957' 2024-09-11T06:24:36.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:36.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:36.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=17957 2024-09-11T06:24:36.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 17957 2024-09-11T06:24:36.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:24:36.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:24:36.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:24:36.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:24:36.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:36.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:24:36.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:24:36.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:24:36.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:24:36.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:24:36.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:24:36.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:24:36.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:36.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:36.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:24:36.343 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:24:36.343 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:24:36.343 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:24:36.343 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:24:36.343 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:24:36.345 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:24:36.345 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:24:36.345 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:24:36.345 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:36.345 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:24:36.346 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:24:36.346 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:24:36.346 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:24:36.346 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:24:36.346 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:24:36.346 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:24:36.346 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:36.346 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:24:36.346 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:36.346 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:24:36.346 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:36.347 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:24:36.347 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:24:36.347 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:24:36.347 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:24:36.347 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:24:36.348 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:24:36.348 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:24:36.348 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:24:36.348 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:36.348 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:36.348 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:36.348 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:24:36.348 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:24:36.348 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:24:36.348 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:24:36.348 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:24:36.372 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:36.373 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:36.373 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:36.373 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:24:36.508 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:24:36.509 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:24:36.509 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:36.509 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:36.509 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:36.509 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:24:36.509 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:36.510 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:36.510 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:36.510 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:24:36.510 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:36.510 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:36.510 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:36.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:36.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:36.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:36.559 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 23 2024-09-11T06:24:36.559 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:36.559 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:36.559 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:36.559 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=92, tag_tid=17, entry_tid=18064], [object_number=95, tag_tid=17, entry_tid=18063], [object_number=94, tag_tid=17, entry_tid=18062], [object_number=93, tag_tid=17, entry_tid=18061]]], state=connected] 2024-09-11T06:24:36.562 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:36.565 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:36.565 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:24:36.565 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:36.565 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:36.565 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:36.565 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:24:36.565 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:36.565 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:36.565 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:36.565 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:24:36.565 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:36.566 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:36.566 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:36.566 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:36.566 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:36.566 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:36.613 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 23 2024-09-11T06:24:36.613 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:36.613 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:36.613 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:36.613 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=92, tag_tid=17, entry_tid=18064], [object_number=95, tag_tid=17, entry_tid=18063], [object_number=94, tag_tid=17, entry_tid=18062], [object_number=93, tag_tid=17, entry_tid=18061]]], state=connected] 2024-09-11T06:24:36.616 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:36.618 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=92, tag_tid=17, entry_tid=18064' 2024-09-11T06:24:36.618 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=92, tag_tid=17, entry_tid=18064' 2024-09-11T06:24:36.618 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=92, tag_tid=17, entry_tid=18064' '!=' 'object_number=93, tag_tid=17, entry_tid=17957' 2024-09-11T06:24:36.618 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:24:36.618 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=92, tag_tid=17, entry_tid=18064' = 'object_number=93, tag_tid=17, entry_tid=17957' 2024-09-11T06:24:36.618 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=92, tag_tid=17, entry_tid=18064' 2024-09-11T06:24:36.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:36.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:36.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:36.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:24:36.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=92, tag_tid=17, entry_tid=18064' 2024-09-11T06:24:36.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:36.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:36.621 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:24:36.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:36.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:36.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:36.623 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:24:36.624 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=92, tag_tid=17, entry_tid=18064' 2024-09-11T06:24:36.624 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:36.624 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:36.625 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=18064 2024-09-11T06:24:36.625 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 18064 2024-09-11T06:24:36.625 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:24:36.625 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:24:36.625 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:24:36.827 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:24:36.827 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:36.827 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:24:36.827 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:24:36.827 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:24:36.827 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:24:36.827 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:24:36.827 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:24:36.827 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:24:36.827 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:36.828 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:36.828 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:24:36.828 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:24:36.828 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:24:36.828 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:24:36.828 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:24:36.828 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:24:36.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:24:36.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:24:36.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:24:36.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:36.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:24:36.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:24:36.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:24:36.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:24:36.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:24:36.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:24:36.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:24:36.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:36.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:24:36.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:36.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:24:36.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:36.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:24:36.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:24:36.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:24:36.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:24:36.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:24:36.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:24:36.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:24:36.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:24:36.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:36.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:36.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:36.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:24:36.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:24:36.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:24:36.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:24:36.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:24:36.857 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:36.857 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:36.857 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:36.857 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:24:36.990 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:24:36.991 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:24:36.991 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:36.992 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:36.992 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:36.992 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:24:36.992 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:36.992 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:36.992 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:36.992 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:24:36.992 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:36.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:36.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:36.994 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:36.994 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:36.994 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:37.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 23 2024-09-11T06:24:37.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:37.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:37.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:37.050 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=93, tag_tid=17, entry_tid=18157], [object_number=92, tag_tid=17, entry_tid=18156], [object_number=95, tag_tid=17, entry_tid=18155], [object_number=94, tag_tid=17, entry_tid=18154]]], state=connected] 2024-09-11T06:24:37.052 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:37.054 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:37.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:24:37.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:37.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:37.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:37.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:24:37.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:37.056 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:37.056 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:37.056 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:24:37.056 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:37.056 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:37.056 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:37.057 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:37.057 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:37.057 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:37.104 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 23 2024-09-11T06:24:37.104 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:37.104 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:37.104 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:37.104 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=93, tag_tid=17, entry_tid=18157], [object_number=92, tag_tid=17, entry_tid=18156], [object_number=95, tag_tid=17, entry_tid=18155], [object_number=94, tag_tid=17, entry_tid=18154]]], state=connected] 2024-09-11T06:24:37.108 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:37.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=93, tag_tid=17, entry_tid=18157' 2024-09-11T06:24:37.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=93, tag_tid=17, entry_tid=18157' 2024-09-11T06:24:37.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=93, tag_tid=17, entry_tid=18157' '!=' 'object_number=92, tag_tid=17, entry_tid=18064' 2024-09-11T06:24:37.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:24:37.110 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=93, tag_tid=17, entry_tid=18157' = 'object_number=92, tag_tid=17, entry_tid=18064' 2024-09-11T06:24:37.110 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=93, tag_tid=17, entry_tid=18157' 2024-09-11T06:24:37.110 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:37.110 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:37.110 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:37.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:24:37.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=17, entry_tid=18157' 2024-09-11T06:24:37.112 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:37.112 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:37.112 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:24:37.113 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:37.113 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:37.113 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:37.114 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:24:37.115 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=17, entry_tid=18157' 2024-09-11T06:24:37.115 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:37.115 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:37.116 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=18157 2024-09-11T06:24:37.116 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 18157 2024-09-11T06:24:37.116 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:24:37.117 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:24:37.117 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:24:37.318 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:24:37.318 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:37.318 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:24:37.318 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:24:37.318 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:24:37.319 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:24:37.319 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:24:37.319 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:24:37.319 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:24:37.319 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:37.319 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:37.319 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:24:37.319 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:24:37.319 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:24:37.319 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:24:37.320 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:24:37.320 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:24:37.323 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:24:37.324 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:24:37.324 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:24:37.324 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:37.324 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:24:37.324 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:24:37.324 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:24:37.324 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:24:37.324 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:24:37.324 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:24:37.324 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:24:37.324 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:37.324 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:24:37.324 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:37.325 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:24:37.325 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:37.325 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:24:37.325 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:24:37.325 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:24:37.325 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:24:37.325 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:24:37.325 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:24:37.325 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:24:37.325 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:24:37.325 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:37.325 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:37.325 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:37.325 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:24:37.326 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:24:37.326 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:24:37.326 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:24:37.326 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:24:37.349 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:37.352 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:37.353 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:37.353 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:24:37.478 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:24:37.479 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:24:37.479 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:37.479 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:37.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:37.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:24:37.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:37.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:37.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:37.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:24:37.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:37.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:37.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:37.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:37.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:37.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:37.536 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 23 2024-09-11T06:24:37.536 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:37.536 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:37.536 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:37.536 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=92, tag_tid=17, entry_tid=18244], [object_number=95, tag_tid=17, entry_tid=18243], [object_number=94, tag_tid=17, entry_tid=18242], [object_number=93, tag_tid=17, entry_tid=18241]]], state=connected] 2024-09-11T06:24:37.539 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:37.540 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:37.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:24:37.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:37.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:37.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:37.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:24:37.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:37.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:37.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:37.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:24:37.543 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:37.543 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:37.543 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:37.543 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:37.543 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:37.544 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:37.589 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 23 2024-09-11T06:24:37.590 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:37.590 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:37.590 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:37.590 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=92, tag_tid=17, entry_tid=18244], [object_number=95, tag_tid=17, entry_tid=18243], [object_number=94, tag_tid=17, entry_tid=18242], [object_number=93, tag_tid=17, entry_tid=18241]]], state=connected] 2024-09-11T06:24:37.593 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:37.594 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=92, tag_tid=17, entry_tid=18244' 2024-09-11T06:24:37.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=92, tag_tid=17, entry_tid=18244' 2024-09-11T06:24:37.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=92, tag_tid=17, entry_tid=18244' '!=' 'object_number=93, tag_tid=17, entry_tid=18157' 2024-09-11T06:24:37.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:24:37.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=92, tag_tid=17, entry_tid=18244' = 'object_number=93, tag_tid=17, entry_tid=18157' 2024-09-11T06:24:37.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=92, tag_tid=17, entry_tid=18244' 2024-09-11T06:24:37.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:37.596 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:37.596 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:37.596 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:24:37.597 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=92, tag_tid=17, entry_tid=18244' 2024-09-11T06:24:37.597 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:37.597 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:37.598 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:24:37.598 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:37.598 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:37.599 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:37.599 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:24:37.600 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=92, tag_tid=17, entry_tid=18244' 2024-09-11T06:24:37.600 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:37.600 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:37.602 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=18244 2024-09-11T06:24:37.602 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 18244 2024-09-11T06:24:37.602 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:24:37.602 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:24:37.602 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:24:37.804 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:24:37.804 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:37.804 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:24:37.804 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:24:37.804 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:24:37.804 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:24:37.804 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:24:37.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:24:37.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:24:37.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:37.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:37.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:24:37.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:24:37.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:24:37.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:24:37.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:24:37.806 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:24:37.808 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:24:37.808 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:24:37.808 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:24:37.808 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:37.808 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:24:37.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:24:37.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:24:37.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:24:37.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:24:37.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:24:37.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:24:37.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:37.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:24:37.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:37.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:24:37.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:37.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:24:37.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:24:37.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:24:37.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:24:37.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:24:37.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:24:37.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:24:37.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:24:37.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:37.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:37.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:37.812 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:24:37.812 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:24:37.812 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:24:37.812 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:24:37.812 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:24:37.836 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:37.836 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:37.836 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:37.837 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:24:37.969 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:24:37.971 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:24:37.971 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:37.971 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:37.971 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:37.971 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:24:37.971 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:37.971 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:37.971 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:37.971 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:24:37.972 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:37.972 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:37.972 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:37.973 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:37.973 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:37.974 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:38.020 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 23 2024-09-11T06:24:38.020 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:38.021 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:38.021 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:38.021 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=95, tag_tid=17, entry_tid=18331], [object_number=94, tag_tid=17, entry_tid=18330], [object_number=93, tag_tid=17, entry_tid=18329], [object_number=92, tag_tid=17, entry_tid=18328]]], state=connected] 2024-09-11T06:24:38.024 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:38.025 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:38.026 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:24:38.026 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:38.026 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:38.026 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:38.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:24:38.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:38.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:38.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:38.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:24:38.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:38.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:38.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:38.028 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:38.028 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:38.028 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:38.280 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 23 2024-09-11T06:24:38.280 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:38.281 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:38.281 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:38.281 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=95, tag_tid=17, entry_tid=18331], [object_number=94, tag_tid=17, entry_tid=18330], [object_number=93, tag_tid=17, entry_tid=18329], [object_number=92, tag_tid=17, entry_tid=18328]]], state=connected] 2024-09-11T06:24:38.286 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:38.288 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=95, tag_tid=17, entry_tid=18331' 2024-09-11T06:24:38.288 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=95, tag_tid=17, entry_tid=18331' 2024-09-11T06:24:38.288 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=95, tag_tid=17, entry_tid=18331' '!=' 'object_number=92, tag_tid=17, entry_tid=18244' 2024-09-11T06:24:38.288 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:24:38.289 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=95, tag_tid=17, entry_tid=18331' = 'object_number=92, tag_tid=17, entry_tid=18244' 2024-09-11T06:24:38.289 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=95, tag_tid=17, entry_tid=18331' 2024-09-11T06:24:38.289 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:38.289 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:38.289 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:38.291 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:24:38.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=95, tag_tid=17, entry_tid=18331' 2024-09-11T06:24:38.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:38.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:38.293 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:24:38.294 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:38.294 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:38.295 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:38.296 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:24:38.297 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=95, tag_tid=17, entry_tid=18331' 2024-09-11T06:24:38.297 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:38.297 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:38.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=18331 2024-09-11T06:24:38.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 18331 2024-09-11T06:24:38.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:24:38.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:24:38.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:24:38.501 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:24:38.501 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:38.501 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:24:38.501 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:24:38.501 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:24:38.502 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:24:38.502 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:24:38.502 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:24:38.502 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:24:38.502 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:38.502 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:38.502 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:24:38.502 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:24:38.502 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:24:38.502 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:24:38.503 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:24:38.503 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:24:38.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:24:38.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:24:38.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:24:38.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:38.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:24:38.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:24:38.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:24:38.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:24:38.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:24:38.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:24:38.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:24:38.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:38.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:24:38.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:38.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:24:38.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:38.507 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:24:38.507 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:24:38.507 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:24:38.507 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:24:38.507 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:24:38.508 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:24:38.508 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:24:38.508 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:24:38.508 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:38.508 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:38.508 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:38.508 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:24:38.508 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:24:38.508 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:24:38.508 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:24:38.508 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:24:38.532 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:38.533 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:38.533 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:38.533 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:24:38.662 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:24:38.663 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:24:38.663 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:38.663 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:38.664 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:38.664 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:24:38.664 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:38.664 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:38.664 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:38.664 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:24:38.664 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:38.665 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:38.665 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:38.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:38.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:38.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:38.721 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 23 2024-09-11T06:24:38.721 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:38.721 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:38.721 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:38.721 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=95, tag_tid=17, entry_tid=18475], [object_number=94, tag_tid=17, entry_tid=18474], [object_number=93, tag_tid=17, entry_tid=18473], [object_number=92, tag_tid=17, entry_tid=18472]]], state=connected] 2024-09-11T06:24:38.725 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:38.726 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:38.727 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:24:38.727 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:38.727 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:38.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:38.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:24:38.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:38.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:38.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:38.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:24:38.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:38.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:38.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:38.729 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:38.729 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:38.729 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:38.776 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 23 2024-09-11T06:24:38.776 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:38.776 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:38.776 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:38.776 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=95, tag_tid=17, entry_tid=18475], [object_number=94, tag_tid=17, entry_tid=18474], [object_number=93, tag_tid=17, entry_tid=18473], [object_number=92, tag_tid=17, entry_tid=18472]]], state=connected] 2024-09-11T06:24:38.779 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:38.781 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=95, tag_tid=17, entry_tid=18475' 2024-09-11T06:24:38.781 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=95, tag_tid=17, entry_tid=18475' 2024-09-11T06:24:38.781 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=95, tag_tid=17, entry_tid=18475' '!=' 'object_number=95, tag_tid=17, entry_tid=18331' 2024-09-11T06:24:38.781 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:24:38.781 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=95, tag_tid=17, entry_tid=18475' = 'object_number=95, tag_tid=17, entry_tid=18331' 2024-09-11T06:24:38.781 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=95, tag_tid=17, entry_tid=18475' 2024-09-11T06:24:38.782 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:38.782 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:38.782 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:38.782 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:24:38.783 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=95, tag_tid=17, entry_tid=18475' 2024-09-11T06:24:38.783 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:38.783 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:38.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:24:38.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:38.785 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:38.785 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:38.785 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:24:38.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=95, tag_tid=17, entry_tid=18475' 2024-09-11T06:24:38.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:38.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:38.787 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=18475 2024-09-11T06:24:38.788 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 18475 2024-09-11T06:24:38.788 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:24:38.788 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:24:38.788 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:24:38.992 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:24:38.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:38.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:24:38.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:24:38.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:24:38.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:24:38.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:24:38.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:24:38.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:24:38.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:38.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:38.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:24:38.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:24:38.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:24:38.994 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:24:38.994 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:24:38.994 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:24:38.996 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:24:38.996 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:24:38.996 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:24:38.996 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:38.996 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:24:38.996 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:24:38.996 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:24:38.996 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:24:38.996 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:24:38.996 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:24:38.997 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:24:38.997 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:38.997 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:24:38.997 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:38.997 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:24:38.997 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:38.997 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:24:38.997 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:24:38.997 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:24:38.997 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:24:38.997 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:24:38.997 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:24:38.997 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:24:38.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:24:38.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:38.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:38.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:38.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:24:38.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:24:38.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:24:38.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:24:38.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:24:39.021 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:39.022 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:39.022 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:39.022 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:24:39.154 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:24:39.155 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:24:39.155 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:39.155 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:39.155 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:39.155 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:24:39.156 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:39.156 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:39.156 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:39.156 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:24:39.156 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:39.156 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:39.156 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:39.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:39.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:39.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:39.215 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 23 2024-09-11T06:24:39.215 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:39.215 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:39.215 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:39.215 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=94, tag_tid=17, entry_tid=18578], [object_number=93, tag_tid=17, entry_tid=18577], [object_number=92, tag_tid=17, entry_tid=18576], [object_number=95, tag_tid=17, entry_tid=18575]]], state=connected] 2024-09-11T06:24:39.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:39.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:39.221 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:24:39.221 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:39.221 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:39.221 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:39.221 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:24:39.221 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:39.221 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:39.221 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:39.222 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:24:39.222 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:39.222 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:39.222 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:39.222 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:39.223 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:39.223 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:39.270 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 23 2024-09-11T06:24:39.270 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:39.270 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:39.271 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:39.271 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=94, tag_tid=17, entry_tid=18578], [object_number=93, tag_tid=17, entry_tid=18577], [object_number=92, tag_tid=17, entry_tid=18576], [object_number=95, tag_tid=17, entry_tid=18575]]], state=connected] 2024-09-11T06:24:39.273 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:39.275 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=94, tag_tid=17, entry_tid=18578' 2024-09-11T06:24:39.275 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=94, tag_tid=17, entry_tid=18578' 2024-09-11T06:24:39.275 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=94, tag_tid=17, entry_tid=18578' '!=' 'object_number=95, tag_tid=17, entry_tid=18475' 2024-09-11T06:24:39.275 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:24:39.275 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=94, tag_tid=17, entry_tid=18578' = 'object_number=95, tag_tid=17, entry_tid=18475' 2024-09-11T06:24:39.275 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=94, tag_tid=17, entry_tid=18578' 2024-09-11T06:24:39.276 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:39.276 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:39.277 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:39.277 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:24:39.277 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=94, tag_tid=17, entry_tid=18578' 2024-09-11T06:24:39.278 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:39.278 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:39.278 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:24:39.279 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:39.279 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:39.279 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:39.280 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:24:39.281 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=94, tag_tid=17, entry_tid=18578' 2024-09-11T06:24:39.281 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:39.281 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:39.282 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=18578 2024-09-11T06:24:39.282 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 18578 2024-09-11T06:24:39.282 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:24:39.283 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:24:39.283 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:24:39.484 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:24:39.484 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:39.484 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:24:39.484 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:24:39.484 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:24:39.484 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:24:39.484 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:24:39.484 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:24:39.485 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:24:39.485 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:39.485 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:39.485 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:24:39.485 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:24:39.485 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:24:39.485 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:24:39.486 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:24:39.486 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:24:39.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:24:39.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:24:39.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:24:39.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:39.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:24:39.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:24:39.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:24:39.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:24:39.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:24:39.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:24:39.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:24:39.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:39.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:24:39.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:39.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:24:39.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:39.490 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:24:39.490 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:24:39.490 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:24:39.490 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:24:39.490 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:24:39.490 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:24:39.490 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:24:39.491 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:24:39.491 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:39.491 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:39.491 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:39.491 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:24:39.491 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:24:39.491 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:24:39.491 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:24:39.491 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:24:39.516 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:39.516 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:39.517 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:39.517 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:24:39.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:24:39.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:24:39.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:39.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:39.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:39.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:24:39.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:39.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:39.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:39.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:24:39.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:39.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:39.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:39.645 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:39.645 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:39.645 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:39.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 23 2024-09-11T06:24:39.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:39.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:39.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:39.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=93, tag_tid=17, entry_tid=18677], [object_number=92, tag_tid=17, entry_tid=18676], [object_number=95, tag_tid=17, entry_tid=18675], [object_number=94, tag_tid=17, entry_tid=18674]]], state=connected] 2024-09-11T06:24:39.695 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:39.696 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:39.697 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:24:39.697 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:39.697 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:39.697 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:39.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:24:39.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:39.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:39.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:39.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:24:39.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:39.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:39.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:39.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:39.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:39.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:39.745 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 23 2024-09-11T06:24:39.745 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:39.745 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:39.746 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:39.746 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=93, tag_tid=17, entry_tid=18677], [object_number=92, tag_tid=17, entry_tid=18676], [object_number=95, tag_tid=17, entry_tid=18675], [object_number=94, tag_tid=17, entry_tid=18674]]], state=connected] 2024-09-11T06:24:39.749 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:39.751 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=93, tag_tid=17, entry_tid=18677' 2024-09-11T06:24:39.751 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=93, tag_tid=17, entry_tid=18677' 2024-09-11T06:24:39.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=93, tag_tid=17, entry_tid=18677' '!=' 'object_number=94, tag_tid=17, entry_tid=18578' 2024-09-11T06:24:39.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:24:39.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=93, tag_tid=17, entry_tid=18677' = 'object_number=94, tag_tid=17, entry_tid=18578' 2024-09-11T06:24:39.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=93, tag_tid=17, entry_tid=18677' 2024-09-11T06:24:39.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:39.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:39.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:39.753 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:24:39.753 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=17, entry_tid=18677' 2024-09-11T06:24:39.753 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:39.753 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:39.754 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:24:39.754 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:39.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:39.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:39.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:24:39.756 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=17, entry_tid=18677' 2024-09-11T06:24:39.756 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:39.756 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:39.757 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=18677 2024-09-11T06:24:39.757 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 18677 2024-09-11T06:24:39.758 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:24:39.758 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:24:39.758 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:24:39.959 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:24:39.960 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:39.960 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:24:39.960 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:24:39.960 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:24:39.960 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:24:39.960 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:24:39.960 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:24:39.960 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:24:39.960 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:39.960 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:39.961 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:24:39.961 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:24:39.961 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:24:39.961 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:24:39.961 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:24:39.962 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:24:39.964 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:24:39.964 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:24:39.964 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:24:39.964 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:39.964 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:24:39.964 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:24:39.964 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:24:39.964 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:24:39.964 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:24:39.965 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:24:39.965 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:24:39.965 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:39.965 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:24:39.965 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:39.965 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:24:39.965 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:39.966 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:24:39.966 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:24:39.966 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:24:39.966 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:24:39.966 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:24:39.966 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:24:39.967 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:24:39.967 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:24:39.967 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:39.967 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:39.967 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:39.967 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:24:39.967 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:24:39.967 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:24:39.967 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:24:39.967 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:24:39.992 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:39.992 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:39.992 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:39.992 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:24:40.123 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:24:40.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:24:40.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:40.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:40.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:40.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:24:40.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:40.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:40.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:40.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:24:40.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:40.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:40.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:40.128 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:40.128 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:40.128 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:40.181 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 23 2024-09-11T06:24:40.181 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:40.181 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:40.181 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:40.181 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=95, tag_tid=17, entry_tid=18771], [object_number=94, tag_tid=17, entry_tid=18770], [object_number=93, tag_tid=17, entry_tid=18769], [object_number=92, tag_tid=17, entry_tid=18768]]], state=connected] 2024-09-11T06:24:40.185 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:40.186 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:40.187 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:24:40.187 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:40.187 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:40.187 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:40.188 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:24:40.188 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:40.188 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:40.188 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:40.188 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:24:40.188 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:40.188 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:40.188 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:40.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:40.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:40.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:40.236 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 23 2024-09-11T06:24:40.236 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:40.237 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:40.237 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:40.237 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=95, tag_tid=17, entry_tid=18771], [object_number=94, tag_tid=17, entry_tid=18770], [object_number=93, tag_tid=17, entry_tid=18769], [object_number=92, tag_tid=17, entry_tid=18768]]], state=connected] 2024-09-11T06:24:40.240 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:40.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=95, tag_tid=17, entry_tid=18771' 2024-09-11T06:24:40.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=95, tag_tid=17, entry_tid=18771' 2024-09-11T06:24:40.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=95, tag_tid=17, entry_tid=18771' '!=' 'object_number=93, tag_tid=17, entry_tid=18677' 2024-09-11T06:24:40.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:24:40.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=95, tag_tid=17, entry_tid=18771' = 'object_number=93, tag_tid=17, entry_tid=18677' 2024-09-11T06:24:40.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=95, tag_tid=17, entry_tid=18771' 2024-09-11T06:24:40.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:40.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:40.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:40.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:24:40.244 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=95, tag_tid=17, entry_tid=18771' 2024-09-11T06:24:40.244 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:40.244 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:40.245 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:24:40.246 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:40.246 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:40.246 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:40.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:24:40.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=95, tag_tid=17, entry_tid=18771' 2024-09-11T06:24:40.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:40.248 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:40.248 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=18771 2024-09-11T06:24:40.248 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 18771 2024-09-11T06:24:40.248 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:24:40.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:24:40.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:24:40.450 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:24:40.451 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:40.451 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:24:40.451 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:24:40.451 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:24:40.451 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:24:40.451 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:24:40.451 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:24:40.451 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:24:40.451 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:40.451 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:40.451 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:24:40.451 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:24:40.452 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:24:40.452 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:24:40.452 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:24:40.452 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:24:40.455 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:24:40.455 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:24:40.455 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:24:40.456 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:40.456 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:24:40.456 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:24:40.456 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:24:40.456 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:24:40.456 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:24:40.456 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:24:40.456 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:24:40.456 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:40.456 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:24:40.456 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:40.456 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:24:40.456 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:40.456 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:24:40.457 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:24:40.457 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:24:40.457 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:24:40.457 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:24:40.457 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:24:40.457 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:24:40.457 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:24:40.457 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:40.457 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:40.457 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:40.457 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:24:40.457 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:24:40.457 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:24:40.458 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:24:40.458 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:24:40.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:40.483 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:40.483 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:40.483 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:24:40.609 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:24:40.611 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:24:40.611 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:40.611 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:40.611 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:40.611 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:24:40.611 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:40.611 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:40.611 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:40.611 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:24:40.611 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:40.612 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:40.612 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:40.613 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:40.613 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:40.613 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:40.660 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 23 2024-09-11T06:24:40.660 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:40.660 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:40.661 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:40.661 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=95, tag_tid=17, entry_tid=18863], [object_number=94, tag_tid=17, entry_tid=18862], [object_number=93, tag_tid=17, entry_tid=18861], [object_number=92, tag_tid=17, entry_tid=18860]]], state=connected] 2024-09-11T06:24:40.664 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:40.665 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:40.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:24:40.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:40.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:40.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:40.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:24:40.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:40.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:40.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:40.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:24:40.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:40.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:40.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:40.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:40.668 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:40.668 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:40.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 23 2024-09-11T06:24:40.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:40.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:40.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:40.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=95, tag_tid=17, entry_tid=18863], [object_number=94, tag_tid=17, entry_tid=18862], [object_number=93, tag_tid=17, entry_tid=18861], [object_number=92, tag_tid=17, entry_tid=18860]]], state=connected] 2024-09-11T06:24:40.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:40.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=95, tag_tid=17, entry_tid=18863' 2024-09-11T06:24:40.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=95, tag_tid=17, entry_tid=18863' 2024-09-11T06:24:40.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=95, tag_tid=17, entry_tid=18863' '!=' 'object_number=95, tag_tid=17, entry_tid=18771' 2024-09-11T06:24:40.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:24:40.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=95, tag_tid=17, entry_tid=18863' = 'object_number=95, tag_tid=17, entry_tid=18771' 2024-09-11T06:24:40.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=95, tag_tid=17, entry_tid=18863' 2024-09-11T06:24:40.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:40.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:40.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:40.721 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:24:40.721 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=95, tag_tid=17, entry_tid=18863' 2024-09-11T06:24:40.721 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:40.721 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:40.722 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:24:40.723 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:40.723 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:40.723 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:40.724 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:24:40.725 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=95, tag_tid=17, entry_tid=18863' 2024-09-11T06:24:40.725 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:40.725 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:40.726 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=18863 2024-09-11T06:24:40.726 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 18863 2024-09-11T06:24:40.726 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:24:40.726 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:24:40.726 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:24:40.928 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:24:40.928 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:40.928 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:24:40.929 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:24:40.929 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:24:40.929 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:24:40.929 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:24:40.929 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:24:40.929 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:24:40.929 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:40.929 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:40.929 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:24:40.929 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:24:40.929 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:24:40.929 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:24:40.930 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:24:40.930 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:24:40.933 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:24:40.933 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:24:40.933 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:24:40.933 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:40.933 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:24:40.933 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:24:40.933 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:24:40.933 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:24:40.933 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:24:40.933 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:24:40.934 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:24:40.934 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:40.934 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:24:40.934 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:40.934 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:24:40.934 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:40.934 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:24:40.934 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:24:40.934 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:24:40.934 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:24:40.934 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:24:40.934 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:24:40.934 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:24:40.934 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:24:40.935 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:40.935 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:40.935 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:40.935 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:24:40.935 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:24:40.935 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:24:40.935 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:24:40.935 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:24:40.959 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:40.959 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:40.960 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:40.960 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:24:41.089 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:24:41.090 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:24:41.090 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:41.091 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:41.091 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:41.091 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:24:41.091 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:41.091 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:41.091 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:41.091 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:24:41.092 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:41.092 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:41.092 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:41.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:41.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:41.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:41.146 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 23 2024-09-11T06:24:41.146 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:41.146 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:41.146 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:41.147 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=94, tag_tid=17, entry_tid=18950], [object_number=93, tag_tid=17, entry_tid=18949], [object_number=92, tag_tid=17, entry_tid=18948], [object_number=95, tag_tid=17, entry_tid=18947]]], state=connected] 2024-09-11T06:24:41.150 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:41.151 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:41.152 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:24:41.152 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:41.152 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:41.153 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:41.153 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:24:41.153 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:41.153 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:41.153 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:41.153 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:24:41.153 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:41.153 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:41.153 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:41.154 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:41.154 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:41.154 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:41.201 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 23 2024-09-11T06:24:41.201 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:41.201 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:41.201 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:41.201 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=94, tag_tid=17, entry_tid=18950], [object_number=93, tag_tid=17, entry_tid=18949], [object_number=92, tag_tid=17, entry_tid=18948], [object_number=95, tag_tid=17, entry_tid=18947]]], state=connected] 2024-09-11T06:24:41.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:41.206 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=94, tag_tid=17, entry_tid=18950' 2024-09-11T06:24:41.206 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=94, tag_tid=17, entry_tid=18950' 2024-09-11T06:24:41.206 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=94, tag_tid=17, entry_tid=18950' '!=' 'object_number=95, tag_tid=17, entry_tid=18863' 2024-09-11T06:24:41.206 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:24:41.207 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=94, tag_tid=17, entry_tid=18950' = 'object_number=95, tag_tid=17, entry_tid=18863' 2024-09-11T06:24:41.207 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=94, tag_tid=17, entry_tid=18950' 2024-09-11T06:24:41.207 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:41.207 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:41.207 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:41.207 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:24:41.208 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=94, tag_tid=17, entry_tid=18950' 2024-09-11T06:24:41.208 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:41.208 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:41.209 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:24:41.210 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:41.210 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:41.210 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:41.211 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:24:41.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=94, tag_tid=17, entry_tid=18950' 2024-09-11T06:24:41.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:41.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:41.213 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=18950 2024-09-11T06:24:41.213 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 18950 2024-09-11T06:24:41.213 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:24:41.213 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:24:41.213 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:24:41.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:24:41.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:41.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:24:41.416 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:24:41.416 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:24:41.416 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:24:41.416 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:24:41.416 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:24:41.416 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:24:41.416 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:41.416 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:41.416 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:24:41.416 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:24:41.416 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:24:41.416 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:24:41.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:24:41.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:24:41.420 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:24:41.420 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:24:41.420 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:24:41.420 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:41.420 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:24:41.420 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:24:41.420 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:24:41.420 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:24:41.420 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:24:41.420 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:24:41.420 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:24:41.421 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:41.421 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:24:41.421 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:41.421 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:24:41.421 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:41.421 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:24:41.421 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:24:41.421 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:24:41.422 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:24:41.422 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:24:41.422 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:24:41.422 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:24:41.422 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:24:41.422 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:41.422 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:41.422 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:41.422 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:24:41.422 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:24:41.422 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:24:41.423 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:24:41.423 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:24:41.448 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:41.449 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:41.449 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:41.449 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:24:41.582 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:24:41.583 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:24:41.583 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:41.583 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:41.583 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:41.583 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:24:41.584 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:41.584 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:41.584 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:41.584 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:24:41.584 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:41.584 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:41.584 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:41.586 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:41.587 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:41.587 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:41.633 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 23 2024-09-11T06:24:41.633 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:41.633 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:41.633 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:41.633 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=94, tag_tid=17, entry_tid=19038], [object_number=93, tag_tid=17, entry_tid=19037], [object_number=92, tag_tid=17, entry_tid=19036], [object_number=95, tag_tid=17, entry_tid=19035]]], state=connected] 2024-09-11T06:24:41.637 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:41.638 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:41.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:24:41.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:41.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:41.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:41.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:24:41.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:41.640 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:41.640 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:41.640 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:24:41.640 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:41.640 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:41.640 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:41.640 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:41.640 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:41.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:41.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 23 2024-09-11T06:24:41.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:41.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:41.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:41.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=94, tag_tid=17, entry_tid=19038], [object_number=93, tag_tid=17, entry_tid=19037], [object_number=92, tag_tid=17, entry_tid=19036], [object_number=95, tag_tid=17, entry_tid=19035]]], state=connected] 2024-09-11T06:24:41.690 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:41.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=94, tag_tid=17, entry_tid=19038' 2024-09-11T06:24:41.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=94, tag_tid=17, entry_tid=19038' 2024-09-11T06:24:41.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=94, tag_tid=17, entry_tid=19038' '!=' 'object_number=94, tag_tid=17, entry_tid=18950' 2024-09-11T06:24:41.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:24:41.693 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=94, tag_tid=17, entry_tid=19038' = 'object_number=94, tag_tid=17, entry_tid=18950' 2024-09-11T06:24:41.693 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=94, tag_tid=17, entry_tid=19038' 2024-09-11T06:24:41.693 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:41.693 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:41.693 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:41.694 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:24:41.694 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=94, tag_tid=17, entry_tid=19038' 2024-09-11T06:24:41.695 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:41.695 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:41.695 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:24:41.696 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:41.696 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:41.696 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:41.697 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:24:41.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=94, tag_tid=17, entry_tid=19038' 2024-09-11T06:24:41.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:41.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:41.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=19038 2024-09-11T06:24:41.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 19038 2024-09-11T06:24:41.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:24:41.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:24:41.700 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:24:41.901 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:24:41.901 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:41.901 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:24:41.901 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:24:41.901 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:24:41.902 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:24:41.902 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:24:41.902 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:24:41.902 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:24:41.902 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:41.902 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:41.902 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:24:41.902 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:24:41.902 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:24:41.902 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:24:41.902 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:24:41.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:24:41.905 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:24:41.905 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:24:41.905 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:24:41.905 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:41.905 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:24:41.905 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:24:41.906 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:24:41.906 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:24:41.906 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:24:41.906 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:24:41.906 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:24:41.906 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:41.906 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:24:41.906 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:41.906 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:24:41.906 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:41.907 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:24:41.907 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:24:41.907 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:24:41.907 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:24:41.907 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:24:41.908 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:24:41.908 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:24:41.908 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:24:41.908 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:41.908 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:41.908 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:41.908 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:24:41.908 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:24:41.908 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:24:41.908 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:24:41.908 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:24:41.932 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:41.933 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:41.933 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:41.933 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:24:42.065 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:24:42.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:24:42.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:42.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:42.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:42.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:24:42.067 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:42.067 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:42.067 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:42.067 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:24:42.067 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:42.067 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:42.067 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:42.068 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:42.069 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:42.069 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:42.128 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 23 2024-09-11T06:24:42.128 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:42.128 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:42.128 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:42.128 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=93, tag_tid=17, entry_tid=19125], [object_number=92, tag_tid=17, entry_tid=19124], [object_number=95, tag_tid=17, entry_tid=19123], [object_number=94, tag_tid=17, entry_tid=19122]]], state=connected] 2024-09-11T06:24:42.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:42.133 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:42.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:24:42.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:42.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:42.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:42.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:24:42.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:42.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:42.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:42.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:24:42.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:42.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:42.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:42.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:42.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:42.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:42.183 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 23 2024-09-11T06:24:42.183 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:42.183 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:42.183 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:42.183 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=93, tag_tid=17, entry_tid=19125], [object_number=92, tag_tid=17, entry_tid=19124], [object_number=95, tag_tid=17, entry_tid=19123], [object_number=94, tag_tid=17, entry_tid=19122]]], state=connected] 2024-09-11T06:24:42.187 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:42.188 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=93, tag_tid=17, entry_tid=19125' 2024-09-11T06:24:42.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=93, tag_tid=17, entry_tid=19125' 2024-09-11T06:24:42.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=93, tag_tid=17, entry_tid=19125' '!=' 'object_number=94, tag_tid=17, entry_tid=19038' 2024-09-11T06:24:42.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:24:42.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=93, tag_tid=17, entry_tid=19125' = 'object_number=94, tag_tid=17, entry_tid=19038' 2024-09-11T06:24:42.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=93, tag_tid=17, entry_tid=19125' 2024-09-11T06:24:42.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:42.190 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:42.190 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:42.190 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:24:42.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=17, entry_tid=19125' 2024-09-11T06:24:42.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:42.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:42.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:24:42.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:42.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:42.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:42.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:24:42.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=17, entry_tid=19125' 2024-09-11T06:24:42.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:42.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:42.196 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=19125 2024-09-11T06:24:42.196 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 19125 2024-09-11T06:24:42.196 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:24:42.196 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:24:42.196 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:24:42.397 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:24:42.397 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:42.397 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:24:42.398 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:24:42.398 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:24:42.398 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:24:42.398 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:24:42.398 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:24:42.398 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:24:42.398 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:42.398 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:42.398 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:24:42.398 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:24:42.398 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:24:42.398 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:24:42.399 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:24:42.399 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:24:42.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:24:42.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:24:42.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:24:42.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:42.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:24:42.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:24:42.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:24:42.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:24:42.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:24:42.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:24:42.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:24:42.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:42.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:24:42.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:42.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:24:42.403 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:42.403 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:24:42.403 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:24:42.403 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:24:42.403 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:24:42.403 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:24:42.403 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:24:42.403 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:24:42.403 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:24:42.403 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:42.403 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:42.403 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:42.403 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:24:42.403 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:24:42.404 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:24:42.404 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:24:42.404 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:24:42.430 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:42.431 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:42.431 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:42.431 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:24:42.569 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:24:42.570 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:24:42.570 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:42.570 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:42.570 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:42.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:24:42.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:42.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:42.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:42.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:24:42.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:42.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:42.572 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:42.574 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:42.574 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:42.574 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:42.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 23 2024-09-11T06:24:42.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:42.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:42.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:42.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=93, tag_tid=17, entry_tid=19221], [object_number=92, tag_tid=17, entry_tid=19220], [object_number=95, tag_tid=17, entry_tid=19219], [object_number=94, tag_tid=17, entry_tid=19218]]], state=connected] 2024-09-11T06:24:42.623 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:42.625 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:42.626 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:24:42.626 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:42.626 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:42.626 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:42.626 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:24:42.626 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:42.626 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:42.626 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:42.626 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:24:42.627 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:42.627 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:42.627 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:42.627 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:42.627 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:42.627 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:42.674 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 23 2024-09-11T06:24:42.674 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:42.674 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:42.674 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:42.674 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=93, tag_tid=17, entry_tid=19221], [object_number=92, tag_tid=17, entry_tid=19220], [object_number=95, tag_tid=17, entry_tid=19219], [object_number=94, tag_tid=17, entry_tid=19218]]], state=connected] 2024-09-11T06:24:42.677 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:42.679 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=93, tag_tid=17, entry_tid=19221' 2024-09-11T06:24:42.679 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=93, tag_tid=17, entry_tid=19221' 2024-09-11T06:24:42.680 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=93, tag_tid=17, entry_tid=19221' '!=' 'object_number=93, tag_tid=17, entry_tid=19125' 2024-09-11T06:24:42.680 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:24:42.680 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=93, tag_tid=17, entry_tid=19221' = 'object_number=93, tag_tid=17, entry_tid=19125' 2024-09-11T06:24:42.680 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=93, tag_tid=17, entry_tid=19221' 2024-09-11T06:24:42.680 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:42.680 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:42.680 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:42.680 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:24:42.681 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=17, entry_tid=19221' 2024-09-11T06:24:42.681 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:42.681 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:42.682 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:24:42.683 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:42.683 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:42.683 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:42.684 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:24:42.685 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=17, entry_tid=19221' 2024-09-11T06:24:42.685 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:42.685 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:42.686 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=19221 2024-09-11T06:24:42.686 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 19221 2024-09-11T06:24:42.686 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:24:42.686 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:24:42.686 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:24:42.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:24:42.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:42.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:24:42.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:24:42.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:24:42.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:24:42.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:24:42.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:24:42.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:24:42.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:42.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:42.889 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:24:42.889 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:24:42.889 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:24:42.889 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:24:42.889 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:24:42.889 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:24:42.894 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:24:42.894 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:24:42.894 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:24:42.894 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:42.894 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:24:42.894 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:24:42.894 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:24:42.894 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:24:42.894 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:24:42.894 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:24:42.894 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:24:42.895 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:42.895 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:24:42.895 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:42.895 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:24:42.895 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:42.895 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:24:42.895 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:24:42.895 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:24:42.895 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:24:42.895 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:24:42.895 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:24:42.895 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:24:42.895 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:24:42.895 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:42.896 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:42.896 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:42.896 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:24:42.896 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:24:42.896 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:24:42.896 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:24:42.896 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:24:42.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:42.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:42.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:42.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:24:43.050 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:24:43.051 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:24:43.051 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:43.051 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:43.051 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:43.051 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:24:43.051 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:43.051 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:43.051 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:43.051 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:24:43.052 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:43.052 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:43.052 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:43.053 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:43.054 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:43.054 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:43.107 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 23 2024-09-11T06:24:43.107 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:43.107 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:43.107 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:43.108 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=93, tag_tid=17, entry_tid=19305], [object_number=92, tag_tid=17, entry_tid=19304], [object_number=95, tag_tid=17, entry_tid=19303], [object_number=94, tag_tid=17, entry_tid=19302]]], state=connected] 2024-09-11T06:24:43.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:43.112 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:43.113 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:24:43.113 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:43.113 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:43.113 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:43.113 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:24:43.114 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:43.114 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:43.114 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:43.114 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:24:43.114 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:43.114 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:43.114 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:43.115 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:43.115 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:43.115 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:43.162 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 23 2024-09-11T06:24:43.163 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:43.163 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:43.163 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:43.163 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=93, tag_tid=17, entry_tid=19305], [object_number=92, tag_tid=17, entry_tid=19304], [object_number=95, tag_tid=17, entry_tid=19303], [object_number=94, tag_tid=17, entry_tid=19302]]], state=connected] 2024-09-11T06:24:43.166 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:43.168 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=93, tag_tid=17, entry_tid=19305' 2024-09-11T06:24:43.168 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=93, tag_tid=17, entry_tid=19305' 2024-09-11T06:24:43.168 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=93, tag_tid=17, entry_tid=19305' '!=' 'object_number=93, tag_tid=17, entry_tid=19221' 2024-09-11T06:24:43.168 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:24:43.168 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=93, tag_tid=17, entry_tid=19305' = 'object_number=93, tag_tid=17, entry_tid=19221' 2024-09-11T06:24:43.168 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=93, tag_tid=17, entry_tid=19305' 2024-09-11T06:24:43.169 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:43.169 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:43.169 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:43.169 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:24:43.170 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=17, entry_tid=19305' 2024-09-11T06:24:43.170 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:43.170 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:43.171 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:24:43.171 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:43.172 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:43.172 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:43.172 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:24:43.173 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=17, entry_tid=19305' 2024-09-11T06:24:43.173 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:43.173 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:43.174 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=19305 2024-09-11T06:24:43.175 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 19305 2024-09-11T06:24:43.175 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:24:43.175 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:24:43.175 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:24:43.376 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:24:43.376 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:43.376 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:24:43.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:24:43.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:24:43.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:24:43.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:24:43.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:24:43.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:24:43.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:43.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:43.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:24:43.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:24:43.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:24:43.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:24:43.378 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:24:43.378 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:24:43.381 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:24:43.381 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:24:43.381 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:24:43.382 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:43.382 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:24:43.382 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:24:43.382 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:24:43.382 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:24:43.382 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:24:43.382 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:24:43.382 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:24:43.382 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:43.382 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:24:43.382 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:43.382 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:24:43.382 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:43.383 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:24:43.383 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:24:43.383 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:24:43.383 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:24:43.383 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:24:43.383 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:24:43.383 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:24:43.383 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:24:43.383 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:43.383 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:43.383 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:43.383 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:24:43.383 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:24:43.384 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:24:43.384 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:24:43.384 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:24:43.408 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:43.409 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:43.409 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:43.409 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:24:43.546 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:24:43.547 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:24:43.547 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:43.547 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:43.548 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:43.548 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:24:43.548 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:43.548 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:43.548 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:43.548 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:24:43.548 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:43.549 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:43.549 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:43.550 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:43.550 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:43.550 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:43.602 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 23 2024-09-11T06:24:43.602 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:43.602 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:43.602 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:43.603 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=92, tag_tid=17, entry_tid=19384], [object_number=95, tag_tid=17, entry_tid=19383], [object_number=94, tag_tid=17, entry_tid=19382], [object_number=93, tag_tid=17, entry_tid=19381]]], state=connected] 2024-09-11T06:24:43.605 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:43.607 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:43.608 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:24:43.608 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:43.609 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:43.609 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:43.609 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:24:43.609 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:43.609 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:43.609 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:43.609 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:24:43.609 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:43.609 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:43.609 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:43.610 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:43.610 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:43.610 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:43.658 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 23 2024-09-11T06:24:43.658 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:43.658 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:43.658 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:43.658 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=92, tag_tid=17, entry_tid=19384], [object_number=95, tag_tid=17, entry_tid=19383], [object_number=94, tag_tid=17, entry_tid=19382], [object_number=93, tag_tid=17, entry_tid=19381]]], state=connected] 2024-09-11T06:24:43.661 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:43.664 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=92, tag_tid=17, entry_tid=19384' 2024-09-11T06:24:43.664 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=92, tag_tid=17, entry_tid=19384' 2024-09-11T06:24:43.664 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=92, tag_tid=17, entry_tid=19384' '!=' 'object_number=93, tag_tid=17, entry_tid=19305' 2024-09-11T06:24:43.664 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:24:43.664 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=92, tag_tid=17, entry_tid=19384' = 'object_number=93, tag_tid=17, entry_tid=19305' 2024-09-11T06:24:43.664 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=92, tag_tid=17, entry_tid=19384' 2024-09-11T06:24:43.664 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:43.664 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:43.664 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:43.665 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:24:43.665 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=92, tag_tid=17, entry_tid=19384' 2024-09-11T06:24:43.665 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:43.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:43.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:24:43.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:43.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:43.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:43.668 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:24:43.669 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=92, tag_tid=17, entry_tid=19384' 2024-09-11T06:24:43.669 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:43.669 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:43.670 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=19384 2024-09-11T06:24:43.670 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 19384 2024-09-11T06:24:43.670 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:24:43.670 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:24:43.671 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:24:43.872 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:24:43.872 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:43.872 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:24:43.872 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:24:43.873 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:24:43.873 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:24:43.873 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:24:43.873 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:24:43.873 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:24:43.873 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:43.873 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:43.873 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:24:43.873 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:24:43.873 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:24:43.873 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:24:43.874 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:24:43.874 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:24:43.877 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:24:43.877 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:24:43.877 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:24:43.877 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:43.877 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:24:43.877 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:24:43.877 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:24:43.877 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:24:43.877 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:24:43.878 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:24:43.878 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:24:43.878 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:43.878 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:24:43.878 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:43.878 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:24:43.878 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:43.878 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:24:43.878 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:24:43.878 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:24:43.878 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:24:43.878 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:24:43.878 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:24:43.879 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:24:43.879 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:24:43.879 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:43.879 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:43.879 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:43.879 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:24:43.879 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:24:43.879 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:24:43.879 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:24:43.879 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:24:43.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:43.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:43.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:43.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:24:44.037 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:24:44.038 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:24:44.038 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:44.038 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:44.038 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:44.038 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:24:44.038 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:44.038 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:44.038 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:44.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:24:44.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:44.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:44.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:44.040 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:44.041 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:44.041 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:44.089 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 23 2024-09-11T06:24:44.089 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:44.089 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:44.089 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:44.089 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=93, tag_tid=17, entry_tid=19461], [object_number=92, tag_tid=17, entry_tid=19460], [object_number=95, tag_tid=17, entry_tid=19459], [object_number=94, tag_tid=17, entry_tid=19458]]], state=connected] 2024-09-11T06:24:44.093 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:44.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:44.095 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:24:44.095 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:44.095 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:44.095 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:44.095 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:24:44.095 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:44.096 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:44.096 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:44.096 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:24:44.096 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:44.096 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:44.096 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:44.097 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:44.097 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:44.097 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:44.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 23 2024-09-11T06:24:44.143 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:44.143 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:44.143 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:44.143 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=93, tag_tid=17, entry_tid=19461], [object_number=92, tag_tid=17, entry_tid=19460], [object_number=95, tag_tid=17, entry_tid=19459], [object_number=94, tag_tid=17, entry_tid=19458]]], state=connected] 2024-09-11T06:24:44.146 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:44.148 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=93, tag_tid=17, entry_tid=19461' 2024-09-11T06:24:44.148 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=93, tag_tid=17, entry_tid=19461' 2024-09-11T06:24:44.148 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=93, tag_tid=17, entry_tid=19461' '!=' 'object_number=92, tag_tid=17, entry_tid=19384' 2024-09-11T06:24:44.148 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:24:44.148 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=93, tag_tid=17, entry_tid=19461' = 'object_number=92, tag_tid=17, entry_tid=19384' 2024-09-11T06:24:44.148 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=93, tag_tid=17, entry_tid=19461' 2024-09-11T06:24:44.149 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:44.149 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:44.149 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:44.150 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:24:44.150 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=17, entry_tid=19461' 2024-09-11T06:24:44.150 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:44.151 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:44.151 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:24:44.152 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:44.152 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:44.152 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:44.153 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:24:44.154 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=17, entry_tid=19461' 2024-09-11T06:24:44.154 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:44.154 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:44.155 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=19461 2024-09-11T06:24:44.155 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 19461 2024-09-11T06:24:44.155 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:24:44.155 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:24:44.155 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:24:44.357 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:24:44.486 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:44.486 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:24:44.486 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:24:44.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:24:44.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:24:44.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:24:44.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:24:44.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:24:44.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:44.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:44.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:24:44.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:24:44.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:24:44.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:24:44.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:24:44.493 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:24:44.493 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:24:44.493 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:24:44.493 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:24:44.493 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:44.493 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:24:44.493 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:24:44.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:24:44.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:24:44.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:24:44.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:24:44.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:24:44.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:44.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:24:44.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:44.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:24:44.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:44.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:24:44.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:24:44.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:24:44.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:24:44.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:24:44.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:24:44.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:24:44.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:24:44.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:44.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:44.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:44.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:24:44.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:24:44.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:24:44.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:24:44.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:24:44.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:44.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:44.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:44.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:24:44.519 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:24:44.520 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:24:44.520 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:44.521 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:44.521 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:44.521 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:24:44.521 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:44.521 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:44.521 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:44.521 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:24:44.522 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:44.522 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:44.522 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:44.524 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:44.524 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:44.524 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:44.574 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 23 2024-09-11T06:24:44.574 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:44.574 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:44.574 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:44.574 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=94, tag_tid=17, entry_tid=19550], [object_number=93, tag_tid=17, entry_tid=19549], [object_number=92, tag_tid=17, entry_tid=19548], [object_number=95, tag_tid=17, entry_tid=19547]]], state=connected] 2024-09-11T06:24:44.577 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:44.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:44.580 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:24:44.580 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:44.580 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:44.580 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:44.580 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:24:44.580 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:44.580 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:44.580 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:44.580 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:24:44.581 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:44.581 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:44.581 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:44.581 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:44.581 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:44.581 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:44.629 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 23 2024-09-11T06:24:44.629 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:44.629 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:44.629 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:44.629 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=94, tag_tid=17, entry_tid=19550], [object_number=93, tag_tid=17, entry_tid=19549], [object_number=92, tag_tid=17, entry_tid=19548], [object_number=95, tag_tid=17, entry_tid=19547]]], state=connected] 2024-09-11T06:24:44.632 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:44.634 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=94, tag_tid=17, entry_tid=19550' 2024-09-11T06:24:44.634 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=94, tag_tid=17, entry_tid=19550' 2024-09-11T06:24:44.634 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=94, tag_tid=17, entry_tid=19550' '!=' 'object_number=93, tag_tid=17, entry_tid=19461' 2024-09-11T06:24:44.634 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:24:44.634 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=94, tag_tid=17, entry_tid=19550' = 'object_number=93, tag_tid=17, entry_tid=19461' 2024-09-11T06:24:44.634 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=94, tag_tid=17, entry_tid=19550' 2024-09-11T06:24:44.635 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:44.635 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:44.635 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:44.635 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:24:44.636 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=94, tag_tid=17, entry_tid=19550' 2024-09-11T06:24:44.636 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:44.636 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:44.637 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:24:44.638 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:44.638 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:44.638 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:44.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:24:44.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=94, tag_tid=17, entry_tid=19550' 2024-09-11T06:24:44.640 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:44.640 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:44.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=19550 2024-09-11T06:24:44.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 19550 2024-09-11T06:24:44.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:24:44.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:24:44.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:24:44.843 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:24:44.843 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:44.843 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:24:44.843 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:24:44.843 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:24:44.843 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:24:44.843 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:24:44.843 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:24:44.843 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:24:44.843 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:44.844 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:44.844 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:24:44.844 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:24:44.844 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:24:44.844 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:24:44.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:24:44.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:24:44.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:24:44.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:24:44.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:24:44.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:44.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:24:44.848 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:24:44.848 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:24:44.848 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:24:44.848 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:24:44.848 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:24:44.848 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:24:44.848 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:44.848 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:24:44.848 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:44.848 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:24:44.848 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:44.848 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:24:44.848 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:24:44.848 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:24:44.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:24:44.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:24:44.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:24:44.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:24:44.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:24:44.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:44.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:44.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:44.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:24:44.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:24:44.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:24:44.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:24:44.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:24:44.867 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:44.868 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:44.868 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:44.868 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:24:44.999 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:24:45.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:24:45.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:45.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:45.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:45.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:24:45.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:45.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:45.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:45.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:24:45.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:45.002 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:45.002 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:45.003 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:45.003 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:45.003 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:45.050 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 23 2024-09-11T06:24:45.050 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:45.050 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:45.050 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:45.050 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=95, tag_tid=17, entry_tid=19639], [object_number=94, tag_tid=17, entry_tid=19638], [object_number=93, tag_tid=17, entry_tid=19637], [object_number=92, tag_tid=17, entry_tid=19636]]], state=connected] 2024-09-11T06:24:45.053 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:45.054 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:45.056 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:24:45.056 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:45.056 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:45.056 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:45.056 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:24:45.056 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:45.056 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:45.056 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:45.056 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:24:45.057 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:45.057 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:45.057 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:45.057 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:45.057 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:45.058 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:45.103 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 23 2024-09-11T06:24:45.103 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:45.104 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:45.104 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:45.104 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=95, tag_tid=17, entry_tid=19639], [object_number=94, tag_tid=17, entry_tid=19638], [object_number=93, tag_tid=17, entry_tid=19637], [object_number=92, tag_tid=17, entry_tid=19636]]], state=connected] 2024-09-11T06:24:45.106 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:45.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=95, tag_tid=17, entry_tid=19639' 2024-09-11T06:24:45.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=95, tag_tid=17, entry_tid=19639' 2024-09-11T06:24:45.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=95, tag_tid=17, entry_tid=19639' '!=' 'object_number=94, tag_tid=17, entry_tid=19550' 2024-09-11T06:24:45.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:24:45.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=95, tag_tid=17, entry_tid=19639' = 'object_number=94, tag_tid=17, entry_tid=19550' 2024-09-11T06:24:45.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=95, tag_tid=17, entry_tid=19639' 2024-09-11T06:24:45.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:45.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:45.110 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:45.110 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:24:45.110 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=95, tag_tid=17, entry_tid=19639' 2024-09-11T06:24:45.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:45.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:45.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:24:45.112 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:45.112 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:45.112 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:45.113 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:24:45.114 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=95, tag_tid=17, entry_tid=19639' 2024-09-11T06:24:45.114 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:45.114 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:45.115 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=19639 2024-09-11T06:24:45.115 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 19639 2024-09-11T06:24:45.115 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:24:45.115 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:24:45.115 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:24:45.317 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:24:45.317 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:45.317 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:24:45.317 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:24:45.317 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:24:45.317 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:24:45.317 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:24:45.317 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:24:45.317 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:24:45.318 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:45.318 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:45.318 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:24:45.318 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:24:45.318 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:24:45.318 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:24:45.318 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:24:45.319 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:24:45.322 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:24:45.322 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:24:45.322 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:24:45.322 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:45.322 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:24:45.322 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:24:45.322 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:24:45.322 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:24:45.322 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:24:45.323 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:24:45.323 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:24:45.323 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:45.323 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:24:45.323 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:45.323 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:24:45.323 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:45.323 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:24:45.323 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:24:45.323 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:24:45.323 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:24:45.323 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:24:45.323 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:24:45.323 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:24:45.324 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:24:45.324 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:45.324 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:45.324 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:45.324 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:24:45.324 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:24:45.324 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:24:45.324 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:24:45.324 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:24:45.348 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:45.348 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:45.349 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:45.349 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:24:45.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:24:45.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:24:45.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:45.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:45.483 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:45.483 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:24:45.483 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:45.483 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:45.483 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:45.483 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:24:45.484 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:45.485 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:45.485 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:45.485 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:45.485 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:45.486 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:45.531 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 23 2024-09-11T06:24:45.531 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:45.531 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:45.532 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:45.532 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=93, tag_tid=17, entry_tid=19729], [object_number=92, tag_tid=17, entry_tid=19728], [object_number=95, tag_tid=17, entry_tid=19727], [object_number=94, tag_tid=17, entry_tid=19726]]], state=connected] 2024-09-11T06:24:45.534 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:45.536 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:45.538 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:24:45.539 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:45.539 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:45.539 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:45.539 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:24:45.539 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:45.540 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:45.540 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:45.540 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:24:45.540 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:45.540 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:45.541 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:45.541 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:45.541 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:45.541 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:45.585 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 23 2024-09-11T06:24:45.585 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:45.585 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:45.586 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:45.586 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=93, tag_tid=17, entry_tid=19729], [object_number=92, tag_tid=17, entry_tid=19728], [object_number=95, tag_tid=17, entry_tid=19727], [object_number=94, tag_tid=17, entry_tid=19726]]], state=connected] 2024-09-11T06:24:45.588 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:45.591 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=93, tag_tid=17, entry_tid=19729' 2024-09-11T06:24:45.591 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=93, tag_tid=17, entry_tid=19729' 2024-09-11T06:24:45.591 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=93, tag_tid=17, entry_tid=19729' '!=' 'object_number=95, tag_tid=17, entry_tid=19639' 2024-09-11T06:24:45.591 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:24:45.592 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=93, tag_tid=17, entry_tid=19729' = 'object_number=95, tag_tid=17, entry_tid=19639' 2024-09-11T06:24:45.592 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=93, tag_tid=17, entry_tid=19729' 2024-09-11T06:24:45.592 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:45.592 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:45.592 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:45.594 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:24:45.594 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=17, entry_tid=19729' 2024-09-11T06:24:45.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:45.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:45.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:24:45.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:45.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:45.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:45.597 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:24:45.598 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=17, entry_tid=19729' 2024-09-11T06:24:45.598 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:45.598 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:45.598 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=19729 2024-09-11T06:24:45.598 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 19729 2024-09-11T06:24:45.599 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:24:45.599 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:24:45.599 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:24:45.799 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:24:45.799 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:45.799 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:24:45.799 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:24:45.799 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:24:45.799 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:24:45.800 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:24:45.800 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:24:45.800 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:24:45.800 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:45.800 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:45.800 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:24:45.800 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:24:45.800 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:24:45.800 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:24:45.800 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:24:45.801 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:24:45.804 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:24:45.804 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:24:45.804 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:24:45.804 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:45.804 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:24:45.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:24:45.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:24:45.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:24:45.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:24:45.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:24:45.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:24:45.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:45.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:24:45.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:45.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:24:45.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:45.807 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:24:45.807 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:24:45.807 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:24:45.807 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:24:45.807 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:24:45.807 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:24:45.807 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:24:45.807 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:24:45.807 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:45.807 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:45.808 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:45.808 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:24:45.808 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:24:45.808 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:24:45.808 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:24:45.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:24:45.831 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:45.831 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:45.831 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:45.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:24:45.960 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:24:45.961 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:24:45.961 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:45.962 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:45.962 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:45.962 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:24:45.962 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:45.962 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:45.962 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:45.962 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:24:45.962 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:45.963 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:45.963 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:45.964 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:45.964 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:45.964 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:46.017 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 23 2024-09-11T06:24:46.017 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:46.017 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:46.017 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:46.017 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=94, tag_tid=17, entry_tid=19818], [object_number=93, tag_tid=17, entry_tid=19817], [object_number=92, tag_tid=17, entry_tid=19816], [object_number=95, tag_tid=17, entry_tid=19815]]], state=connected] 2024-09-11T06:24:46.020 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:46.021 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:46.022 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:24:46.022 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:46.022 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:46.022 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:46.022 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:24:46.022 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:46.023 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:46.023 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:46.023 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:24:46.023 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:46.023 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:46.023 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:46.023 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:46.024 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:46.024 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:46.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 23 2024-09-11T06:24:46.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:46.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:46.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:46.071 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=94, tag_tid=17, entry_tid=19818], [object_number=93, tag_tid=17, entry_tid=19817], [object_number=92, tag_tid=17, entry_tid=19816], [object_number=95, tag_tid=17, entry_tid=19815]]], state=connected] 2024-09-11T06:24:46.074 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:46.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=94, tag_tid=17, entry_tid=19818' 2024-09-11T06:24:46.076 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=94, tag_tid=17, entry_tid=19818' 2024-09-11T06:24:46.076 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=94, tag_tid=17, entry_tid=19818' '!=' 'object_number=93, tag_tid=17, entry_tid=19729' 2024-09-11T06:24:46.076 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:24:46.076 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=94, tag_tid=17, entry_tid=19818' = 'object_number=93, tag_tid=17, entry_tid=19729' 2024-09-11T06:24:46.076 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=94, tag_tid=17, entry_tid=19818' 2024-09-11T06:24:46.076 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:46.077 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:46.077 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:46.077 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:24:46.078 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=94, tag_tid=17, entry_tid=19818' 2024-09-11T06:24:46.078 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:46.078 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:46.079 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:24:46.079 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:46.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:46.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:46.081 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:24:46.081 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=94, tag_tid=17, entry_tid=19818' 2024-09-11T06:24:46.081 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:46.082 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:46.083 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=19818 2024-09-11T06:24:46.083 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 19818 2024-09-11T06:24:46.083 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:24:46.083 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:24:46.083 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:24:46.284 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:24:46.285 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:46.285 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:24:46.285 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:24:46.285 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:24:46.285 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:24:46.285 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:24:46.285 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:24:46.285 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:24:46.285 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:46.285 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:46.285 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:24:46.285 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:24:46.286 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:24:46.286 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:24:46.286 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:24:46.286 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:24:46.289 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:24:46.289 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:24:46.289 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:24:46.289 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:46.289 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:24:46.289 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:24:46.289 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:24:46.289 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:24:46.289 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:24:46.289 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:24:46.290 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:24:46.290 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:46.290 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:24:46.290 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:46.290 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:24:46.290 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:46.291 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:24:46.291 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:24:46.291 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:24:46.291 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:24:46.291 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:24:46.291 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:24:46.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:24:46.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:24:46.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:46.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:46.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:46.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:24:46.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:24:46.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:24:46.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:24:46.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:24:46.315 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:46.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:46.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:46.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:24:46.447 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:24:46.448 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:24:46.448 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:46.448 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:46.448 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:46.448 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:24:46.448 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:46.449 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:46.449 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:46.449 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:24:46.449 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:46.449 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:46.449 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:46.450 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:46.451 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:46.451 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:46.509 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 23 2024-09-11T06:24:46.509 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:46.509 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:46.509 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:46.509 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=93, tag_tid=17, entry_tid=19905], [object_number=92, tag_tid=17, entry_tid=19904], [object_number=95, tag_tid=17, entry_tid=19903], [object_number=94, tag_tid=17, entry_tid=19902]]], state=connected] 2024-09-11T06:24:46.523 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:46.525 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:46.526 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:24:46.526 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:46.527 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:46.527 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:46.527 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:24:46.527 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:46.527 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:46.527 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:46.527 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:24:46.527 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:46.527 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:46.527 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:46.528 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:46.528 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:46.528 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:46.578 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 23 2024-09-11T06:24:46.578 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:46.578 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:46.578 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:46.578 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=93, tag_tid=17, entry_tid=19905], [object_number=92, tag_tid=17, entry_tid=19904], [object_number=95, tag_tid=17, entry_tid=19903], [object_number=94, tag_tid=17, entry_tid=19902]]], state=connected] 2024-09-11T06:24:46.590 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:46.591 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=93, tag_tid=17, entry_tid=19905' 2024-09-11T06:24:46.592 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=93, tag_tid=17, entry_tid=19905' 2024-09-11T06:24:46.592 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=93, tag_tid=17, entry_tid=19905' '!=' 'object_number=94, tag_tid=17, entry_tid=19818' 2024-09-11T06:24:46.592 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:24:46.592 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=93, tag_tid=17, entry_tid=19905' = 'object_number=94, tag_tid=17, entry_tid=19818' 2024-09-11T06:24:46.592 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=93, tag_tid=17, entry_tid=19905' 2024-09-11T06:24:46.593 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:46.593 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:46.593 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:46.593 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:24:46.594 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=17, entry_tid=19905' 2024-09-11T06:24:46.594 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:46.594 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:46.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:24:46.596 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:46.596 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:46.596 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:46.597 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:24:46.597 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=17, entry_tid=19905' 2024-09-11T06:24:46.597 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:46.598 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:46.599 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=19905 2024-09-11T06:24:46.599 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 19905 2024-09-11T06:24:46.599 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:24:46.599 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:24:46.599 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:24:46.801 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:24:46.801 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:46.801 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:24:46.801 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:24:46.801 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:24:46.801 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:24:46.802 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:24:46.802 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:24:46.802 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:24:46.802 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:46.802 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:46.802 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:24:46.802 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:24:46.802 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:24:46.802 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:24:46.803 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:24:46.803 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:24:46.806 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:24:46.806 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:24:46.806 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:24:46.806 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:46.806 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:24:46.806 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:24:46.806 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:24:46.806 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:24:46.806 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:24:46.807 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:24:46.807 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:24:46.807 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:46.807 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:24:46.807 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:46.807 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:24:46.807 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:46.807 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:24:46.807 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:24:46.807 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:24:46.807 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:24:46.807 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:24:46.807 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:24:46.808 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:24:46.808 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:24:46.808 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:46.808 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:46.808 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:46.808 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:24:46.808 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:24:46.808 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:24:46.808 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:24:46.808 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:24:46.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:46.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:46.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:46.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:24:46.960 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:24:46.965 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:24:46.965 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:46.965 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:46.965 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:46.965 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:24:46.965 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:46.966 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:46.966 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:46.966 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:24:46.966 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:46.966 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:46.966 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:46.966 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:46.966 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:46.966 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:47.022 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 23 2024-09-11T06:24:47.022 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:47.022 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:47.022 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:47.022 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=94, tag_tid=17, entry_tid=19998], [object_number=93, tag_tid=17, entry_tid=19997], [object_number=92, tag_tid=17, entry_tid=19996], [object_number=95, tag_tid=17, entry_tid=19995]]], state=connected] 2024-09-11T06:24:47.026 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:47.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:47.028 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:24:47.028 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:47.028 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:47.028 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:47.029 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:24:47.029 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:47.029 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:47.029 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:47.029 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:24:47.029 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:47.029 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:47.029 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:47.029 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:47.030 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:47.030 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:47.077 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 23 2024-09-11T06:24:47.077 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:47.077 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:47.077 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:47.077 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=94, tag_tid=17, entry_tid=19998], [object_number=93, tag_tid=17, entry_tid=19997], [object_number=92, tag_tid=17, entry_tid=19996], [object_number=95, tag_tid=17, entry_tid=19995]]], state=connected] 2024-09-11T06:24:47.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:47.082 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=94, tag_tid=17, entry_tid=19998' 2024-09-11T06:24:47.082 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=94, tag_tid=17, entry_tid=19998' 2024-09-11T06:24:47.082 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=94, tag_tid=17, entry_tid=19998' '!=' 'object_number=93, tag_tid=17, entry_tid=19905' 2024-09-11T06:24:47.082 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:24:47.082 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=94, tag_tid=17, entry_tid=19998' = 'object_number=93, tag_tid=17, entry_tid=19905' 2024-09-11T06:24:47.082 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=94, tag_tid=17, entry_tid=19998' 2024-09-11T06:24:47.083 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:47.083 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:47.083 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:47.083 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:24:47.084 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=94, tag_tid=17, entry_tid=19998' 2024-09-11T06:24:47.084 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:47.084 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:47.085 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:24:47.086 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:47.086 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:47.086 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:47.087 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:24:47.088 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=94, tag_tid=17, entry_tid=19998' 2024-09-11T06:24:47.088 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:47.088 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:47.089 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=19998 2024-09-11T06:24:47.089 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 19998 2024-09-11T06:24:47.089 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:24:47.089 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:24:47.089 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:24:47.291 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:24:47.291 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:47.291 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:24:47.291 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:24:47.291 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:24:47.291 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:24:47.291 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:24:47.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:24:47.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:24:47.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:47.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:47.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:24:47.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:24:47.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:24:47.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:24:47.293 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:24:47.293 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:24:47.296 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:24:47.296 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:24:47.296 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:24:47.296 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:47.296 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:24:47.296 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:24:47.296 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:24:47.296 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:24:47.296 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:24:47.296 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:24:47.297 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:24:47.297 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:47.297 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:24:47.297 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:47.297 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:24:47.297 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:47.297 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:24:47.297 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:24:47.297 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:24:47.297 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:24:47.297 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:24:47.297 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:24:47.297 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:24:47.297 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:24:47.298 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:47.298 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:47.298 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:47.298 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:24:47.298 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:24:47.298 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:24:47.298 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:24:47.298 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:24:47.322 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:47.323 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:47.323 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:47.323 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:24:47.449 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:24:47.450 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:24:47.450 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:47.450 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:47.450 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:47.450 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:24:47.450 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:47.450 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:47.451 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:47.451 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:24:47.451 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:47.451 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:47.451 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:47.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:47.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:47.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:47.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 23 2024-09-11T06:24:47.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:47.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:47.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:47.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=95, tag_tid=17, entry_tid=20087], [object_number=94, tag_tid=17, entry_tid=20086], [object_number=93, tag_tid=17, entry_tid=20085], [object_number=92, tag_tid=17, entry_tid=20084]]], state=connected] 2024-09-11T06:24:47.503 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:47.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:47.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:24:47.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:47.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:47.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:47.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:24:47.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:47.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:47.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:47.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:24:47.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:47.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:47.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:47.507 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:47.507 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:47.507 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:47.555 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 23 2024-09-11T06:24:47.555 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:47.555 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:47.556 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:47.556 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=95, tag_tid=17, entry_tid=20087], [object_number=94, tag_tid=17, entry_tid=20086], [object_number=93, tag_tid=17, entry_tid=20085], [object_number=92, tag_tid=17, entry_tid=20084]]], state=connected] 2024-09-11T06:24:47.559 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:47.560 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=95, tag_tid=17, entry_tid=20087' 2024-09-11T06:24:47.560 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=95, tag_tid=17, entry_tid=20087' 2024-09-11T06:24:47.561 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=95, tag_tid=17, entry_tid=20087' '!=' 'object_number=94, tag_tid=17, entry_tid=19998' 2024-09-11T06:24:47.561 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:24:47.561 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=95, tag_tid=17, entry_tid=20087' = 'object_number=94, tag_tid=17, entry_tid=19998' 2024-09-11T06:24:47.561 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=95, tag_tid=17, entry_tid=20087' 2024-09-11T06:24:47.561 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:47.562 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:47.562 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:47.562 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:24:47.563 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=95, tag_tid=17, entry_tid=20087' 2024-09-11T06:24:47.563 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:47.563 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:47.564 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:24:47.564 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:47.564 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:47.565 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:47.565 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:24:47.566 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=95, tag_tid=17, entry_tid=20087' 2024-09-11T06:24:47.566 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:47.566 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:47.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=20087 2024-09-11T06:24:47.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 20087 2024-09-11T06:24:47.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:24:47.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:24:47.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:24:47.769 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:24:47.769 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:47.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:24:47.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:24:47.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:24:47.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:24:47.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:24:47.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:24:47.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:24:47.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:47.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:47.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:24:47.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:24:47.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:24:47.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:24:47.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:24:47.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:24:47.774 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:24:47.774 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:24:47.774 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:24:47.774 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:47.774 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:24:47.774 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:24:47.774 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:24:47.774 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:24:47.774 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:24:47.774 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:24:47.775 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:24:47.775 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:47.775 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:24:47.775 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:47.775 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:24:47.775 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:47.775 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:24:47.775 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:24:47.775 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:24:47.775 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:24:47.775 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:24:47.775 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:24:47.775 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:24:47.776 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:24:47.776 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:47.776 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:47.776 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:47.776 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:24:47.776 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:24:47.776 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:24:47.776 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:24:47.776 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:24:47.800 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:47.800 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:47.800 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:47.800 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:24:47.932 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:24:47.933 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:24:47.933 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:47.933 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:47.933 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:47.933 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:24:47.933 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:47.933 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:47.933 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:47.933 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:24:47.934 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:47.934 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:47.934 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:47.936 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:47.936 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:47.936 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:47.986 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 23 2024-09-11T06:24:47.987 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:47.987 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:47.987 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:47.987 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=94, tag_tid=17, entry_tid=20174], [object_number=93, tag_tid=17, entry_tid=20173], [object_number=92, tag_tid=17, entry_tid=20172], [object_number=95, tag_tid=17, entry_tid=20171]]], state=connected] 2024-09-11T06:24:47.987 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:47.987 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:47.989 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:24:47.989 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:47.989 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:47.989 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:47.989 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:24:47.989 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:47.989 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:47.989 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:47.989 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:24:47.989 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:47.989 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:47.989 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:47.990 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:47.990 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:47.990 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:48.037 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 23 2024-09-11T06:24:48.037 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:48.037 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:48.037 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:48.037 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=94, tag_tid=17, entry_tid=20174], [object_number=93, tag_tid=17, entry_tid=20173], [object_number=92, tag_tid=17, entry_tid=20172], [object_number=95, tag_tid=17, entry_tid=20171]]], state=connected] 2024-09-11T06:24:48.040 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:48.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=94, tag_tid=17, entry_tid=20174' 2024-09-11T06:24:48.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=94, tag_tid=17, entry_tid=20174' 2024-09-11T06:24:48.043 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=94, tag_tid=17, entry_tid=20174' '!=' 'object_number=95, tag_tid=17, entry_tid=20087' 2024-09-11T06:24:48.043 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:24:48.043 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=94, tag_tid=17, entry_tid=20174' = 'object_number=95, tag_tid=17, entry_tid=20087' 2024-09-11T06:24:48.043 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=94, tag_tid=17, entry_tid=20174' 2024-09-11T06:24:48.043 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:48.043 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:48.043 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:48.044 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:24:48.044 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=94, tag_tid=17, entry_tid=20174' 2024-09-11T06:24:48.044 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:48.044 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:48.045 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:24:48.046 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:48.046 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:48.046 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:48.047 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:24:48.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=94, tag_tid=17, entry_tid=20174' 2024-09-11T06:24:48.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:48.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:48.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=20174 2024-09-11T06:24:48.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 20174 2024-09-11T06:24:48.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:24:48.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:24:48.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:24:48.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:24:48.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:48.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:24:48.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:24:48.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:24:48.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:24:48.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:24:48.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:24:48.252 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:24:48.252 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:48.252 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:48.252 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:24:48.252 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:24:48.252 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:24:48.252 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:24:48.252 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:24:48.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:24:48.256 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:24:48.256 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:24:48.256 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:24:48.256 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:48.256 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:24:48.256 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:24:48.256 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:24:48.256 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:24:48.256 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:24:48.257 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:24:48.257 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:24:48.257 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:48.257 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:24:48.257 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:48.257 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:24:48.257 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:48.257 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:24:48.257 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:24:48.257 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:24:48.257 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:24:48.257 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:24:48.257 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:24:48.258 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:24:48.258 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:24:48.258 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:48.258 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:48.258 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:48.258 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:24:48.258 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:24:48.258 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:24:48.258 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:24:48.258 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:24:48.281 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:48.281 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:48.282 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:48.282 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:24:48.412 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:24:48.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:24:48.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:48.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:48.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:48.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:24:48.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:48.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:48.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:48.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:24:48.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:48.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:48.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:48.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:48.416 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:48.416 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:48.462 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 23 2024-09-11T06:24:48.462 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:48.462 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:48.462 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:48.462 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=95, tag_tid=17, entry_tid=20263], [object_number=94, tag_tid=17, entry_tid=20262], [object_number=93, tag_tid=17, entry_tid=20261], [object_number=92, tag_tid=17, entry_tid=20260]]], state=connected] 2024-09-11T06:24:48.465 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:48.467 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:48.468 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:24:48.468 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:48.468 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:48.468 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:48.468 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:24:48.468 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:48.468 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:48.468 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:48.468 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:24:48.468 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:48.469 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:48.469 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:48.469 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:48.469 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:48.469 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:48.517 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 23 2024-09-11T06:24:48.517 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:48.517 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:48.517 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:48.517 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=95, tag_tid=17, entry_tid=20263], [object_number=94, tag_tid=17, entry_tid=20262], [object_number=93, tag_tid=17, entry_tid=20261], [object_number=92, tag_tid=17, entry_tid=20260]]], state=connected] 2024-09-11T06:24:48.520 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:48.522 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=95, tag_tid=17, entry_tid=20263' 2024-09-11T06:24:48.522 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=95, tag_tid=17, entry_tid=20263' 2024-09-11T06:24:48.522 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=95, tag_tid=17, entry_tid=20263' '!=' 'object_number=94, tag_tid=17, entry_tid=20174' 2024-09-11T06:24:48.523 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:24:48.523 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=95, tag_tid=17, entry_tid=20263' = 'object_number=94, tag_tid=17, entry_tid=20174' 2024-09-11T06:24:48.523 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=95, tag_tid=17, entry_tid=20263' 2024-09-11T06:24:48.523 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:48.523 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:48.523 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:48.523 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:24:48.524 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=95, tag_tid=17, entry_tid=20263' 2024-09-11T06:24:48.524 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:48.524 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:48.525 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:24:48.526 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:48.526 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:48.526 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:48.527 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:24:48.527 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=95, tag_tid=17, entry_tid=20263' 2024-09-11T06:24:48.527 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:48.528 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:48.529 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=20263 2024-09-11T06:24:48.529 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 20263 2024-09-11T06:24:48.529 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:24:48.529 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:24:48.529 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:24:48.731 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:24:48.731 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:48.731 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:24:48.731 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:24:48.731 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:24:48.731 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:24:48.732 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:24:48.732 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:24:48.732 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:24:48.732 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:48.732 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:48.732 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:24:48.732 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:24:48.732 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:24:48.732 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:24:48.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:24:48.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:24:48.736 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:24:48.737 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:24:48.737 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:24:48.737 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:48.737 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:24:48.737 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:24:48.737 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:24:48.737 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:24:48.737 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:24:48.737 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:24:48.737 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:24:48.737 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:48.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:24:48.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:48.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:24:48.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:48.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:24:48.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:24:48.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:24:48.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:24:48.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:24:48.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:24:48.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:24:48.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:24:48.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:48.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:48.739 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:48.739 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:24:48.739 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:24:48.739 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:24:48.739 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:24:48.739 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:24:48.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:48.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:48.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:48.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:24:48.892 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:24:48.894 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:24:48.894 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:48.894 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:48.894 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:48.894 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:24:48.894 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:48.894 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:48.894 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:48.894 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:24:48.895 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:48.895 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:48.895 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:48.896 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:48.896 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:48.896 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:48.943 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 23 2024-09-11T06:24:48.943 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:48.943 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:48.943 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:48.944 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=94, tag_tid=17, entry_tid=20294], [object_number=93, tag_tid=17, entry_tid=20293], [object_number=92, tag_tid=17, entry_tid=20292], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:48.946 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:48.948 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:48.949 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:24:48.949 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:48.949 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:48.949 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:48.949 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:24:48.949 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:48.949 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:48.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:48.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:24:48.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:48.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:48.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:48.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:48.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:48.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:48.997 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 23 2024-09-11T06:24:48.997 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:48.997 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:48.997 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:48.997 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=94, tag_tid=17, entry_tid=20294], [object_number=93, tag_tid=17, entry_tid=20293], [object_number=92, tag_tid=17, entry_tid=20292], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:49.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:49.002 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=94, tag_tid=17, entry_tid=20294' 2024-09-11T06:24:49.002 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=94, tag_tid=17, entry_tid=20294' 2024-09-11T06:24:49.002 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=94, tag_tid=17, entry_tid=20294' '!=' 'object_number=95, tag_tid=17, entry_tid=20263' 2024-09-11T06:24:49.002 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:24:49.002 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=94, tag_tid=17, entry_tid=20294' = 'object_number=95, tag_tid=17, entry_tid=20263' 2024-09-11T06:24:49.002 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=94, tag_tid=17, entry_tid=20294' 2024-09-11T06:24:49.003 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:49.003 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:49.003 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:49.004 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=18 2024-09-11T06:24:49.004 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=94, tag_tid=17, entry_tid=20294' 2024-09-11T06:24:49.004 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:24:49.004 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:49.005 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=17 2024-09-11T06:24:49.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:49.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:49.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:49.007 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:24:49.008 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=94, tag_tid=17, entry_tid=20294' 2024-09-11T06:24:49.008 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:24:49.008 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:24:49.009 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=20294 2024-09-11T06:24:49.009 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 18 = 17 -a 1 -le 20294 2024-09-11T06:24:49.009 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:24:49.009 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:24:49.009 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:24:49.211 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:24:49.211 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:49.211 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:24:49.211 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:24:49.211 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:24:49.211 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:24:49.211 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:24:49.211 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:24:49.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:24:49.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:49.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:49.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:24:49.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:24:49.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:24:49.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:24:49.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:24:49.213 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:24:49.215 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:24:49.216 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:24:49.216 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:24:49.216 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:49.216 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:24:49.216 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:24:49.216 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:24:49.216 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:24:49.216 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:24:49.216 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:24:49.216 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:24:49.216 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:49.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:24:49.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:49.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:24:49.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:49.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:24:49.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:24:49.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:24:49.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:24:49.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:24:49.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:24:49.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:24:49.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:24:49.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:49.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:49.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:49.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:24:49.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:24:49.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:24:49.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:24:49.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:24:49.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:49.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:49.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:49.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:24:49.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:24:49.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:24:49.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:49.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:49.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:49.362 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:24:49.362 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:49.362 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:49.362 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:49.362 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:24:49.362 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:49.362 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:49.363 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:49.364 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:49.364 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:49.365 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:49.413 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 23 2024-09-11T06:24:49.413 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:49.413 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:49.413 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:49.413 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=94, tag_tid=17, entry_tid=20294], [object_number=93, tag_tid=17, entry_tid=20293], [object_number=92, tag_tid=17, entry_tid=20292], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:49.416 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:49.418 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:49.418 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:24:49.418 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:49.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:49.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:49.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:24:49.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:49.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:49.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:49.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:24:49.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:49.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:49.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:49.420 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:49.420 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:49.420 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:49.468 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 23 2024-09-11T06:24:49.468 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:49.468 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:49.468 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:49.468 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=94, tag_tid=17, entry_tid=20294], [object_number=93, tag_tid=17, entry_tid=20293], [object_number=92, tag_tid=17, entry_tid=20292], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:49.471 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:49.473 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=94, tag_tid=17, entry_tid=20294' 2024-09-11T06:24:49.473 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=94, tag_tid=17, entry_tid=20294' 2024-09-11T06:24:49.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=94, tag_tid=17, entry_tid=20294' '!=' 'object_number=94, tag_tid=17, entry_tid=20294' 2024-09-11T06:24:49.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:24:49.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.4 2024-09-11T06:24:49.875 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:24:49.876 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:49.876 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:24:49.876 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:24:49.876 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:24:49.876 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:24:49.876 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:24:49.876 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:24:49.876 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:24:49.876 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:49.876 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:49.876 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:24:49.876 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:24:49.876 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:24:49.877 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:24:49.877 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:24:49.877 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:24:49.880 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:24:49.880 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:24:49.880 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:24:49.880 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:49.880 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:24:49.880 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:24:49.880 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:24:49.880 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:24:49.880 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:24:49.880 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:24:49.880 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:24:49.880 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:49.881 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:24:49.881 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:49.881 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:24:49.881 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:49.881 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:24:49.881 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:24:49.881 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:24:49.881 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:24:49.881 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:24:49.881 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:24:49.881 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:24:49.881 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:24:49.881 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:49.881 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:49.882 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:49.882 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:24:49.882 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:24:49.882 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:24:49.882 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:24:49.882 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:24:49.905 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:49.905 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:49.905 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:49.905 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:24:50.031 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:24:50.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:24:50.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:50.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:50.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:50.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:24:50.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:50.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:50.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:50.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:24:50.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:50.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:50.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:50.035 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:50.036 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:50.036 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:50.090 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 23 2024-09-11T06:24:50.090 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:50.090 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:50.090 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:50.091 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=94, tag_tid=17, entry_tid=20294], [object_number=93, tag_tid=17, entry_tid=20293], [object_number=92, tag_tid=17, entry_tid=20292], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:50.096 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:50.096 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:50.098 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:24:50.098 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:50.098 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:50.098 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:50.098 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:24:50.098 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:50.098 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:50.098 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:50.098 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:24:50.098 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:50.098 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:50.098 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:50.099 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:50.099 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:50.100 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:50.148 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 23 2024-09-11T06:24:50.149 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:50.149 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:50.149 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:50.149 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=94, tag_tid=17, entry_tid=20294], [object_number=93, tag_tid=17, entry_tid=20293], [object_number=92, tag_tid=17, entry_tid=20292], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:50.152 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:50.154 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=94, tag_tid=17, entry_tid=20294' 2024-09-11T06:24:50.154 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=94, tag_tid=17, entry_tid=20294' 2024-09-11T06:24:50.154 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=94, tag_tid=17, entry_tid=20294' '!=' 'object_number=94, tag_tid=17, entry_tid=20294' 2024-09-11T06:24:50.154 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:24:50.154 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.8 2024-09-11T06:24:50.955 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:24:50.956 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:50.956 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:24:50.956 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:24:50.956 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:24:50.956 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:24:50.956 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:24:50.956 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:24:50.956 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:24:50.956 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:50.956 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:50.956 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:24:50.957 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:24:50.957 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:24:50.957 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:24:50.957 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:24:50.957 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:24:50.960 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:24:50.961 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:24:50.961 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:24:50.961 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:24:50.961 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:24:50.961 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:24:50.961 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:24:50.961 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:24:50.961 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:24:50.961 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:24:50.961 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:24:50.961 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:50.962 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:24:50.962 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:50.962 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:24:50.962 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:50.962 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:24:50.962 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:24:50.962 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:24:50.962 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:24:50.962 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:24:50.962 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:24:50.962 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:24:50.962 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:24:50.962 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:24:50.962 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:24:50.963 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:24:50.963 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:24:50.963 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:24:50.963 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:24:50.963 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:24:50.963 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:24:50.987 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:50.987 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:50.987 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:24:50.988 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:24:51.115 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:24:51.117 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:24:51.117 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:51.117 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:51.117 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:51.117 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:24:51.117 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:51.117 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:51.117 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:51.117 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:24:51.117 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:51.118 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:51.118 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:51.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:51.120 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:51.120 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:51.168 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 23 2024-09-11T06:24:51.168 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:51.168 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:51.168 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:51.168 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:51.172 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:51.173 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:51.174 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:24:51.174 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:51.174 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:51.175 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:51.175 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:24:51.175 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:24:51.175 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:24:51.175 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:24:51.175 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:24:51.175 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:24:51.175 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:24:51.175 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:24:51.176 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:51.176 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:24:51.176 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:51.222 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 23 2024-09-11T06:24:51.222 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 23 2024-09-11T06:24:51.222 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:24:51.222 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:51.222 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=93, tag_tid=18, entry_tid=1], [object_number=92, tag_tid=18, entry_tid=0], [object_number=94, tag_tid=17, entry_tid=20294], [object_number=95, tag_tid=17, entry_tid=20291]]], state=connected] 2024-09-11T06:24:51.225 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:24:51.227 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:51.227 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=93, tag_tid=18, entry_tid=1' -a 'object_number=93, tag_tid=18, entry_tid=1' = 'object_number=93, tag_tid=18, entry_tid=1' 2024-09-11T06:24:51.227 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:24:51.227 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_snap_present cluster1 mirror test snap6 2024-09-11T06:24:51.227 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:24:51.228 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:24:51.228 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:24:51.228 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap6 2024-09-11T06:24:51.228 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:24:51.228 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T06:24:51.228 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:24:52.228 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 info mirror/test@snap6 2024-09-11T06:24:52.298 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:rbd image 'test': 2024-09-11T06:24:52.298 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: size 512 MiB in 128 objects 2024-09-11T06:24:52.298 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: order 22 (4 MiB objects) 2024-09-11T06:24:52.298 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: snapshot_count: 11 2024-09-11T06:24:52.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: id: 10a618cfe337 2024-09-11T06:24:52.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: block_name_prefix: rbd_data.10a618cfe337 2024-09-11T06:24:52.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: format: 2 2024-09-11T06:24:52.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: features: layering, exclusive-lock, journaling 2024-09-11T06:24:52.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: op_features: 2024-09-11T06:24:52.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: flags: 2024-09-11T06:24:52.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: create_timestamp: Wed Sep 11 06:16:46 2024 2024-09-11T06:24:52.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: access_timestamp: Wed Sep 11 06:22:00 2024 2024-09-11T06:24:52.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: modify_timestamp: Wed Sep 11 06:23:55 2024 2024-09-11T06:24:52.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: protected: False 2024-09-11T06:24:52.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: journal: 10a618cfe337 2024-09-11T06:24:52.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring state: enabled 2024-09-11T06:24:52.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring mode: journal 2024-09-11T06:24:52.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring global id: a0f7d515-874e-4c67-a26b-4958c7ab3083 2024-09-11T06:24:52.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring primary: false 2024-09-11T06:24:52.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:24:52.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n snap5-clean ']' 2024-09-11T06:24:52.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ compare_image_snaps mirror test snap5-clean 2024-09-11T06:24:52.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:24:52.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:24:52.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap5-clean 2024-09-11T06:24:52.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local ret=0 2024-09-11T06:24:52.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local rmt_export=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export 2024-09-11T06:24:52.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loc_export=/tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:24:52.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:24:52.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5-clean /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export 2024-09-11T06:24:52.987 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T06:24:52.997 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5-clean /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:24:54.594 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T06:24:54.614 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmp /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:24:54.891 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:24:55.026 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:24:55.026 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ compare_image_snaps mirror test snap6 2024-09-11T06:24:55.026 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:24:55.026 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:24:55.026 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap6 2024-09-11T06:24:55.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local ret=0 2024-09-11T06:24:55.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local rmt_export=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export 2024-09-11T06:24:55.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loc_export=/tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:24:55.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:24:55.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6 /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export 2024-09-11T06:24:55.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T06:24:55.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6 /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:24:57.314 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T06:24:57.335 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmp /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:24:57.616 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:24:57.754 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:24:57.754 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ clean_snap_name=snap6-clean 2024-09-11T06:24:57.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_snap cluster2 mirror test snap6-clean 2024-09-11T06:24:57.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:24:57.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:24:57.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:24:57.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap6-clean 2024-09-11T06:24:57.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap6-clean --debug-rbd=20 --debug-journaler=20 2024-09-11T06:24:58.278 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 10` 2024-09-11T06:24:58.279 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ stress_write_image cluster2 mirror test 2024-09-11T06:24:58.279 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:24:58.279 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:24:58.279 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:24:58.279 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T06:24:58.281 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local duration=12 2024-09-11T06:24:58.281 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set +e 2024-09-11T06:24:58.281 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-11T06:25:00.112 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:100 IOs, 33755136 bytes 2024-09-11T06:25:00.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:200 IOs, 60808192 bytes 2024-09-11T06:25:01.796 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:300 IOs, 69607424 bytes 2024-09-11T06:25:04.201 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:400 IOs, 113320960 bytes 2024-09-11T06:25:04.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:500 IOs, 133209088 bytes 2024-09-11T06:25:05.255 INFO:tasks.ceph.cluster2.osd.0.smithi112.stderr:2024-09-11T06:25:05.258+0000 7f3fa0017640 -1 reset not still connected to 0x563d1fa4c4e0 2024-09-11T06:25:05.256 INFO:tasks.ceph.cluster2.osd.0.smithi112.stderr:2024-09-11T06:25:05.258+0000 7f3fa0017640 -1 reset not still connected to 0x563d206f7ba0 2024-09-11T06:25:08.515 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:600 IOs, 251703296 bytes 2024-09-11T06:25:10.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ error_code=124 2024-09-11T06:25:10.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set -e 2024-09-11T06:25:10.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T06:25:10.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:25:10.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-11T06:25:10.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:25:10.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:25:10.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:25:10.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state_pattern=up+replaying 2024-09-11T06:25:10.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local description_pattern= 2024-09-11T06:25:10.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local service_pattern= 2024-09-11T06:25:10.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:25:10.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:25:11.293 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-11T06:25:11.293 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:25:11.293 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:25:11.293 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:25:11.294 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state_pattern=up+replaying 2024-09-11T06:25:11.294 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local description_pattern= 2024-09-11T06:25:11.294 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local service_pattern= 2024-09-11T06:25:11.294 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local status 2024-09-11T06:25:11.294 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ CEPH_ARGS= 2024-09-11T06:25:11.294 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-11T06:25:11.384 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ status='test: 2024-09-11T06:25:11.384 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: global_id: a0f7d515-874e-4c67-a26b-4958c7ab3083 2024-09-11T06:25:11.384 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: state: up+replaying 2024-09-11T06:25:11.384 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: description: replaying, {"bytes_per_second":2952123.2,"entries_behind_primary":745,"entries_per_second":186.47,"non_primary_position":{"entry_tid":19550,"object_number":94,"tag_tid":17},"primary_position":{"entry_tid":1,"object_number":93,"tag_tid":18},"seconds_until_synced":4} 2024-09-11T06:25:11.384 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: service: mirror.0 on smithi112 2024-09-11T06:25:11.384 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: last_update: 2024-09-11 06:24:44 2024-09-11T06:25:11.385 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: peer_sites: 2024-09-11T06:25:11.385 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: name: cluster2 2024-09-11T06:25:11.385 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: state: up+stopped 2024-09-11T06:25:11.385 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: description: local image is primary 2024-09-11T06:25:11.385 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: last_update: 2024-09-11 06:25:09' 2024-09-11T06:25:11.385 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ grep '^ state: .*up+replaying' 2024-09-11T06:25:11.385 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: state: up+replaying 2024-09-11T06:25:11.386 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ grep '^ description: .*' 2024-09-11T06:25:11.386 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: description: replaying, {"bytes_per_second":2952123.2,"entries_behind_primary":745,"entries_per_second":186.47,"non_primary_position":{"entry_tid":19550,"object_number":94,"tag_tid":17},"primary_position":{"entry_tid":1,"object_number":93,"tag_tid":18},"seconds_until_synced":4} 2024-09-11T06:25:11.386 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n '' ']' 2024-09-11T06:25:11.386 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ echo up+replaying 2024-09-11T06:25:11.387 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ grep '^up+' 2024-09-11T06:25:11.387 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:up+replaying 2024-09-11T06:25:11.387 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ grep 'service: *mirror..* on ' 2024-09-11T06:25:11.388 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: service: mirror.0 on smithi112 2024-09-11T06:25:11.388 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local last_update 2024-09-11T06:25:11.389 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-11T06:25:11.391 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_update='2024-09-11 06:24:44' 2024-09-11T06:25:11.391 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-11 06:24:44' 2024-09-11T06:25:11.391 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:25:11.392 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:25:11.392 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:25:11.392 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state_pattern=up+replaying 2024-09-11T06:25:11.392 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'prev_last_update=2024-09-11 06:24:44' 2024-09-11T06:25:11.392 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local status 2024-09-11T06:25:11.392 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ CEPH_ARGS= 2024-09-11T06:25:11.392 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-11T06:25:11.478 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ status='OKOKOK142554262mirror.0smithi11219.1.1-298-gb53b6d3etrueOK42594265mirror.1smithi11219.1.1-298-gb53b6d3efalseOKtesta0f7d515-874e-4c67-a26b-4958c7ab3083up+replayingreplaying, {"bytes_per_second":2952123.2,"entries_behind_primary":745,"entries_per_second":186.47,"non_primary_position":{"entry_tid":19550,"object_number":94,"tag_tid":17},"primary_position":{"entry_tid":1,"object_number":93,"tag_tid":18},"seconds_until_synced":4}42554262mirror.0smithi1122024-09-11 06:24:44cluster20bfc6eda-f3c5-424a-8ff7-5c4d3d9fd488up+stoppedlocal image is primary2024-09-11 06:25:09' 2024-09-11T06:25:11.478 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local last_update state 2024-09-11T06:25:11.478 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-11T06:25:11.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_update='2024-09-11 06:24:44' 2024-09-11T06:25:11.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-11T06:25:11.484 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ state=up+replaying 2024-09-11T06:25:11.484 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ echo up+replaying 2024-09-11T06:25:11.485 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ grep up+replaying 2024-09-11T06:25:11.485 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:up+replaying 2024-09-11T06:25:11.486 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:25:11.486 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:25:11.486 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ snap_name=snap7 2024-09-11T06:25:11.486 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_snap cluster2 mirror test snap7 2024-09-11T06:25:11.486 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:25:11.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:25:11.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:25:11.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap7 2024-09-11T06:25:11.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap7 --debug-rbd=20 --debug-journaler=20 2024-09-11T06:25:40.335 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-11T06:25:40.335 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:25:40.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:25:40.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:25:40.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-11T06:25:40.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:25:40.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:25:40.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:25:40.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=started 2024-09-11T06:25:40.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:25:40.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:25:40.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:25:41.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-11T06:25:41.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:25:41.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:25:41.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:25:41.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=started 2024-09-11T06:25:41.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local status_result 2024-09-11T06:25:41.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=stopped 2024-09-11T06:25:41.337 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-11T06:25:41.337 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_instance=cluster1 2024-09-11T06:25:41.337 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:25:41.337 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -i state 2024-09-11T06:25:41.337 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:25:41.338 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance=cluster1 2024-09-11T06:25:41.338 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local loop_instance 2024-09-11T06:25:41.338 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:25:41.338 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sleep 0 2024-09-11T06:25:41.339 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:25:41.339 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ seq 0 1 2024-09-11T06:25:41.343 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:25:41.343 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-11T06:25:41.343 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1:0 2024-09-11T06:25:41.343 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:25:41.343 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:25:41.343 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:25:41.343 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:25:41.343 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=cluster 2024-09-11T06:25:41.343 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:25:41.343 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:25:41.343 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:25:41.344 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:25:41.344 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval cluster=cluster1 2024-09-11T06:25:41.344 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:25:41.344 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:25:41.344 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:25:41.344 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:25:41.344 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local local_cluster=cluster1:0 2024-09-11T06:25:41.344 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster=cluster1 2024-09-11T06:25:41.344 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance 2024-09-11T06:25:41.344 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:25:41.344 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local val=cluster1:0 2024-09-11T06:25:41.344 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T06:25:41.345 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance_var_name=instance 2024-09-11T06:25:41.345 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:25:41.345 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:25:41.345 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T06:25:41.345 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval local_cluster=cluster1 2024-09-11T06:25:41.345 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ local_cluster=cluster1 2024-09-11T06:25:41.345 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval instance=0 2024-09-11T06:25:41.345 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ instance=0 2024-09-11T06:25:41.345 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:25:41.363 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:25:41.363 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:25:41.363 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:25:41.363 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-11T06:25:41.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ return 0 2024-09-11T06:25:41.483 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T06:25:41.483 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ echo ' "state": "Replaying"' 2024-09-11T06:25:41.483 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ grep -i Replaying 2024-09-11T06:25:41.484 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: "state": "Replaying" 2024-09-11T06:25:41.486 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_state=started 2024-09-11T06:25:41.486 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test started = started 2024-09-11T06:25:41.486 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:25:41.486 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-11T06:25:41.486 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:25:41.486 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:25:41.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:25:41.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:25:41.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' journal = journal ']' 2024-09-11T06:25:41.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-09-11T06:25:41.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:25:41.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:25:41.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:25:41.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:25:41.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-11T06:25:41.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-11T06:25:41.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:25:41.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:25:41.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:25:41.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:25:41.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:25:41.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:25:41.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:25:41.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:25:41.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:25:41.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:25:41.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:25:41.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:25:41.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:25:41.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:25:41.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:25:41.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:25:41.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:25:41.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:25:41.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:25:41.689 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:25:41.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:25:41.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:25:41.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:25:41.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:25:41.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:25:41.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:25:41.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:25:41.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:25:41.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:25:41.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:25:41.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:25:41.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:25:41.693 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:25:41.693 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:25:41.693 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:25:41.693 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:25:41.693 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:25:41.693 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:25:41.693 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:25:41.693 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:25:41.693 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:25:41.693 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:25:41.693 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:25:41.693 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:25:41.693 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:25:41.693 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:25:41.694 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:25:41.694 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:25:41.694 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:25:41.694 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:25:41.694 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:25:41.694 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:25:41.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:25:41.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:25:41.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:25:41.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:25:41.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:25:41.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:25:41.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:41.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:41.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:41.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:25:41.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:41.854 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:41.854 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:41.854 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:25:41.854 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:25:41.854 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:25:41.854 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:25:41.855 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:41.855 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:25:41.855 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:41.901 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 25 2024-09-11T06:25:41.902 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:25:41.902 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:25:41.902 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:25:41.902 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=102, tag_tid=20, entry_tid=6690], [object_number=101, tag_tid=20, entry_tid=6689], [object_number=100, tag_tid=20, entry_tid=6688], [object_number=103, tag_tid=20, entry_tid=6687]]], state=connected] 2024-09-11T06:25:41.905 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:41.908 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:25:41.908 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:25:41.908 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:41.908 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:41.908 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:41.908 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:25:41.908 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:41.908 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:41.908 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:41.908 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:25:41.908 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:25:41.909 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:25:41.909 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:25:41.909 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:41.909 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:25:41.909 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:41.956 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 25 2024-09-11T06:25:41.957 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:25:41.957 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:25:41.957 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:25:41.957 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=102, tag_tid=20, entry_tid=6690], [object_number=101, tag_tid=20, entry_tid=6689], [object_number=100, tag_tid=20, entry_tid=6688], [object_number=103, tag_tid=20, entry_tid=6687]]], state=connected] 2024-09-11T06:25:41.960 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:41.962 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=102, tag_tid=20, entry_tid=6690' 2024-09-11T06:25:41.962 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=102, tag_tid=20, entry_tid=6690' 2024-09-11T06:25:41.962 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=102, tag_tid=20, entry_tid=6690' '!=' '' 2024-09-11T06:25:41.962 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:25:41.962 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=102, tag_tid=20, entry_tid=6690' = '' 2024-09-11T06:25:41.962 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=102, tag_tid=20, entry_tid=6690' 2024-09-11T06:25:41.962 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:25:41.962 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:25:41.962 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:41.963 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:25:41.964 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=102, tag_tid=20, entry_tid=6690' 2024-09-11T06:25:41.964 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:25:41.964 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:41.965 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:25:41.965 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:25:41.966 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:25:41.966 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:41.966 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:25:41.967 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=102, tag_tid=20, entry_tid=6690' 2024-09-11T06:25:41.967 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:25:41.967 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:41.968 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=6690 2024-09-11T06:25:41.969 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 6690 2024-09-11T06:25:41.969 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:25:41.969 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:25:41.969 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:25:42.170 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:25:42.170 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:25:42.171 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:25:42.171 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:25:42.171 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:25:42.171 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:25:42.171 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:25:42.171 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:25:42.171 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:25:42.171 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:25:42.171 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:25:42.171 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:25:42.171 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:25:42.171 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:25:42.171 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:25:42.172 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:25:42.172 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:25:42.175 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:25:42.176 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:25:42.176 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:25:42.176 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:25:42.176 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:25:42.176 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:25:42.176 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:25:42.176 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:25:42.176 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:25:42.176 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:25:42.176 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:25:42.176 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:25:42.176 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:25:42.176 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:25:42.177 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:25:42.177 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:25:42.177 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:25:42.177 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:25:42.177 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:25:42.177 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:25:42.177 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:25:42.177 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:25:42.177 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:25:42.177 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:25:42.177 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:25:42.177 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:25:42.177 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:25:42.177 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:25:42.177 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:25:42.178 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:25:42.178 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:25:42.178 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:25:42.201 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:25:42.201 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:25:42.201 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:25:42.201 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:25:42.330 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:25:42.331 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:25:42.332 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:42.332 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:42.332 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:42.332 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:25:42.332 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:42.332 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:42.332 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:42.332 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:25:42.333 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:25:42.333 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:25:42.333 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:25:42.334 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:42.334 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:25:42.334 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:42.382 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 25 2024-09-11T06:25:42.382 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:25:42.382 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:25:42.382 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:25:42.382 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=103, tag_tid=20, entry_tid=6783], [object_number=102, tag_tid=20, entry_tid=6782], [object_number=101, tag_tid=20, entry_tid=6781], [object_number=100, tag_tid=20, entry_tid=6780]]], state=connected] 2024-09-11T06:25:42.430 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:42.431 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:25:42.432 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:25:42.432 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:42.432 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:42.432 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:42.432 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:25:42.433 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:42.433 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:42.433 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:42.433 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:25:42.433 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:25:42.433 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:25:42.434 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:25:42.435 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:42.436 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:42.436 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:25:42.491 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 25 2024-09-11T06:25:42.491 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:25:42.491 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:25:42.492 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:25:42.492 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=103, tag_tid=20, entry_tid=6783], [object_number=102, tag_tid=20, entry_tid=6782], [object_number=101, tag_tid=20, entry_tid=6781], [object_number=100, tag_tid=20, entry_tid=6780]]], state=connected] 2024-09-11T06:25:42.526 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:42.528 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=103, tag_tid=20, entry_tid=6783' 2024-09-11T06:25:42.528 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=103, tag_tid=20, entry_tid=6783' 2024-09-11T06:25:42.528 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=103, tag_tid=20, entry_tid=6783' '!=' 'object_number=102, tag_tid=20, entry_tid=6690' 2024-09-11T06:25:42.528 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:25:42.528 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=103, tag_tid=20, entry_tid=6783' = 'object_number=102, tag_tid=20, entry_tid=6690' 2024-09-11T06:25:42.528 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=103, tag_tid=20, entry_tid=6783' 2024-09-11T06:25:42.529 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:25:42.529 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:25:42.529 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:42.530 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:25:42.530 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=103, tag_tid=20, entry_tid=6783' 2024-09-11T06:25:42.530 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:25:42.531 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:42.531 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:25:42.532 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:25:42.532 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:25:42.532 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:42.533 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:25:42.534 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=103, tag_tid=20, entry_tid=6783' 2024-09-11T06:25:42.534 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:25:42.534 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:42.536 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=6783 2024-09-11T06:25:42.536 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 6783 2024-09-11T06:25:42.536 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:25:42.536 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:25:42.536 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:25:42.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:25:42.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:25:42.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:25:42.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:25:42.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:25:42.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:25:42.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:25:42.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:25:42.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:25:42.739 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:25:42.739 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:25:42.739 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:25:42.739 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:25:42.739 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:25:42.739 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:25:42.739 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:25:42.739 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:25:42.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:25:42.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:25:42.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:25:42.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:25:42.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:25:42.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:25:42.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:25:42.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:25:42.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:25:42.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:25:42.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:25:42.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:25:42.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:25:42.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:25:42.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:25:42.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:25:42.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:25:42.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:25:42.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:25:42.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:25:42.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:25:42.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:25:42.745 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:25:42.745 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:25:42.745 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:25:42.745 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:25:42.745 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:25:42.745 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:25:42.745 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:25:42.745 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:25:42.745 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:25:42.745 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:25:42.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:25:42.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:25:42.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:25:42.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:25:42.901 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:25:42.902 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:25:42.902 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:42.902 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:42.902 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:42.902 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:25:42.902 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:42.902 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:42.902 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:42.902 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:25:42.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:25:42.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:25:42.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:25:42.906 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:42.906 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:25:42.906 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:42.963 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 25 2024-09-11T06:25:42.964 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:25:42.964 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:25:42.964 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:25:42.964 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=103, tag_tid=20, entry_tid=6883], [object_number=102, tag_tid=20, entry_tid=6882], [object_number=101, tag_tid=20, entry_tid=6881], [object_number=100, tag_tid=20, entry_tid=6880]]], state=connected] 2024-09-11T06:25:42.968 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:42.969 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:25:42.970 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:25:42.970 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:42.970 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:42.970 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:42.970 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:25:42.970 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:42.970 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:42.970 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:42.970 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:25:42.971 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:25:42.971 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:25:42.971 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:25:42.971 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:42.971 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:25:42.971 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:43.019 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 25 2024-09-11T06:25:43.020 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:25:43.020 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:25:43.020 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:25:43.020 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=103, tag_tid=20, entry_tid=6883], [object_number=102, tag_tid=20, entry_tid=6882], [object_number=101, tag_tid=20, entry_tid=6881], [object_number=100, tag_tid=20, entry_tid=6880]]], state=connected] 2024-09-11T06:25:43.023 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:43.025 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=103, tag_tid=20, entry_tid=6883' 2024-09-11T06:25:43.025 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=103, tag_tid=20, entry_tid=6883' 2024-09-11T06:25:43.025 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=103, tag_tid=20, entry_tid=6883' '!=' 'object_number=103, tag_tid=20, entry_tid=6783' 2024-09-11T06:25:43.025 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:25:43.025 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=103, tag_tid=20, entry_tid=6883' = 'object_number=103, tag_tid=20, entry_tid=6783' 2024-09-11T06:25:43.025 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=103, tag_tid=20, entry_tid=6883' 2024-09-11T06:25:43.026 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:25:43.026 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:25:43.026 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:43.026 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:25:43.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=103, tag_tid=20, entry_tid=6883' 2024-09-11T06:25:43.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:25:43.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:43.028 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:25:43.029 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:25:43.029 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:25:43.029 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:43.030 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:25:43.031 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=103, tag_tid=20, entry_tid=6883' 2024-09-11T06:25:43.031 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:25:43.031 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:43.032 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=6883 2024-09-11T06:25:43.032 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 6883 2024-09-11T06:25:43.032 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:25:43.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:25:43.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:25:43.234 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:25:43.234 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:25:43.234 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:25:43.234 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:25:43.234 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:25:43.234 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:25:43.235 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:25:43.235 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:25:43.235 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:25:43.235 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:25:43.235 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:25:43.235 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:25:43.235 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:25:43.235 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:25:43.235 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:25:43.236 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:25:43.236 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:25:43.239 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:25:43.239 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:25:43.239 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:25:43.239 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:25:43.239 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:25:43.240 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:25:43.240 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:25:43.240 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:25:43.240 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:25:43.240 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:25:43.240 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:25:43.240 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:25:43.240 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:25:43.240 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:25:43.240 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:25:43.240 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:25:43.240 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:25:43.240 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:25:43.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:25:43.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:25:43.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:25:43.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:25:43.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:25:43.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:25:43.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:25:43.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:25:43.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:25:43.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:25:43.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:25:43.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:25:43.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:25:43.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:25:43.265 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:25:43.265 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:25:43.265 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:25:43.265 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:25:43.404 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:25:43.405 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:25:43.405 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:43.405 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:43.405 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:43.405 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:25:43.405 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:43.405 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:43.405 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:43.405 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:25:43.406 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:25:43.406 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:25:43.406 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:25:43.407 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:43.408 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:25:43.408 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:43.466 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 25 2024-09-11T06:25:43.466 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:25:43.466 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:25:43.466 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:25:43.466 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=101, tag_tid=20, entry_tid=6965], [object_number=100, tag_tid=20, entry_tid=6964], [object_number=103, tag_tid=20, entry_tid=6963], [object_number=102, tag_tid=20, entry_tid=6962]]], state=connected] 2024-09-11T06:25:43.469 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:43.470 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:25:43.472 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:25:43.472 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:43.472 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:43.472 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:43.472 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:25:43.472 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:43.472 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:43.472 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:43.472 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:25:43.472 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:25:43.472 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:25:43.473 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:25:43.473 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:43.473 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:25:43.473 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:43.520 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 25 2024-09-11T06:25:43.520 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:25:43.521 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:25:43.521 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:25:43.521 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=101, tag_tid=20, entry_tid=6965], [object_number=100, tag_tid=20, entry_tid=6964], [object_number=103, tag_tid=20, entry_tid=6963], [object_number=102, tag_tid=20, entry_tid=6962]]], state=connected] 2024-09-11T06:25:43.524 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:43.525 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=101, tag_tid=20, entry_tid=6965' 2024-09-11T06:25:43.525 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=101, tag_tid=20, entry_tid=6965' 2024-09-11T06:25:43.526 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=101, tag_tid=20, entry_tid=6965' '!=' 'object_number=103, tag_tid=20, entry_tid=6883' 2024-09-11T06:25:43.526 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:25:43.526 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=101, tag_tid=20, entry_tid=6965' = 'object_number=103, tag_tid=20, entry_tid=6883' 2024-09-11T06:25:43.526 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=101, tag_tid=20, entry_tid=6965' 2024-09-11T06:25:43.526 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:25:43.526 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:25:43.527 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:43.527 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:25:43.528 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=101, tag_tid=20, entry_tid=6965' 2024-09-11T06:25:43.528 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:25:43.528 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:43.529 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:25:43.530 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:25:43.530 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:25:43.530 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:43.530 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:25:43.531 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=101, tag_tid=20, entry_tid=6965' 2024-09-11T06:25:43.531 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:25:43.531 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:43.533 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=6965 2024-09-11T06:25:43.533 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 6965 2024-09-11T06:25:43.533 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:25:43.533 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:25:43.533 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:25:43.734 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:25:43.735 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:25:43.735 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:25:43.735 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:25:43.735 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:25:43.735 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:25:43.735 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:25:43.735 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:25:43.735 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:25:43.735 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:25:43.735 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:25:43.735 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:25:43.735 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:25:43.736 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:25:43.736 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:25:43.736 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:25:43.736 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:25:43.739 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:25:43.739 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:25:43.739 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:25:43.739 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:25:43.739 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:25:43.739 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:25:43.739 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:25:43.739 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:25:43.739 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:25:43.739 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:25:43.740 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:25:43.740 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:25:43.740 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:25:43.740 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:25:43.740 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:25:43.740 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:25:43.740 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:25:43.740 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:25:43.740 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:25:43.740 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:25:43.740 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:25:43.740 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:25:43.740 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:25:43.741 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:25:43.741 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:25:43.741 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:25:43.741 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:25:43.741 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:25:43.741 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:25:43.741 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:25:43.741 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:25:43.741 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:25:43.765 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:25:43.765 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:25:43.765 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:25:43.765 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:25:43.900 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:25:43.901 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:25:43.901 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:43.901 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:43.901 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:43.901 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:25:43.901 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:43.901 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:43.901 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:43.901 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:25:43.902 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:25:43.902 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:25:43.902 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:25:43.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:43.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:25:43.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:43.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 25 2024-09-11T06:25:43.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:25:43.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:25:43.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:25:43.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=101, tag_tid=20, entry_tid=7049], [object_number=100, tag_tid=20, entry_tid=7048], [object_number=103, tag_tid=20, entry_tid=7047], [object_number=102, tag_tid=20, entry_tid=7046]]], state=connected] 2024-09-11T06:25:43.954 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:43.955 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:25:43.956 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:25:43.956 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:43.957 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:43.957 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:43.957 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:25:43.957 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:43.957 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:43.957 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:43.957 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:25:43.957 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:25:43.957 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:25:43.957 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:25:43.958 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:43.958 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:25:43.958 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:44.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 25 2024-09-11T06:25:44.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:25:44.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:25:44.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:25:44.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=101, tag_tid=20, entry_tid=7049], [object_number=100, tag_tid=20, entry_tid=7048], [object_number=103, tag_tid=20, entry_tid=7047], [object_number=102, tag_tid=20, entry_tid=7046]]], state=connected] 2024-09-11T06:25:44.009 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:44.011 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=101, tag_tid=20, entry_tid=7049' 2024-09-11T06:25:44.011 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=101, tag_tid=20, entry_tid=7049' 2024-09-11T06:25:44.012 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=101, tag_tid=20, entry_tid=7049' '!=' 'object_number=101, tag_tid=20, entry_tid=6965' 2024-09-11T06:25:44.012 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:25:44.012 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=101, tag_tid=20, entry_tid=7049' = 'object_number=101, tag_tid=20, entry_tid=6965' 2024-09-11T06:25:44.012 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=101, tag_tid=20, entry_tid=7049' 2024-09-11T06:25:44.012 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:25:44.012 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:25:44.012 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:44.013 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:25:44.013 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=101, tag_tid=20, entry_tid=7049' 2024-09-11T06:25:44.013 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:25:44.013 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:44.014 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:25:44.015 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:25:44.015 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:25:44.015 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:44.016 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:25:44.017 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=101, tag_tid=20, entry_tid=7049' 2024-09-11T06:25:44.017 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:25:44.017 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:44.018 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=7049 2024-09-11T06:25:44.018 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 7049 2024-09-11T06:25:44.018 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:25:44.018 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:25:44.018 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:25:44.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:25:44.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:25:44.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:25:44.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:25:44.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:25:44.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:25:44.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:25:44.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:25:44.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:25:44.221 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:25:44.221 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:25:44.221 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:25:44.221 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:25:44.221 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:25:44.221 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:25:44.221 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:25:44.222 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:25:44.225 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:25:44.225 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:25:44.225 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:25:44.225 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:25:44.225 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:25:44.225 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:25:44.225 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:25:44.225 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:25:44.226 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:25:44.226 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:25:44.226 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:25:44.226 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:25:44.226 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:25:44.226 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:25:44.226 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:25:44.226 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:25:44.226 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:25:44.226 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:25:44.226 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:25:44.226 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:25:44.226 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:25:44.227 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:25:44.227 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:25:44.227 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:25:44.227 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:25:44.227 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:25:44.227 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:25:44.227 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:25:44.227 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:25:44.227 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:25:44.227 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:25:44.227 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:25:44.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:25:44.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:25:44.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:25:44.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:25:44.386 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:25:44.388 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:25:44.388 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:44.388 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:44.388 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:44.388 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:25:44.388 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:44.388 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:44.388 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:44.388 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:25:44.389 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:25:44.389 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:25:44.389 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:25:44.390 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:44.391 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:25:44.391 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:44.444 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 25 2024-09-11T06:25:44.444 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:25:44.444 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:25:44.444 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:25:44.444 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=102, tag_tid=20, entry_tid=7142], [object_number=101, tag_tid=20, entry_tid=7141], [object_number=100, tag_tid=20, entry_tid=7140], [object_number=103, tag_tid=20, entry_tid=7139]]], state=connected] 2024-09-11T06:25:44.447 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:44.449 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:25:44.450 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:25:44.450 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:44.450 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:44.450 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:44.450 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:25:44.450 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:44.450 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:44.451 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:44.451 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:25:44.451 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:25:44.451 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:25:44.451 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:25:44.452 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:44.452 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:25:44.452 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:44.499 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 25 2024-09-11T06:25:44.499 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:25:44.499 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:25:44.499 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:25:44.499 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=102, tag_tid=20, entry_tid=7142], [object_number=101, tag_tid=20, entry_tid=7141], [object_number=100, tag_tid=20, entry_tid=7140], [object_number=103, tag_tid=20, entry_tid=7139]]], state=connected] 2024-09-11T06:25:44.503 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:44.504 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=102, tag_tid=20, entry_tid=7142' 2024-09-11T06:25:44.504 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=102, tag_tid=20, entry_tid=7142' 2024-09-11T06:25:44.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=102, tag_tid=20, entry_tid=7142' '!=' 'object_number=101, tag_tid=20, entry_tid=7049' 2024-09-11T06:25:44.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:25:44.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=102, tag_tid=20, entry_tid=7142' = 'object_number=101, tag_tid=20, entry_tid=7049' 2024-09-11T06:25:44.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=102, tag_tid=20, entry_tid=7142' 2024-09-11T06:25:44.507 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:25:44.507 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:25:44.507 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:44.507 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:25:44.507 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=102, tag_tid=20, entry_tid=7142' 2024-09-11T06:25:44.507 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:25:44.507 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:44.508 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:25:44.509 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:25:44.509 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:25:44.509 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:44.509 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:25:44.510 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=102, tag_tid=20, entry_tid=7142' 2024-09-11T06:25:44.510 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:25:44.510 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:44.511 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=7142 2024-09-11T06:25:44.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 7142 2024-09-11T06:25:44.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:25:44.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:25:44.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:25:44.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:25:44.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:25:44.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:25:44.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:25:44.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:25:44.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:25:44.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:25:44.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:25:44.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:25:44.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:25:44.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:25:44.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:25:44.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:25:44.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:25:44.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:25:44.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:25:44.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:25:44.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:25:44.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:25:44.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:25:44.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:25:44.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:25:44.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:25:44.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:25:44.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:25:44.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:25:44.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:25:44.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:25:44.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:25:44.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:25:44.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:25:44.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:25:44.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:25:44.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:25:44.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:25:44.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:25:44.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:25:44.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:25:44.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:25:44.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:25:44.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:25:44.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:25:44.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:25:44.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:25:44.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:25:44.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:25:44.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:25:44.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:25:44.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:25:44.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:25:44.745 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:25:44.745 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:25:44.745 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:25:44.879 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:25:44.880 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:25:44.880 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:44.880 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:44.880 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:44.880 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:25:44.880 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:44.880 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:44.880 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:44.881 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:25:44.881 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:25:44.881 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:25:44.881 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:25:44.882 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:44.882 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:25:44.882 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:44.935 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 25 2024-09-11T06:25:44.935 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:25:44.935 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:25:44.935 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:25:44.936 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=101, tag_tid=20, entry_tid=7221], [object_number=100, tag_tid=20, entry_tid=7220], [object_number=103, tag_tid=20, entry_tid=7219], [object_number=102, tag_tid=20, entry_tid=7218]]], state=connected] 2024-09-11T06:25:44.938 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:44.940 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:25:44.941 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:25:44.941 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:44.941 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:44.941 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:44.941 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:25:44.941 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:44.941 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:44.941 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:44.942 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:25:44.942 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:25:44.942 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:25:44.942 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:25:44.942 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:44.942 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:25:44.942 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:44.990 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 25 2024-09-11T06:25:44.990 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:25:44.990 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:25:44.990 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:25:44.990 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=101, tag_tid=20, entry_tid=7221], [object_number=100, tag_tid=20, entry_tid=7220], [object_number=103, tag_tid=20, entry_tid=7219], [object_number=102, tag_tid=20, entry_tid=7218]]], state=connected] 2024-09-11T06:25:44.994 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:44.996 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=101, tag_tid=20, entry_tid=7221' 2024-09-11T06:25:44.996 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=101, tag_tid=20, entry_tid=7221' 2024-09-11T06:25:44.996 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=101, tag_tid=20, entry_tid=7221' '!=' 'object_number=102, tag_tid=20, entry_tid=7142' 2024-09-11T06:25:44.997 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:25:44.997 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=101, tag_tid=20, entry_tid=7221' = 'object_number=102, tag_tid=20, entry_tid=7142' 2024-09-11T06:25:44.997 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=101, tag_tid=20, entry_tid=7221' 2024-09-11T06:25:44.997 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:25:44.997 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:25:44.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:44.999 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:25:45.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=101, tag_tid=20, entry_tid=7221' 2024-09-11T06:25:45.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:25:45.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:45.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:25:45.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:25:45.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:25:45.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:45.002 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:25:45.002 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=101, tag_tid=20, entry_tid=7221' 2024-09-11T06:25:45.002 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:25:45.003 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:45.003 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=7221 2024-09-11T06:25:45.003 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 7221 2024-09-11T06:25:45.003 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:25:45.003 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:25:45.003 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:25:45.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:25:45.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:25:45.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:25:45.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:25:45.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:25:45.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:25:45.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:25:45.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:25:45.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:25:45.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:25:45.205 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:25:45.205 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:25:45.205 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:25:45.205 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:25:45.205 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:25:45.205 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:25:45.206 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:25:45.210 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:25:45.210 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:25:45.210 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:25:45.210 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:25:45.210 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:25:45.210 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:25:45.210 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:25:45.210 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:25:45.211 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:25:45.211 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:25:45.211 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:25:45.211 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:25:45.211 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:25:45.211 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:25:45.211 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:25:45.211 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:25:45.211 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:25:45.211 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:25:45.211 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:25:45.211 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:25:45.211 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:25:45.211 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:25:45.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:25:45.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:25:45.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:25:45.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:25:45.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:25:45.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:25:45.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:25:45.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:25:45.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:25:45.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:25:45.235 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:25:45.235 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:25:45.235 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:25:45.236 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:25:45.363 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:25:45.364 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:25:45.364 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:45.364 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:45.364 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:45.364 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:25:45.365 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:45.365 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:45.365 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:45.365 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:25:45.365 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:25:45.365 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:25:45.365 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:25:45.367 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:45.367 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:25:45.367 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:45.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 25 2024-09-11T06:25:45.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:25:45.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:25:45.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:25:45.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=102, tag_tid=20, entry_tid=7302], [object_number=101, tag_tid=20, entry_tid=7301], [object_number=100, tag_tid=20, entry_tid=7300], [object_number=103, tag_tid=20, entry_tid=7299]]], state=connected] 2024-09-11T06:25:45.423 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:45.424 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:25:45.425 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:25:45.425 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:45.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:45.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:45.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:25:45.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:45.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:45.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:45.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:25:45.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:25:45.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:25:45.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:25:45.427 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:45.427 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:25:45.427 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:45.477 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 25 2024-09-11T06:25:45.477 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:25:45.477 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:25:45.478 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:25:45.478 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=102, tag_tid=20, entry_tid=7302], [object_number=101, tag_tid=20, entry_tid=7301], [object_number=100, tag_tid=20, entry_tid=7300], [object_number=103, tag_tid=20, entry_tid=7299]]], state=connected] 2024-09-11T06:25:45.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:45.483 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=102, tag_tid=20, entry_tid=7302' 2024-09-11T06:25:45.483 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=102, tag_tid=20, entry_tid=7302' 2024-09-11T06:25:45.483 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=102, tag_tid=20, entry_tid=7302' '!=' 'object_number=101, tag_tid=20, entry_tid=7221' 2024-09-11T06:25:45.483 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:25:45.483 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=102, tag_tid=20, entry_tid=7302' = 'object_number=101, tag_tid=20, entry_tid=7221' 2024-09-11T06:25:45.483 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=102, tag_tid=20, entry_tid=7302' 2024-09-11T06:25:45.484 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:25:45.484 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:25:45.484 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:45.484 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:25:45.485 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=102, tag_tid=20, entry_tid=7302' 2024-09-11T06:25:45.485 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:25:45.485 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:45.486 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:25:45.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:25:45.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:25:45.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:45.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:25:45.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=102, tag_tid=20, entry_tid=7302' 2024-09-11T06:25:45.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:25:45.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:45.490 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=7302 2024-09-11T06:25:45.490 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 7302 2024-09-11T06:25:45.490 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:25:45.490 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:25:45.490 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:25:45.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:25:45.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:25:45.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:25:45.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:25:45.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:25:45.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:25:45.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:25:45.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:25:45.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:25:45.693 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:25:45.693 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:25:45.693 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:25:45.693 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:25:45.693 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:25:45.693 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:25:45.694 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:25:45.694 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:25:45.696 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:25:45.696 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:25:45.696 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:25:45.697 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:25:45.697 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:25:45.697 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:25:45.697 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:25:45.697 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:25:45.697 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:25:45.697 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:25:45.697 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:25:45.697 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:25:45.697 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:25:45.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:25:45.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:25:45.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:25:45.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:25:45.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:25:45.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:25:45.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:25:45.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:25:45.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:25:45.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:25:45.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:25:45.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:25:45.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:25:45.700 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:25:45.700 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:25:45.700 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:25:45.700 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:25:45.700 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:25:45.700 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:25:45.723 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:25:45.724 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:25:45.724 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:25:45.724 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:25:45.855 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:25:45.857 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:25:45.857 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:45.857 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:45.857 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:45.857 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:25:45.857 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:45.857 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:45.857 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:45.857 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:25:45.858 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:25:45.858 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:25:45.858 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:25:45.859 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:45.860 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:25:45.860 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:46.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 25 2024-09-11T06:25:46.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:25:46.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:25:46.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:25:46.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=102, tag_tid=20, entry_tid=7382], [object_number=101, tag_tid=20, entry_tid=7381], [object_number=100, tag_tid=20, entry_tid=7380], [object_number=103, tag_tid=20, entry_tid=7379]]], state=connected] 2024-09-11T06:25:46.124 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:46.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:25:46.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:25:46.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:46.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:46.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:46.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:25:46.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:46.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:46.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:46.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:25:46.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:25:46.128 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:25:46.128 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:25:46.128 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:46.129 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:25:46.129 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:46.187 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 25 2024-09-11T06:25:46.187 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:25:46.187 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:25:46.187 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:25:46.187 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=102, tag_tid=20, entry_tid=7382], [object_number=101, tag_tid=20, entry_tid=7381], [object_number=100, tag_tid=20, entry_tid=7380], [object_number=103, tag_tid=20, entry_tid=7379]]], state=connected] 2024-09-11T06:25:46.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:46.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=102, tag_tid=20, entry_tid=7382' 2024-09-11T06:25:46.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=102, tag_tid=20, entry_tid=7382' 2024-09-11T06:25:46.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=102, tag_tid=20, entry_tid=7382' '!=' 'object_number=102, tag_tid=20, entry_tid=7302' 2024-09-11T06:25:46.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:25:46.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=102, tag_tid=20, entry_tid=7382' = 'object_number=102, tag_tid=20, entry_tid=7302' 2024-09-11T06:25:46.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=102, tag_tid=20, entry_tid=7382' 2024-09-11T06:25:46.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:25:46.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:25:46.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:46.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:25:46.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=102, tag_tid=20, entry_tid=7382' 2024-09-11T06:25:46.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:25:46.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:46.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:25:46.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:25:46.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:25:46.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:46.197 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:25:46.197 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=102, tag_tid=20, entry_tid=7382' 2024-09-11T06:25:46.197 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:25:46.197 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:46.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=7382 2024-09-11T06:25:46.199 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 7382 2024-09-11T06:25:46.199 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:25:46.199 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:25:46.199 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:25:46.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:25:46.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:25:46.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:25:46.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:25:46.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:25:46.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:25:46.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:25:46.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:25:46.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:25:46.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:25:46.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:25:46.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:25:46.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:25:46.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:25:46.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:25:46.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:25:46.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:25:46.406 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:25:46.406 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:25:46.406 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:25:46.406 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:25:46.406 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:25:46.406 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:25:46.406 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:25:46.407 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:25:46.407 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:25:46.407 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:25:46.407 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:25:46.407 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:25:46.407 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:25:46.407 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:25:46.407 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:25:46.407 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:25:46.407 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:25:46.407 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:25:46.407 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:25:46.407 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:25:46.408 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:25:46.408 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:25:46.408 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:25:46.408 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:25:46.408 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:25:46.408 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:25:46.408 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:25:46.408 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:25:46.408 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:25:46.408 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:25:46.408 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:25:46.408 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:25:46.433 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:25:46.433 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:25:46.433 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:25:46.434 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:25:46.603 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:25:46.604 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:25:46.604 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:46.604 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:46.604 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:46.604 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:25:46.604 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:46.605 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:46.605 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:46.605 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:25:46.605 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:25:46.605 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:25:46.605 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:25:46.607 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:46.607 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:25:46.607 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:46.661 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 25 2024-09-11T06:25:46.661 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:25:46.661 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:25:46.661 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:25:46.661 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=100, tag_tid=20, entry_tid=7476], [object_number=103, tag_tid=20, entry_tid=7475], [object_number=102, tag_tid=20, entry_tid=7474], [object_number=101, tag_tid=20, entry_tid=7473]]], state=connected] 2024-09-11T06:25:46.663 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:46.665 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:25:46.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:25:46.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:46.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:46.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:46.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:25:46.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:46.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:46.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:46.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:25:46.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:25:46.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:25:46.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:25:46.668 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:46.668 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:25:46.668 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:46.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 25 2024-09-11T06:25:46.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:25:46.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:25:46.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:25:46.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=100, tag_tid=20, entry_tid=7476], [object_number=103, tag_tid=20, entry_tid=7475], [object_number=102, tag_tid=20, entry_tid=7474], [object_number=101, tag_tid=20, entry_tid=7473]]], state=connected] 2024-09-11T06:25:46.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:46.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=100, tag_tid=20, entry_tid=7476' 2024-09-11T06:25:46.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=100, tag_tid=20, entry_tid=7476' 2024-09-11T06:25:46.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=100, tag_tid=20, entry_tid=7476' '!=' 'object_number=102, tag_tid=20, entry_tid=7382' 2024-09-11T06:25:46.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:25:46.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=100, tag_tid=20, entry_tid=7476' = 'object_number=102, tag_tid=20, entry_tid=7382' 2024-09-11T06:25:46.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=100, tag_tid=20, entry_tid=7476' 2024-09-11T06:25:46.721 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:25:46.721 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:25:46.721 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:46.721 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:25:46.722 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=100, tag_tid=20, entry_tid=7476' 2024-09-11T06:25:46.722 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:25:46.722 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:46.723 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:25:46.724 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:25:46.724 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:25:46.724 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:46.725 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:25:46.725 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=100, tag_tid=20, entry_tid=7476' 2024-09-11T06:25:46.725 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:25:46.725 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:46.727 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=7476 2024-09-11T06:25:46.727 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 7476 2024-09-11T06:25:46.727 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:25:46.727 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:25:46.727 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:25:46.929 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:25:46.929 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:25:46.929 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:25:46.929 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:25:46.929 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:25:46.930 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:25:46.930 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:25:46.930 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:25:46.930 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:25:46.930 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:25:46.930 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:25:46.930 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:25:46.930 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:25:46.930 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:25:46.930 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:25:46.931 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:25:46.931 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:25:46.933 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:25:46.933 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:25:46.933 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:25:46.933 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:25:46.933 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:25:46.933 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:25:46.933 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:25:46.933 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:25:46.933 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:25:46.933 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:25:46.933 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:25:46.933 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:25:46.933 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:25:46.933 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:25:46.934 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:25:46.934 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:25:46.935 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:25:46.935 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:25:46.935 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:25:46.935 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:25:46.935 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:25:46.935 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:25:46.935 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:25:46.936 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:25:46.936 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:25:46.936 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:25:46.936 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:25:46.936 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:25:46.936 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:25:46.936 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:25:46.936 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:25:46.936 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:25:46.960 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:25:46.960 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:25:46.960 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:25:46.960 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:25:47.090 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:25:47.091 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:25:47.091 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:47.091 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:47.091 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:47.091 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:25:47.092 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:47.092 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:47.092 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:47.092 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:25:47.092 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:25:47.093 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:25:47.093 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:25:47.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:47.095 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:25:47.095 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:47.145 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 25 2024-09-11T06:25:47.145 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:25:47.146 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:25:47.146 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:25:47.146 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=106, tag_tid=20, entry_tid=7506], [object_number=105, tag_tid=20, entry_tid=7505], [object_number=104, tag_tid=20, entry_tid=7504], [object_number=103, tag_tid=20, entry_tid=7503]]], state=connected] 2024-09-11T06:25:47.148 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:47.151 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:25:47.152 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:25:47.152 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:47.152 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:47.152 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:47.152 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:25:47.152 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:47.153 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:47.153 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:47.153 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:25:47.153 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:25:47.153 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:25:47.154 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:25:47.155 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:47.155 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:25:47.155 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:47.199 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 25 2024-09-11T06:25:47.199 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:25:47.199 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:25:47.199 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:25:47.199 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=106, tag_tid=20, entry_tid=7506], [object_number=105, tag_tid=20, entry_tid=7505], [object_number=104, tag_tid=20, entry_tid=7504], [object_number=103, tag_tid=20, entry_tid=7503]]], state=connected] 2024-09-11T06:25:47.202 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:47.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=106, tag_tid=20, entry_tid=7506' 2024-09-11T06:25:47.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=106, tag_tid=20, entry_tid=7506' 2024-09-11T06:25:47.205 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=106, tag_tid=20, entry_tid=7506' '!=' 'object_number=100, tag_tid=20, entry_tid=7476' 2024-09-11T06:25:47.205 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:25:47.205 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=106, tag_tid=20, entry_tid=7506' = 'object_number=100, tag_tid=20, entry_tid=7476' 2024-09-11T06:25:47.205 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=106, tag_tid=20, entry_tid=7506' 2024-09-11T06:25:47.205 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:25:47.205 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:25:47.205 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:47.205 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:25:47.206 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=106, tag_tid=20, entry_tid=7506' 2024-09-11T06:25:47.206 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:25:47.207 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:47.207 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:25:47.208 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:25:47.208 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:25:47.208 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:47.209 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:25:47.209 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=106, tag_tid=20, entry_tid=7506' 2024-09-11T06:25:47.210 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:25:47.210 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:47.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=7506 2024-09-11T06:25:47.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 7506 2024-09-11T06:25:47.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:25:47.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:25:47.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:25:47.413 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:25:47.413 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:25:47.413 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:25:47.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:25:47.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:25:47.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:25:47.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:25:47.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:25:47.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:25:47.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:25:47.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:25:47.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:25:47.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:25:47.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:25:47.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:25:47.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:25:47.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:25:47.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:25:47.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:25:47.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:25:47.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:25:47.418 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:25:47.418 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:25:47.418 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:25:47.418 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:25:47.418 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:25:47.418 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:25:47.418 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:25:47.418 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:25:47.418 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:25:47.418 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:25:47.418 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:25:47.418 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:25:47.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:25:47.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:25:47.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:25:47.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:25:47.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:25:47.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:25:47.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:25:47.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:25:47.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:25:47.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:25:47.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:25:47.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:25:47.420 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:25:47.420 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:25:47.420 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:25:47.420 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:25:47.442 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:25:47.443 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:25:47.443 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:25:47.443 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:25:47.585 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:25:47.587 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:25:47.588 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:47.588 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:47.588 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:47.588 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:25:47.588 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:47.588 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:47.589 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:47.589 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:25:47.589 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:25:47.589 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:25:47.589 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:25:47.590 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:47.590 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:47.591 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:25:47.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 26 2024-09-11T06:25:47.649 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:25:47.649 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:25:47.649 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:25:47.649 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=104, tag_tid=20, entry_tid=7564], [object_number=107, tag_tid=20, entry_tid=7563], [object_number=106, tag_tid=20, entry_tid=7562], [object_number=105, tag_tid=20, entry_tid=7561]]], state=connected] 2024-09-11T06:25:47.652 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:47.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:25:47.654 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:25:47.654 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:47.654 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:47.655 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:47.655 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:25:47.655 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:47.655 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:47.655 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:47.655 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:25:47.655 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:25:47.655 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:25:47.655 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:25:47.656 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:47.656 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:25:47.656 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:47.702 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 26 2024-09-11T06:25:47.702 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:25:47.702 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:25:47.702 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:25:47.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=104, tag_tid=20, entry_tid=7564], [object_number=107, tag_tid=20, entry_tid=7563], [object_number=106, tag_tid=20, entry_tid=7562], [object_number=105, tag_tid=20, entry_tid=7561]]], state=connected] 2024-09-11T06:25:47.706 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:47.707 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=104, tag_tid=20, entry_tid=7564' 2024-09-11T06:25:47.707 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=104, tag_tid=20, entry_tid=7564' 2024-09-11T06:25:47.708 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=104, tag_tid=20, entry_tid=7564' '!=' 'object_number=106, tag_tid=20, entry_tid=7506' 2024-09-11T06:25:47.708 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:25:47.708 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=104, tag_tid=20, entry_tid=7564' = 'object_number=106, tag_tid=20, entry_tid=7506' 2024-09-11T06:25:47.708 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=104, tag_tid=20, entry_tid=7564' 2024-09-11T06:25:47.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:25:47.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:25:47.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:47.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:25:47.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=104, tag_tid=20, entry_tid=7564' 2024-09-11T06:25:47.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:25:47.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:47.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:25:47.711 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:25:47.711 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:25:47.712 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:47.712 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:25:47.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=104, tag_tid=20, entry_tid=7564' 2024-09-11T06:25:47.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:25:47.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:47.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=7564 2024-09-11T06:25:47.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 7564 2024-09-11T06:25:47.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:25:47.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:25:47.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:25:47.916 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:25:47.916 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:25:47.916 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:25:47.916 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:25:47.916 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:25:47.917 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:25:47.917 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:25:47.917 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:25:47.917 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:25:47.917 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:25:47.917 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:25:47.917 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:25:47.917 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:25:47.918 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:25:47.918 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:25:47.918 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:25:47.918 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:25:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:25:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:25:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:25:47.922 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:25:47.922 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:25:47.922 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:25:47.922 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:25:47.922 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:25:47.922 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:25:47.922 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:25:47.922 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:25:47.922 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:25:47.922 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:25:47.922 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:25:47.922 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:25:47.922 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:25:47.922 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:25:47.923 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:25:47.923 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:25:47.923 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:25:47.923 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:25:47.923 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:25:47.923 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:25:47.923 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:25:47.923 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:25:47.923 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:25:47.923 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:25:47.923 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:25:47.923 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:25:47.923 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:25:47.924 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:25:47.924 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:25:47.948 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:25:47.948 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:25:47.948 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:25:47.949 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:25:48.073 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:25:48.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:25:48.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:48.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:48.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:48.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:25:48.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:48.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:48.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:48.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:25:48.076 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:25:48.076 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:25:48.076 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:25:48.077 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:48.077 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:25:48.077 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:48.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 26 2024-09-11T06:25:48.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:25:48.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:25:48.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:25:48.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=104, tag_tid=20, entry_tid=7636], [object_number=107, tag_tid=20, entry_tid=7635], [object_number=106, tag_tid=20, entry_tid=7634], [object_number=105, tag_tid=20, entry_tid=7633]]], state=connected] 2024-09-11T06:25:48.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:48.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:25:48.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:25:48.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:48.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:48.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:48.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:25:48.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:48.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:48.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:48.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:25:48.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:25:48.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:25:48.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:25:48.139 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:48.139 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:25:48.139 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:48.186 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 26 2024-09-11T06:25:48.187 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:25:48.187 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:25:48.187 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:25:48.187 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=104, tag_tid=20, entry_tid=7636], [object_number=107, tag_tid=20, entry_tid=7635], [object_number=106, tag_tid=20, entry_tid=7634], [object_number=105, tag_tid=20, entry_tid=7633]]], state=connected] 2024-09-11T06:25:48.190 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:48.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=104, tag_tid=20, entry_tid=7636' 2024-09-11T06:25:48.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=104, tag_tid=20, entry_tid=7636' 2024-09-11T06:25:48.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=104, tag_tid=20, entry_tid=7636' '!=' 'object_number=104, tag_tid=20, entry_tid=7564' 2024-09-11T06:25:48.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:25:48.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=104, tag_tid=20, entry_tid=7636' = 'object_number=104, tag_tid=20, entry_tid=7564' 2024-09-11T06:25:48.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=104, tag_tid=20, entry_tid=7636' 2024-09-11T06:25:48.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:25:48.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:25:48.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:48.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:25:48.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=104, tag_tid=20, entry_tid=7636' 2024-09-11T06:25:48.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:25:48.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:48.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:25:48.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:25:48.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:25:48.196 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:48.196 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:25:48.197 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=104, tag_tid=20, entry_tid=7636' 2024-09-11T06:25:48.197 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:25:48.197 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:48.199 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=7636 2024-09-11T06:25:48.199 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 7636 2024-09-11T06:25:48.199 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:25:48.199 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:25:48.199 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:25:48.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:25:48.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:25:48.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:25:48.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:25:48.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:25:48.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:25:48.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:25:48.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:25:48.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:25:48.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:25:48.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:25:48.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:25:48.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:25:48.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:25:48.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:25:48.403 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:25:48.403 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:25:48.407 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:25:48.407 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:25:48.407 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:25:48.407 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:25:48.407 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:25:48.407 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:25:48.407 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:25:48.408 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:25:48.408 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:25:48.408 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:25:48.408 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:25:48.408 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:25:48.408 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:25:48.408 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:25:48.408 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:25:48.408 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:25:48.408 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:25:48.408 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:25:48.408 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:25:48.408 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:25:48.408 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:25:48.409 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:25:48.409 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:25:48.409 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:25:48.409 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:25:48.409 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:25:48.409 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:25:48.409 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:25:48.409 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:25:48.409 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:25:48.409 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:25:48.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:25:48.431 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:25:48.431 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:25:48.431 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:25:48.432 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:25:48.560 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:25:48.561 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:25:48.561 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:48.561 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:48.562 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:48.562 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:25:48.562 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:48.562 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:48.562 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:48.562 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:25:48.562 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:25:48.563 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:25:48.563 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:25:48.563 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:48.564 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:48.564 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:25:48.611 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 26 2024-09-11T06:25:48.611 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:25:48.611 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:25:48.611 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:25:48.611 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=107, tag_tid=20, entry_tid=7711], [object_number=106, tag_tid=20, entry_tid=7710], [object_number=105, tag_tid=20, entry_tid=7709], [object_number=104, tag_tid=20, entry_tid=7708]]], state=connected] 2024-09-11T06:25:48.615 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:48.617 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:25:48.617 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:25:48.617 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:48.618 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:48.618 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:48.618 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:25:48.618 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:48.618 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:48.618 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:48.618 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:25:48.618 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:25:48.618 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:25:48.618 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:25:48.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:48.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:25:48.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:48.665 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 26 2024-09-11T06:25:48.665 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:25:48.665 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:25:48.665 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:25:48.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=107, tag_tid=20, entry_tid=7711], [object_number=106, tag_tid=20, entry_tid=7710], [object_number=105, tag_tid=20, entry_tid=7709], [object_number=104, tag_tid=20, entry_tid=7708]]], state=connected] 2024-09-11T06:25:48.669 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:48.671 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=107, tag_tid=20, entry_tid=7711' 2024-09-11T06:25:48.671 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=107, tag_tid=20, entry_tid=7711' 2024-09-11T06:25:48.671 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=107, tag_tid=20, entry_tid=7711' '!=' 'object_number=104, tag_tid=20, entry_tid=7636' 2024-09-11T06:25:48.671 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:25:48.671 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=107, tag_tid=20, entry_tid=7711' = 'object_number=104, tag_tid=20, entry_tid=7636' 2024-09-11T06:25:48.671 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=107, tag_tid=20, entry_tid=7711' 2024-09-11T06:25:48.671 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:25:48.671 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:25:48.671 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:48.672 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:25:48.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=107, tag_tid=20, entry_tid=7711' 2024-09-11T06:25:48.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:25:48.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:48.674 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:25:48.674 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:25:48.674 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:25:48.674 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:48.675 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:25:48.676 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=107, tag_tid=20, entry_tid=7711' 2024-09-11T06:25:48.676 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:25:48.676 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:48.677 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=7711 2024-09-11T06:25:48.677 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 7711 2024-09-11T06:25:48.677 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:25:48.677 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:25:48.678 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:25:48.879 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:25:48.879 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:25:48.880 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:25:48.880 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:25:48.880 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:25:48.880 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:25:48.880 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:25:48.880 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:25:48.880 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:25:48.880 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:25:48.880 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:25:48.880 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:25:48.880 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:25:48.880 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:25:48.880 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:25:48.881 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:25:48.881 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:25:48.884 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:25:48.884 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:25:48.884 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:25:48.885 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:25:48.885 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:25:48.885 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:25:48.885 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:25:48.885 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:25:48.885 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:25:48.885 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:25:48.885 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:25:48.885 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:25:48.885 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:25:48.885 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:25:48.885 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:25:48.885 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:25:48.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:25:48.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:25:48.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:25:48.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:25:48.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:25:48.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:25:48.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:25:48.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:25:48.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:25:48.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:25:48.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:25:48.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:25:48.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:25:48.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:25:48.887 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:25:48.887 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:25:48.910 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:25:48.910 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:25:48.910 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:25:48.910 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:25:49.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:25:49.041 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:25:49.041 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:49.041 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:49.041 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:49.041 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:25:49.041 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:49.041 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:49.041 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:49.041 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:25:49.041 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:25:49.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:25:49.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:25:49.043 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:49.043 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:25:49.043 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:49.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 26 2024-09-11T06:25:49.095 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:25:49.095 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:25:49.095 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:25:49.095 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=104, tag_tid=20, entry_tid=7784], [object_number=107, tag_tid=20, entry_tid=7783], [object_number=106, tag_tid=20, entry_tid=7782], [object_number=105, tag_tid=20, entry_tid=7781]]], state=connected] 2024-09-11T06:25:49.098 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:49.099 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:25:49.101 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:25:49.101 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:49.101 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:49.101 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:49.101 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:25:49.101 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:49.101 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:49.101 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:49.101 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:25:49.101 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:25:49.102 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:25:49.102 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:25:49.102 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:49.102 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:25:49.102 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:49.148 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 26 2024-09-11T06:25:49.149 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:25:49.149 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:25:49.149 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:25:49.149 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=104, tag_tid=20, entry_tid=7784], [object_number=107, tag_tid=20, entry_tid=7783], [object_number=106, tag_tid=20, entry_tid=7782], [object_number=105, tag_tid=20, entry_tid=7781]]], state=connected] 2024-09-11T06:25:49.152 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:49.155 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=104, tag_tid=20, entry_tid=7784' 2024-09-11T06:25:49.155 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=104, tag_tid=20, entry_tid=7784' 2024-09-11T06:25:49.155 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=104, tag_tid=20, entry_tid=7784' '!=' 'object_number=107, tag_tid=20, entry_tid=7711' 2024-09-11T06:25:49.155 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:25:49.155 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=104, tag_tid=20, entry_tid=7784' = 'object_number=107, tag_tid=20, entry_tid=7711' 2024-09-11T06:25:49.155 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=104, tag_tid=20, entry_tid=7784' 2024-09-11T06:25:49.155 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:25:49.155 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:25:49.155 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:49.156 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:25:49.156 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=104, tag_tid=20, entry_tid=7784' 2024-09-11T06:25:49.156 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:25:49.156 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:49.157 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:25:49.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:25:49.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:25:49.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:49.159 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:25:49.160 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=104, tag_tid=20, entry_tid=7784' 2024-09-11T06:25:49.160 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:25:49.160 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:49.161 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=7784 2024-09-11T06:25:49.161 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 7784 2024-09-11T06:25:49.161 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:25:49.161 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:25:49.161 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:25:49.363 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:25:49.363 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:25:49.363 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:25:49.363 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:25:49.363 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:25:49.364 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:25:49.364 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:25:49.364 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:25:49.364 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:25:49.364 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:25:49.364 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:25:49.364 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:25:49.364 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:25:49.364 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:25:49.364 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:25:49.365 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:25:49.365 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:25:49.367 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:25:49.367 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:25:49.368 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:25:49.368 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:25:49.368 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:25:49.368 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:25:49.368 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:25:49.368 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:25:49.368 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:25:49.368 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:25:49.368 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:25:49.368 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:25:49.368 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:25:49.368 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:25:49.368 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:25:49.369 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:25:49.369 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:25:49.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:25:49.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:25:49.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:25:49.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:25:49.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:25:49.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:25:49.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:25:49.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:25:49.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:25:49.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:25:49.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:25:49.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:25:49.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:25:49.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:25:49.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:25:49.394 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:25:49.394 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:25:49.394 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:25:49.394 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:25:49.526 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:25:49.527 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:25:49.527 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:49.527 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:49.528 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:49.528 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:25:49.528 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:49.528 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:49.528 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:49.528 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:25:49.528 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:25:49.529 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:25:49.529 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:25:49.531 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:49.531 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:49.531 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:25:49.585 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 26 2024-09-11T06:25:49.585 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:25:49.585 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:25:49.585 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:25:49.585 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=106, tag_tid=20, entry_tid=7858], [object_number=105, tag_tid=20, entry_tid=7857], [object_number=104, tag_tid=20, entry_tid=7856], [object_number=107, tag_tid=20, entry_tid=7855]]], state=connected] 2024-09-11T06:25:49.588 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:49.590 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:25:49.590 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:25:49.591 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:49.591 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:49.591 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:49.591 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:25:49.591 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:49.591 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:49.591 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:49.591 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:25:49.591 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:25:49.591 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:25:49.591 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:25:49.592 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:49.592 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:25:49.592 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:49.640 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 26 2024-09-11T06:25:49.640 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:25:49.640 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:25:49.640 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:25:49.640 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=106, tag_tid=20, entry_tid=7858], [object_number=105, tag_tid=20, entry_tid=7857], [object_number=104, tag_tid=20, entry_tid=7856], [object_number=107, tag_tid=20, entry_tid=7855]]], state=connected] 2024-09-11T06:25:49.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:49.646 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=106, tag_tid=20, entry_tid=7858' 2024-09-11T06:25:49.646 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=106, tag_tid=20, entry_tid=7858' 2024-09-11T06:25:49.646 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=106, tag_tid=20, entry_tid=7858' '!=' 'object_number=104, tag_tid=20, entry_tid=7784' 2024-09-11T06:25:49.646 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:25:49.646 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=106, tag_tid=20, entry_tid=7858' = 'object_number=104, tag_tid=20, entry_tid=7784' 2024-09-11T06:25:49.646 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=106, tag_tid=20, entry_tid=7858' 2024-09-11T06:25:49.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:25:49.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:25:49.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:49.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:25:49.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=106, tag_tid=20, entry_tid=7858' 2024-09-11T06:25:49.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:25:49.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:49.649 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:25:49.650 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:25:49.650 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:25:49.650 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:49.651 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:25:49.651 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=106, tag_tid=20, entry_tid=7858' 2024-09-11T06:25:49.651 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:25:49.651 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:49.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=7858 2024-09-11T06:25:49.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 7858 2024-09-11T06:25:49.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:25:49.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:25:49.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:25:49.854 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:25:49.854 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:25:49.854 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:25:49.854 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:25:49.854 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:25:49.855 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:25:49.855 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:25:49.855 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:25:49.855 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:25:49.855 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:25:49.855 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:25:49.855 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:25:49.855 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:25:49.855 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:25:49.855 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:25:49.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:25:49.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:25:49.857 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:25:49.857 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:25:49.857 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:25:49.857 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:25:49.857 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:25:49.857 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:25:49.857 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:25:49.857 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:25:49.857 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:25:49.857 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:25:49.857 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:25:49.858 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:25:49.858 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:25:49.858 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:25:49.858 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:25:49.858 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:25:49.859 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:25:49.859 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:25:49.859 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:25:49.859 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:25:49.860 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:25:49.860 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:25:49.860 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:25:49.860 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:25:49.860 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:25:49.860 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:25:49.860 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:25:49.860 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:25:49.860 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:25:49.860 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:25:49.860 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:25:49.860 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:25:49.884 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:25:49.884 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:25:49.884 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:25:49.884 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:25:50.018 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:25:50.019 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:25:50.019 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:50.019 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:50.019 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:50.019 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:25:50.019 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:50.019 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:50.020 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:50.020 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:25:50.020 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:25:50.020 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:25:50.020 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:25:50.022 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:50.022 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:25:50.022 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:50.073 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 26 2024-09-11T06:25:50.074 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:25:50.074 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:25:50.074 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:25:50.074 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=104, tag_tid=20, entry_tid=7944], [object_number=107, tag_tid=20, entry_tid=7943], [object_number=106, tag_tid=20, entry_tid=7942], [object_number=105, tag_tid=20, entry_tid=7941]]], state=connected] 2024-09-11T06:25:50.077 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:50.078 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:25:50.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:25:50.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:50.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:50.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:50.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:25:50.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:50.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:50.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:50.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:25:50.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:25:50.081 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:25:50.081 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:25:50.081 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:50.081 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:25:50.082 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:50.130 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 26 2024-09-11T06:25:50.130 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:25:50.130 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:25:50.130 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:25:50.130 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=104, tag_tid=20, entry_tid=7944], [object_number=107, tag_tid=20, entry_tid=7943], [object_number=106, tag_tid=20, entry_tid=7942], [object_number=105, tag_tid=20, entry_tid=7941]]], state=connected] 2024-09-11T06:25:50.133 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:50.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=104, tag_tid=20, entry_tid=7944' 2024-09-11T06:25:50.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=104, tag_tid=20, entry_tid=7944' 2024-09-11T06:25:50.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=104, tag_tid=20, entry_tid=7944' '!=' 'object_number=106, tag_tid=20, entry_tid=7858' 2024-09-11T06:25:50.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:25:50.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=104, tag_tid=20, entry_tid=7944' = 'object_number=106, tag_tid=20, entry_tid=7858' 2024-09-11T06:25:50.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=104, tag_tid=20, entry_tid=7944' 2024-09-11T06:25:50.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:25:50.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:25:50.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:50.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:25:50.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=104, tag_tid=20, entry_tid=7944' 2024-09-11T06:25:50.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:25:50.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:50.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:25:50.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:25:50.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:25:50.139 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:50.139 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:25:50.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=104, tag_tid=20, entry_tid=7944' 2024-09-11T06:25:50.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:25:50.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:50.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=7944 2024-09-11T06:25:50.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 7944 2024-09-11T06:25:50.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:25:50.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:25:50.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:25:50.343 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:25:50.343 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:25:50.343 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:25:50.343 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:25:50.344 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:25:50.344 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:25:50.344 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:25:50.344 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:25:50.344 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:25:50.344 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:25:50.344 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:25:50.344 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:25:50.344 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:25:50.344 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:25:50.344 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:25:50.345 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:25:50.345 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:25:50.349 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:25:50.349 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:25:50.349 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:25:50.349 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:25:50.349 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:25:50.349 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:25:50.350 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:25:50.350 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:25:50.350 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:25:50.350 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:25:50.350 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:25:50.350 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:25:50.350 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:25:50.350 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:25:50.350 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:25:50.350 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:25:50.350 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:25:50.350 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:25:50.350 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:25:50.350 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:25:50.351 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:25:50.351 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:25:50.351 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:25:50.351 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:25:50.351 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:25:50.351 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:25:50.351 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:25:50.351 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:25:50.351 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:25:50.351 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:25:50.351 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:25:50.351 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:25:50.375 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:25:50.375 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:25:50.375 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:25:50.375 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:25:50.502 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:25:50.504 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:25:50.504 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:50.504 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:50.504 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:50.504 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:25:50.504 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:50.504 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:50.504 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:50.504 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:25:50.504 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:25:50.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:25:50.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:25:50.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:50.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:25:50.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:50.565 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 26 2024-09-11T06:25:50.565 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:25:50.565 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:25:50.565 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:25:50.565 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=107, tag_tid=20, entry_tid=8027], [object_number=106, tag_tid=20, entry_tid=8026], [object_number=105, tag_tid=20, entry_tid=8025], [object_number=104, tag_tid=20, entry_tid=8024]]], state=connected] 2024-09-11T06:25:50.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:50.569 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:25:50.570 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:25:50.570 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:50.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:50.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:50.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:25:50.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:50.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:50.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:50.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:25:50.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:25:50.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:25:50.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:25:50.572 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:50.572 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:25:50.572 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:50.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 26 2024-09-11T06:25:50.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:25:50.621 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:25:50.621 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:25:50.621 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=107, tag_tid=20, entry_tid=8027], [object_number=106, tag_tid=20, entry_tid=8026], [object_number=105, tag_tid=20, entry_tid=8025], [object_number=104, tag_tid=20, entry_tid=8024]]], state=connected] 2024-09-11T06:25:50.624 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:50.626 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=107, tag_tid=20, entry_tid=8027' 2024-09-11T06:25:50.626 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=107, tag_tid=20, entry_tid=8027' 2024-09-11T06:25:50.626 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=107, tag_tid=20, entry_tid=8027' '!=' 'object_number=104, tag_tid=20, entry_tid=7944' 2024-09-11T06:25:50.626 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:25:50.626 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=107, tag_tid=20, entry_tid=8027' = 'object_number=104, tag_tid=20, entry_tid=7944' 2024-09-11T06:25:50.626 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=107, tag_tid=20, entry_tid=8027' 2024-09-11T06:25:50.626 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:25:50.626 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:25:50.626 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:50.627 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:25:50.627 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=107, tag_tid=20, entry_tid=8027' 2024-09-11T06:25:50.628 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:25:50.628 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:50.629 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:25:50.629 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:25:50.629 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:25:50.629 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:50.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:25:50.631 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=107, tag_tid=20, entry_tid=8027' 2024-09-11T06:25:50.631 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:25:50.631 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:50.632 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=8027 2024-09-11T06:25:50.632 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 8027 2024-09-11T06:25:50.632 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:25:50.633 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:25:50.633 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:25:50.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:25:50.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:25:50.835 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:25:50.835 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:25:50.835 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:25:50.835 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:25:50.835 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:25:50.835 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:25:50.835 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:25:50.835 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:25:50.835 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:25:50.835 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:25:50.835 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:25:50.835 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:25:50.835 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:25:50.836 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:25:50.836 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:25:50.841 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:25:50.841 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:25:50.841 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:25:50.841 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:25:50.841 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:25:50.841 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:25:50.841 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:25:50.841 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:25:50.841 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:25:50.841 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:25:50.842 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:25:50.842 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:25:50.842 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:25:50.842 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:25:50.842 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:25:50.842 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:25:50.842 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:25:50.842 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:25:50.842 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:25:50.842 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:25:50.842 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:25:50.842 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:25:50.842 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:25:50.843 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:25:50.843 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:25:50.843 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:25:50.843 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:25:50.843 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:25:50.843 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:25:50.843 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:25:50.843 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:25:50.843 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:25:50.865 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:25:50.865 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:25:50.865 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:25:50.865 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:25:50.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:25:50.999 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:25:50.999 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:50.999 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:50.999 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:50.999 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:25:50.999 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:50.999 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:51.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:51.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:25:51.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:25:51.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:25:51.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:25:51.002 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:51.002 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:25:51.002 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:51.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 26 2024-09-11T06:25:51.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:25:51.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:25:51.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:25:51.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=107, tag_tid=20, entry_tid=8111], [object_number=106, tag_tid=20, entry_tid=8110], [object_number=105, tag_tid=20, entry_tid=8109], [object_number=104, tag_tid=20, entry_tid=8108]]], state=connected] 2024-09-11T06:25:51.059 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:51.060 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:25:51.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:25:51.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:51.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:51.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:51.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:25:51.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:51.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:51.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:51.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:25:51.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:25:51.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:25:51.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:25:51.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:51.063 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:25:51.063 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:51.112 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 26 2024-09-11T06:25:51.112 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:25:51.112 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:25:51.112 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:25:51.112 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=107, tag_tid=20, entry_tid=8111], [object_number=106, tag_tid=20, entry_tid=8110], [object_number=105, tag_tid=20, entry_tid=8109], [object_number=104, tag_tid=20, entry_tid=8108]]], state=connected] 2024-09-11T06:25:51.115 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:51.117 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=107, tag_tid=20, entry_tid=8111' 2024-09-11T06:25:51.117 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=107, tag_tid=20, entry_tid=8111' 2024-09-11T06:25:51.118 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=107, tag_tid=20, entry_tid=8111' '!=' 'object_number=107, tag_tid=20, entry_tid=8027' 2024-09-11T06:25:51.118 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:25:51.118 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=107, tag_tid=20, entry_tid=8111' = 'object_number=107, tag_tid=20, entry_tid=8027' 2024-09-11T06:25:51.118 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=107, tag_tid=20, entry_tid=8111' 2024-09-11T06:25:51.118 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:25:51.118 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:25:51.118 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:51.118 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:25:51.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=107, tag_tid=20, entry_tid=8111' 2024-09-11T06:25:51.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:25:51.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:51.120 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:25:51.121 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:25:51.121 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:25:51.121 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:51.122 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:25:51.122 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=107, tag_tid=20, entry_tid=8111' 2024-09-11T06:25:51.123 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:25:51.123 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:51.124 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=8111 2024-09-11T06:25:51.124 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 8111 2024-09-11T06:25:51.124 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:25:51.124 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:25:51.124 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:25:51.326 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:25:51.326 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:25:51.326 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:25:51.326 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:25:51.326 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:25:51.326 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:25:51.326 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:25:51.326 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:25:51.326 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:25:51.326 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:25:51.327 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:25:51.327 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:25:51.327 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:25:51.327 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:25:51.327 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:25:51.327 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:25:51.327 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:25:51.331 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:25:51.331 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:25:51.332 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:25:51.332 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:25:51.332 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:25:51.332 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:25:51.332 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:25:51.332 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:25:51.332 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:25:51.332 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:25:51.332 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:25:51.332 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:25:51.332 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:25:51.332 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:25:51.332 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:25:51.333 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:25:51.333 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:25:51.333 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:25:51.333 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:25:51.333 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:25:51.333 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:25:51.333 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:25:51.333 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:25:51.333 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:25:51.333 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:25:51.333 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:25:51.333 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:25:51.333 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:25:51.334 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:25:51.334 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:25:51.334 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:25:51.334 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:25:51.356 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:25:51.357 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:25:51.357 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:25:51.357 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:25:51.485 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:25:51.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:25:51.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:51.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:51.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:51.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:25:51.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:51.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:51.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:51.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:25:51.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:25:51.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:25:51.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:25:51.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:51.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:25:51.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:51.546 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 26 2024-09-11T06:25:51.546 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:25:51.547 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:25:51.547 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:25:51.547 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=104, tag_tid=20, entry_tid=8192], [object_number=107, tag_tid=20, entry_tid=8191], [object_number=106, tag_tid=20, entry_tid=8190], [object_number=105, tag_tid=20, entry_tid=8189]]], state=connected] 2024-09-11T06:25:51.550 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:51.551 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:25:51.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:25:51.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:51.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:51.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:51.553 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:25:51.553 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:51.553 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:51.553 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:51.553 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:25:51.553 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:25:51.553 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:25:51.553 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:25:51.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:51.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:25:51.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:51.601 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 26 2024-09-11T06:25:51.601 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:25:51.601 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:25:51.601 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:25:51.601 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=104, tag_tid=20, entry_tid=8192], [object_number=107, tag_tid=20, entry_tid=8191], [object_number=106, tag_tid=20, entry_tid=8190], [object_number=105, tag_tid=20, entry_tid=8189]]], state=connected] 2024-09-11T06:25:51.604 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:51.606 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=104, tag_tid=20, entry_tid=8192' 2024-09-11T06:25:51.606 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=104, tag_tid=20, entry_tid=8192' 2024-09-11T06:25:51.606 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=104, tag_tid=20, entry_tid=8192' '!=' 'object_number=107, tag_tid=20, entry_tid=8111' 2024-09-11T06:25:51.606 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:25:51.606 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=104, tag_tid=20, entry_tid=8192' = 'object_number=107, tag_tid=20, entry_tid=8111' 2024-09-11T06:25:51.607 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=104, tag_tid=20, entry_tid=8192' 2024-09-11T06:25:51.607 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:25:51.607 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:25:51.607 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:51.607 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:25:51.608 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=104, tag_tid=20, entry_tid=8192' 2024-09-11T06:25:51.608 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:25:51.608 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:51.609 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:25:51.610 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:25:51.610 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:25:51.610 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:51.611 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:25:51.611 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=104, tag_tid=20, entry_tid=8192' 2024-09-11T06:25:51.611 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:25:51.612 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:51.613 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=8192 2024-09-11T06:25:51.613 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 8192 2024-09-11T06:25:51.613 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:25:51.613 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:25:51.613 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:25:51.815 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:25:51.815 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:25:51.815 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:25:51.815 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:25:51.815 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:25:51.816 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:25:51.816 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:25:51.816 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:25:51.816 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:25:51.816 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:25:51.816 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:25:51.816 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:25:51.816 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:25:51.816 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:25:51.816 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:25:51.816 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:25:51.818 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:25:51.818 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:25:51.819 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:25:51.819 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:25:51.819 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:25:51.819 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:25:51.819 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:25:51.819 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:25:51.819 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:25:51.819 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:25:51.819 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:25:51.819 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:25:51.819 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:25:51.819 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:25:51.819 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:25:51.820 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:25:51.820 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:25:51.820 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:25:51.820 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:25:51.820 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:25:51.820 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:25:51.820 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:25:51.820 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:25:51.820 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:25:51.820 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:25:51.820 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:25:51.820 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:25:51.820 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:25:51.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:25:51.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:25:51.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:25:51.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:25:51.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:25:51.845 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:25:51.845 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:25:51.845 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:25:51.845 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:25:51.976 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:25:51.977 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:25:51.977 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:51.977 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:51.977 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:51.977 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:25:51.977 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:51.977 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:51.977 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:51.977 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:25:51.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:25:51.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:25:51.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:25:51.979 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:51.980 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:25:51.980 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:52.028 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 26 2024-09-11T06:25:52.028 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:25:52.028 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:25:52.028 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:25:52.028 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=104, tag_tid=20, entry_tid=8280], [object_number=107, tag_tid=20, entry_tid=8279], [object_number=106, tag_tid=20, entry_tid=8278], [object_number=105, tag_tid=20, entry_tid=8277]]], state=connected] 2024-09-11T06:25:52.031 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:52.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:25:52.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:25:52.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:52.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:52.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:52.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:25:52.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:52.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:52.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:52.035 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:25:52.035 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:25:52.035 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:25:52.035 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:25:52.035 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:52.036 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:25:52.036 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:52.082 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 26 2024-09-11T06:25:52.082 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:25:52.082 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:25:52.083 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:25:52.083 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=104, tag_tid=20, entry_tid=8280], [object_number=107, tag_tid=20, entry_tid=8279], [object_number=106, tag_tid=20, entry_tid=8278], [object_number=105, tag_tid=20, entry_tid=8277]]], state=connected] 2024-09-11T06:25:52.086 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:52.088 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=104, tag_tid=20, entry_tid=8280' 2024-09-11T06:25:52.088 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=104, tag_tid=20, entry_tid=8280' 2024-09-11T06:25:52.088 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=104, tag_tid=20, entry_tid=8280' '!=' 'object_number=104, tag_tid=20, entry_tid=8192' 2024-09-11T06:25:52.088 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:25:52.088 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=104, tag_tid=20, entry_tid=8280' = 'object_number=104, tag_tid=20, entry_tid=8192' 2024-09-11T06:25:52.088 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=104, tag_tid=20, entry_tid=8280' 2024-09-11T06:25:52.089 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:25:52.089 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:25:52.089 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:52.089 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:25:52.090 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=104, tag_tid=20, entry_tid=8280' 2024-09-11T06:25:52.090 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:25:52.090 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:52.091 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:25:52.092 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:25:52.092 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:25:52.092 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:52.092 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:25:52.093 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=104, tag_tid=20, entry_tid=8280' 2024-09-11T06:25:52.093 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:25:52.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:52.095 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=8280 2024-09-11T06:25:52.095 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 8280 2024-09-11T06:25:52.095 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:25:52.095 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:25:52.095 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:25:52.296 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:25:52.297 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:25:52.297 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:25:52.297 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:25:52.297 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:25:52.297 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:25:52.297 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:25:52.297 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:25:52.297 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:25:52.297 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:25:52.297 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:25:52.297 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:25:52.298 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:25:52.298 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:25:52.298 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:25:52.298 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:25:52.298 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:25:52.301 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:25:52.301 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:25:52.301 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:25:52.301 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:25:52.301 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:25:52.301 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:25:52.301 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:25:52.301 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:25:52.301 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:25:52.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:25:52.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:25:52.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:25:52.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:25:52.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:25:52.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:25:52.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:25:52.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:25:52.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:25:52.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:25:52.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:25:52.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:25:52.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:25:52.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:25:52.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:25:52.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:25:52.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:25:52.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:25:52.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:25:52.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:25:52.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:25:52.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:25:52.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:25:52.322 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:25:52.322 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:25:52.322 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:25:52.322 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:25:52.451 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:25:52.452 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:25:52.452 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:52.452 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:52.452 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:52.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:25:52.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:52.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:52.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:52.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:25:52.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:25:52.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:25:52.454 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:25:52.455 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:52.456 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:25:52.456 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:52.509 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 26 2024-09-11T06:25:52.509 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:25:52.509 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:25:52.509 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:25:52.509 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=106, tag_tid=20, entry_tid=8370], [object_number=105, tag_tid=20, entry_tid=8369], [object_number=104, tag_tid=20, entry_tid=8368], [object_number=107, tag_tid=20, entry_tid=8367]]], state=connected] 2024-09-11T06:25:52.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:52.514 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:25:52.515 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:25:52.515 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:52.515 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:52.515 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:52.515 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:25:52.515 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:52.516 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:52.516 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:52.516 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:25:52.516 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:25:52.516 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:25:52.516 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:25:52.517 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:52.517 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:25:52.517 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:52.564 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 26 2024-09-11T06:25:52.564 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:25:52.564 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:25:52.565 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:25:52.565 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=106, tag_tid=20, entry_tid=8370], [object_number=105, tag_tid=20, entry_tid=8369], [object_number=104, tag_tid=20, entry_tid=8368], [object_number=107, tag_tid=20, entry_tid=8367]]], state=connected] 2024-09-11T06:25:52.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:52.569 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=106, tag_tid=20, entry_tid=8370' 2024-09-11T06:25:52.569 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=106, tag_tid=20, entry_tid=8370' 2024-09-11T06:25:52.569 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=106, tag_tid=20, entry_tid=8370' '!=' 'object_number=104, tag_tid=20, entry_tid=8280' 2024-09-11T06:25:52.570 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:25:52.570 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=106, tag_tid=20, entry_tid=8370' = 'object_number=104, tag_tid=20, entry_tid=8280' 2024-09-11T06:25:52.570 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=106, tag_tid=20, entry_tid=8370' 2024-09-11T06:25:52.570 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:25:52.570 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:25:52.570 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:52.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:25:52.572 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=106, tag_tid=20, entry_tid=8370' 2024-09-11T06:25:52.572 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:25:52.572 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:52.573 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:25:52.573 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:25:52.573 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:25:52.574 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:52.574 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:25:52.575 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=106, tag_tid=20, entry_tid=8370' 2024-09-11T06:25:52.575 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:25:52.575 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:52.576 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=8370 2024-09-11T06:25:52.577 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 8370 2024-09-11T06:25:52.577 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:25:52.577 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:25:52.577 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:25:52.779 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:25:52.779 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:25:52.779 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:25:52.779 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:25:52.779 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:25:52.779 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:25:52.779 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:25:52.779 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:25:52.779 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:25:52.780 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:25:52.780 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:25:52.780 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:25:52.780 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:25:52.780 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:25:52.780 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:25:52.780 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:25:52.782 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:25:52.782 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:25:52.782 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:25:52.782 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:25:52.782 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:25:52.782 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:25:52.782 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:25:52.782 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:25:52.782 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:25:52.783 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:25:52.783 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:25:52.783 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:25:52.783 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:25:52.783 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:25:52.783 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:25:52.783 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:25:52.783 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:25:52.783 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:25:52.783 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:25:52.783 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:25:52.783 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:25:52.783 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:25:52.783 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:25:52.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:25:52.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:25:52.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:25:52.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:25:52.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:25:52.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:25:52.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:25:52.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:25:52.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:25:52.785 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:25:52.808 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:25:52.808 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:25:52.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:25:52.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:25:52.936 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:25:52.938 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:25:52.938 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:52.938 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:52.938 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:52.938 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:25:52.938 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:52.938 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:52.938 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:52.938 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:25:52.939 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:25:52.939 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:25:52.939 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:25:52.940 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:52.940 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:25:52.941 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:52.988 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 26 2024-09-11T06:25:52.989 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:25:52.989 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:25:52.989 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:25:52.989 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=104, tag_tid=20, entry_tid=8464], [object_number=107, tag_tid=20, entry_tid=8463], [object_number=106, tag_tid=20, entry_tid=8462], [object_number=105, tag_tid=20, entry_tid=8461]]], state=connected] 2024-09-11T06:25:52.991 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:52.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:25:52.994 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:25:52.994 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:52.994 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:52.994 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:52.994 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:25:52.994 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:52.994 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:52.994 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:52.995 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:25:52.995 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:25:52.995 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:25:52.995 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:25:52.995 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:52.995 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:25:52.995 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:53.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 26 2024-09-11T06:25:53.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:25:53.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:25:53.043 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:25:53.043 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=104, tag_tid=20, entry_tid=8464], [object_number=107, tag_tid=20, entry_tid=8463], [object_number=106, tag_tid=20, entry_tid=8462], [object_number=105, tag_tid=20, entry_tid=8461]]], state=connected] 2024-09-11T06:25:53.046 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:53.047 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=104, tag_tid=20, entry_tid=8464' 2024-09-11T06:25:53.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=104, tag_tid=20, entry_tid=8464' 2024-09-11T06:25:53.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=104, tag_tid=20, entry_tid=8464' '!=' 'object_number=106, tag_tid=20, entry_tid=8370' 2024-09-11T06:25:53.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:25:53.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=104, tag_tid=20, entry_tid=8464' = 'object_number=106, tag_tid=20, entry_tid=8370' 2024-09-11T06:25:53.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=104, tag_tid=20, entry_tid=8464' 2024-09-11T06:25:53.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:25:53.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:25:53.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:53.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:25:53.050 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=104, tag_tid=20, entry_tid=8464' 2024-09-11T06:25:53.050 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:25:53.050 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:53.051 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:25:53.051 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:25:53.051 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:25:53.052 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:53.053 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:25:53.053 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=104, tag_tid=20, entry_tid=8464' 2024-09-11T06:25:53.053 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:25:53.053 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:53.054 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=8464 2024-09-11T06:25:53.054 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 8464 2024-09-11T06:25:53.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:25:53.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:25:53.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:25:53.256 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:25:53.256 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:25:53.256 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:25:53.256 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:25:53.256 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:25:53.257 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:25:53.257 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:25:53.257 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:25:53.257 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:25:53.257 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:25:53.257 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:25:53.257 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:25:53.257 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:25:53.257 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:25:53.257 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:25:53.258 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:25:53.258 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:25:53.260 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:25:53.260 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:25:53.261 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:25:53.261 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:25:53.261 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:25:53.261 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:25:53.261 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:25:53.261 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:25:53.261 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:25:53.261 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:25:53.261 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:25:53.261 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:25:53.261 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:25:53.261 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:25:53.261 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:25:53.262 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:25:53.262 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:25:53.262 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:25:53.262 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:25:53.262 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:25:53.262 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:25:53.262 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:25:53.262 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:25:53.262 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:25:53.262 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:25:53.262 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:25:53.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:25:53.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:25:53.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:25:53.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:25:53.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:25:53.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:25:53.286 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:25:53.286 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:25:53.286 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:25:53.286 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:25:53.416 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:25:53.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:25:53.418 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:53.418 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:53.418 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:53.418 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:25:53.418 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:53.418 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:53.418 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:53.418 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:25:53.418 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:25:53.418 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:25:53.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:25:53.420 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:53.420 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:25:53.420 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:53.471 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 26 2024-09-11T06:25:53.471 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:25:53.471 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:25:53.471 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:25:53.472 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=104, tag_tid=20, entry_tid=8544], [object_number=107, tag_tid=20, entry_tid=8543], [object_number=106, tag_tid=20, entry_tid=8542], [object_number=105, tag_tid=20, entry_tid=8541]]], state=connected] 2024-09-11T06:25:53.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:53.476 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:25:53.477 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:25:53.477 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:53.477 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:53.477 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:53.477 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:25:53.477 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:53.478 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:53.478 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:53.478 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:25:53.478 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:25:53.478 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:25:53.478 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:25:53.479 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:53.479 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:25:53.479 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:53.527 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 26 2024-09-11T06:25:53.527 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:25:53.527 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:25:53.527 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:25:53.527 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=104, tag_tid=20, entry_tid=8544], [object_number=107, tag_tid=20, entry_tid=8543], [object_number=106, tag_tid=20, entry_tid=8542], [object_number=105, tag_tid=20, entry_tid=8541]]], state=connected] 2024-09-11T06:25:53.531 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:53.532 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=104, tag_tid=20, entry_tid=8544' 2024-09-11T06:25:53.532 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=104, tag_tid=20, entry_tid=8544' 2024-09-11T06:25:53.533 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=104, tag_tid=20, entry_tid=8544' '!=' 'object_number=104, tag_tid=20, entry_tid=8464' 2024-09-11T06:25:53.533 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:25:53.533 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=104, tag_tid=20, entry_tid=8544' = 'object_number=104, tag_tid=20, entry_tid=8464' 2024-09-11T06:25:53.533 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=104, tag_tid=20, entry_tid=8544' 2024-09-11T06:25:53.533 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:25:53.533 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:25:53.534 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:53.534 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:25:53.534 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=104, tag_tid=20, entry_tid=8544' 2024-09-11T06:25:53.535 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:25:53.535 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:53.535 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:25:53.536 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:25:53.536 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:25:53.536 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:53.537 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:25:53.538 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=104, tag_tid=20, entry_tid=8544' 2024-09-11T06:25:53.538 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:25:53.538 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:53.539 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=8544 2024-09-11T06:25:53.539 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 8544 2024-09-11T06:25:53.539 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:25:53.539 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:25:53.539 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:25:53.741 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:25:53.741 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:25:53.741 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:25:53.741 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:25:53.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:25:53.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:25:53.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:25:53.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:25:53.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:25:53.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:25:53.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:25:53.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:25:53.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:25:53.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:25:53.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:25:53.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:25:53.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:25:53.746 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:25:53.746 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:25:53.746 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:25:53.746 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:25:53.747 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:25:53.747 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:25:53.747 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:25:53.747 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:25:53.747 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:25:53.747 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:25:53.747 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:25:53.747 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:25:53.747 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:25:53.747 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:25:53.747 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:25:53.747 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:25:53.748 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:25:53.748 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:25:53.748 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:25:53.748 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:25:53.748 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:25:53.748 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:25:53.748 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:25:53.748 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:25:53.748 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:25:53.748 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:25:53.748 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:25:53.748 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:25:53.748 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:25:53.749 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:25:53.749 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:25:53.749 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:25:53.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:25:53.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:25:53.772 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:25:53.772 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:25:53.902 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:25:53.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:25:53.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:53.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:53.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:53.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:25:53.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:53.905 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:53.905 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:53.905 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:25:53.905 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:25:53.905 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:25:53.905 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:25:53.907 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:53.907 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:25:53.908 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:53.954 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 26 2024-09-11T06:25:53.955 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:25:53.955 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:25:53.955 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:25:53.955 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=105, tag_tid=20, entry_tid=8625], [object_number=104, tag_tid=20, entry_tid=8624], [object_number=107, tag_tid=20, entry_tid=8623], [object_number=106, tag_tid=20, entry_tid=8622]]], state=connected] 2024-09-11T06:25:53.958 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:53.959 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:25:53.960 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:25:53.960 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:53.960 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:53.961 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:53.961 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:25:53.961 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:53.961 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:53.961 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:53.961 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:25:53.961 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:25:53.961 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:25:53.961 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:25:53.962 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:53.962 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:25:53.962 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:54.010 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 26 2024-09-11T06:25:54.010 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:25:54.010 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:25:54.010 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:25:54.010 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=105, tag_tid=20, entry_tid=8625], [object_number=104, tag_tid=20, entry_tid=8624], [object_number=107, tag_tid=20, entry_tid=8623], [object_number=106, tag_tid=20, entry_tid=8622]]], state=connected] 2024-09-11T06:25:54.013 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:54.015 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=105, tag_tid=20, entry_tid=8625' 2024-09-11T06:25:54.015 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=105, tag_tid=20, entry_tid=8625' 2024-09-11T06:25:54.015 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=105, tag_tid=20, entry_tid=8625' '!=' 'object_number=104, tag_tid=20, entry_tid=8544' 2024-09-11T06:25:54.016 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:25:54.016 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=105, tag_tid=20, entry_tid=8625' = 'object_number=104, tag_tid=20, entry_tid=8544' 2024-09-11T06:25:54.016 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=105, tag_tid=20, entry_tid=8625' 2024-09-11T06:25:54.016 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:25:54.016 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:25:54.016 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:54.016 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:25:54.017 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=105, tag_tid=20, entry_tid=8625' 2024-09-11T06:25:54.017 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:25:54.017 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:54.018 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:25:54.019 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:25:54.019 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:25:54.019 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:54.020 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:25:54.020 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=105, tag_tid=20, entry_tid=8625' 2024-09-11T06:25:54.021 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:25:54.021 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:54.022 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=8625 2024-09-11T06:25:54.022 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 8625 2024-09-11T06:25:54.022 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:25:54.022 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:25:54.022 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:25:54.224 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:25:54.224 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:25:54.224 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:25:54.224 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:25:54.224 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:25:54.224 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:25:54.224 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:25:54.224 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:25:54.224 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:25:54.224 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:25:54.225 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:25:54.225 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:25:54.225 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:25:54.225 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:25:54.225 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:25:54.225 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:25:54.226 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:25:54.228 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:25:54.228 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:25:54.228 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:25:54.228 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:25:54.228 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:25:54.228 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:25:54.228 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:25:54.229 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:25:54.229 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:25:54.229 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:25:54.229 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:25:54.229 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:25:54.229 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:25:54.229 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:25:54.229 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:25:54.229 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:25:54.230 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:25:54.230 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:25:54.230 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:25:54.230 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:25:54.230 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:25:54.231 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:25:54.231 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:25:54.231 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:25:54.231 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:25:54.231 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:25:54.231 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:25:54.231 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:25:54.231 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:25:54.231 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:25:54.231 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:25:54.231 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:25:54.255 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:25:54.255 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:25:54.255 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:25:54.255 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:25:54.381 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:25:54.382 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:25:54.382 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:54.382 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:54.382 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:54.382 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:25:54.382 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:54.382 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:54.382 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:54.383 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:25:54.383 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:25:54.383 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:25:54.383 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:25:54.385 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:54.385 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:54.385 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:25:54.440 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 26 2024-09-11T06:25:54.440 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:25:54.440 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:25:54.440 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:25:54.441 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=106, tag_tid=20, entry_tid=8698], [object_number=105, tag_tid=20, entry_tid=8697], [object_number=104, tag_tid=20, entry_tid=8696], [object_number=107, tag_tid=20, entry_tid=8695]]], state=connected] 2024-09-11T06:25:54.444 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:54.445 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:25:54.446 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:25:54.446 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:54.446 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:54.446 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:54.446 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:25:54.446 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:54.447 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:54.447 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:54.447 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:25:54.447 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:25:54.447 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:25:54.447 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:25:54.448 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:54.448 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:25:54.448 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:54.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 26 2024-09-11T06:25:54.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:25:54.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:25:54.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:25:54.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=106, tag_tid=20, entry_tid=8698], [object_number=105, tag_tid=20, entry_tid=8697], [object_number=104, tag_tid=20, entry_tid=8696], [object_number=107, tag_tid=20, entry_tid=8695]]], state=connected] 2024-09-11T06:25:54.499 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:54.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=106, tag_tid=20, entry_tid=8698' 2024-09-11T06:25:54.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=106, tag_tid=20, entry_tid=8698' 2024-09-11T06:25:54.501 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=106, tag_tid=20, entry_tid=8698' '!=' 'object_number=105, tag_tid=20, entry_tid=8625' 2024-09-11T06:25:54.501 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:25:54.501 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=106, tag_tid=20, entry_tid=8698' = 'object_number=105, tag_tid=20, entry_tid=8625' 2024-09-11T06:25:54.501 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=106, tag_tid=20, entry_tid=8698' 2024-09-11T06:25:54.501 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:25:54.502 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:25:54.502 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:54.502 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:25:54.503 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=106, tag_tid=20, entry_tid=8698' 2024-09-11T06:25:54.503 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:25:54.503 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:54.504 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:25:54.504 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:25:54.504 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:25:54.504 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:54.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:25:54.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=106, tag_tid=20, entry_tid=8698' 2024-09-11T06:25:54.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:25:54.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:54.507 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=8698 2024-09-11T06:25:54.507 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 8698 2024-09-11T06:25:54.507 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:25:54.507 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:25:54.507 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:25:54.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:25:54.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:25:54.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:25:54.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:25:54.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:25:54.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:25:54.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:25:54.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:25:54.711 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:25:54.711 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:25:54.711 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:25:54.711 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:25:54.711 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:25:54.711 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:25:54.711 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:25:54.711 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:25:54.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:25:54.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:25:54.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:25:54.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:25:54.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:25:54.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:25:54.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:25:54.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:25:54.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:25:54.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:25:54.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:25:54.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:25:54.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:25:54.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:25:54.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:25:54.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:25:54.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:25:54.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:25:54.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:25:54.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:25:54.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:25:54.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:25:54.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:25:54.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:25:54.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:25:54.716 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:25:54.716 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:25:54.716 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:25:54.716 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:25:54.716 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:25:54.716 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:25:54.716 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:25:54.716 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:25:54.740 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:25:54.740 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:25:54.740 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:25:54.740 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:25:54.880 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:25:54.881 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:25:54.881 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:54.882 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:54.882 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:54.882 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:25:54.882 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:54.882 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:54.882 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:54.882 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:25:54.882 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:25:54.883 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:25:54.883 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:25:54.884 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:54.884 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:25:54.884 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:54.937 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 26 2024-09-11T06:25:54.937 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:25:54.937 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:25:54.937 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:25:54.937 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=105, tag_tid=20, entry_tid=8785], [object_number=104, tag_tid=20, entry_tid=8784], [object_number=107, tag_tid=20, entry_tid=8783], [object_number=106, tag_tid=20, entry_tid=8782]]], state=connected] 2024-09-11T06:25:54.940 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:54.941 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:25:54.942 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:25:54.942 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:54.943 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:54.943 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:54.943 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:25:54.943 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:54.943 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:54.943 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:54.943 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:25:54.943 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:25:54.943 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:25:54.943 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:25:54.944 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:54.944 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:25:54.944 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:54.991 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 26 2024-09-11T06:25:54.991 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:25:54.992 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:25:54.992 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:25:54.992 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=105, tag_tid=20, entry_tid=8785], [object_number=104, tag_tid=20, entry_tid=8784], [object_number=107, tag_tid=20, entry_tid=8783], [object_number=106, tag_tid=20, entry_tid=8782]]], state=connected] 2024-09-11T06:25:54.995 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:54.997 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=105, tag_tid=20, entry_tid=8785' 2024-09-11T06:25:54.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=105, tag_tid=20, entry_tid=8785' 2024-09-11T06:25:54.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=105, tag_tid=20, entry_tid=8785' '!=' 'object_number=106, tag_tid=20, entry_tid=8698' 2024-09-11T06:25:54.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:25:54.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=105, tag_tid=20, entry_tid=8785' = 'object_number=106, tag_tid=20, entry_tid=8698' 2024-09-11T06:25:54.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=105, tag_tid=20, entry_tid=8785' 2024-09-11T06:25:54.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:25:54.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:25:54.999 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:55.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:25:55.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=105, tag_tid=20, entry_tid=8785' 2024-09-11T06:25:55.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:25:55.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:55.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:25:55.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:25:55.002 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:25:55.002 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:55.002 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:25:55.003 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=105, tag_tid=20, entry_tid=8785' 2024-09-11T06:25:55.003 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:25:55.003 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:55.004 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=8785 2024-09-11T06:25:55.004 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 8785 2024-09-11T06:25:55.005 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:25:55.005 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:25:55.005 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:25:55.205 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:25:55.205 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:25:55.205 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:25:55.206 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:25:55.206 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:25:55.206 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:25:55.206 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:25:55.206 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:25:55.206 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:25:55.206 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:25:55.206 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:25:55.206 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:25:55.206 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:25:55.206 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:25:55.206 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:25:55.207 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:25:55.207 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:25:55.211 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:25:55.211 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:25:55.211 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:25:55.211 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:25:55.211 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:25:55.211 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:25:55.211 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:25:55.211 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:25:55.211 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:25:55.211 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:25:55.211 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:25:55.211 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:25:55.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:25:55.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:25:55.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:25:55.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:25:55.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:25:55.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:25:55.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:25:55.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:25:55.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:25:55.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:25:55.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:25:55.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:25:55.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:25:55.213 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:25:55.213 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:25:55.213 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:25:55.213 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:25:55.213 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:25:55.213 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:25:55.213 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:25:55.235 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:25:55.236 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:25:55.236 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:25:55.236 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:25:55.367 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:25:55.368 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:25:55.368 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:55.368 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:55.368 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:55.369 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:25:55.369 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:55.369 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:55.369 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:55.369 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:25:55.369 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:25:55.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:25:55.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:25:55.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:55.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:25:55.372 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:55.429 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 26 2024-09-11T06:25:55.429 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:25:55.429 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:25:55.429 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:25:55.429 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=106, tag_tid=20, entry_tid=8882], [object_number=105, tag_tid=20, entry_tid=8881], [object_number=104, tag_tid=20, entry_tid=8880], [object_number=107, tag_tid=20, entry_tid=8879]]], state=connected] 2024-09-11T06:25:55.432 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:55.435 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:25:55.435 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:25:55.435 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:55.435 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:55.435 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:55.435 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:25:55.435 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:55.435 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:55.435 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:55.435 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:25:55.435 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:25:55.436 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:25:55.436 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:25:55.436 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:55.436 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:25:55.436 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:55.484 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 26 2024-09-11T06:25:55.484 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:25:55.484 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:25:55.484 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:25:55.484 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=106, tag_tid=20, entry_tid=8882], [object_number=105, tag_tid=20, entry_tid=8881], [object_number=104, tag_tid=20, entry_tid=8880], [object_number=107, tag_tid=20, entry_tid=8879]]], state=connected] 2024-09-11T06:25:55.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:55.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=106, tag_tid=20, entry_tid=8882' 2024-09-11T06:25:55.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=106, tag_tid=20, entry_tid=8882' 2024-09-11T06:25:55.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=106, tag_tid=20, entry_tid=8882' '!=' 'object_number=105, tag_tid=20, entry_tid=8785' 2024-09-11T06:25:55.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:25:55.490 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=106, tag_tid=20, entry_tid=8882' = 'object_number=105, tag_tid=20, entry_tid=8785' 2024-09-11T06:25:55.490 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=106, tag_tid=20, entry_tid=8882' 2024-09-11T06:25:55.490 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:25:55.490 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:25:55.490 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:55.491 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:25:55.491 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=106, tag_tid=20, entry_tid=8882' 2024-09-11T06:25:55.492 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:25:55.492 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:55.492 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:25:55.493 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:25:55.493 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:25:55.493 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:55.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:25:55.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=106, tag_tid=20, entry_tid=8882' 2024-09-11T06:25:55.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:25:55.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:55.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=8882 2024-09-11T06:25:55.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 8882 2024-09-11T06:25:55.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:25:55.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:25:55.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:25:55.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:25:55.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:25:55.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:25:55.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:25:55.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:25:55.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:25:55.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:25:55.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:25:55.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:25:55.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:25:55.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:25:55.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:25:55.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:25:55.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:25:55.700 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:25:55.700 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:25:55.700 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:25:55.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:25:55.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:25:55.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:25:55.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:25:55.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:25:55.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:25:55.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:25:55.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:25:55.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:25:55.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:25:55.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:25:55.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:25:55.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:25:55.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:25:55.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:25:55.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:25:55.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:25:55.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:25:55.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:25:55.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:25:55.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:25:55.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:25:55.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:25:55.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:25:55.706 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:25:55.706 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:25:55.706 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:25:55.706 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:25:55.706 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:25:55.706 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:25:55.706 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:25:55.706 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:25:55.730 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:25:55.730 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:25:55.730 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:25:55.730 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:25:55.862 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:25:55.863 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:25:55.863 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:55.863 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:55.863 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:55.863 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:25:55.863 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:55.863 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:55.863 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:55.863 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:25:55.864 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:25:55.864 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:25:55.864 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:25:55.865 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:55.868 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:25:55.868 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:55.921 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 26 2024-09-11T06:25:55.921 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:25:55.921 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:25:55.921 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:25:55.921 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=106, tag_tid=20, entry_tid=8978], [object_number=105, tag_tid=20, entry_tid=8977], [object_number=104, tag_tid=20, entry_tid=8976], [object_number=107, tag_tid=20, entry_tid=8975]]], state=connected] 2024-09-11T06:25:55.924 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:55.926 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:25:55.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:25:55.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:55.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:55.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:55.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:25:55.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:55.928 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:55.928 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:55.928 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:25:55.928 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:25:55.928 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:25:55.928 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:25:55.929 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:55.929 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:25:55.929 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:55.976 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 26 2024-09-11T06:25:55.977 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:25:55.977 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:25:55.977 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:25:55.977 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=106, tag_tid=20, entry_tid=8978], [object_number=105, tag_tid=20, entry_tid=8977], [object_number=104, tag_tid=20, entry_tid=8976], [object_number=107, tag_tid=20, entry_tid=8975]]], state=connected] 2024-09-11T06:25:55.980 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:55.982 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=106, tag_tid=20, entry_tid=8978' 2024-09-11T06:25:55.982 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=106, tag_tid=20, entry_tid=8978' 2024-09-11T06:25:55.982 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=106, tag_tid=20, entry_tid=8978' '!=' 'object_number=106, tag_tid=20, entry_tid=8882' 2024-09-11T06:25:55.982 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:25:55.982 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=106, tag_tid=20, entry_tid=8978' = 'object_number=106, tag_tid=20, entry_tid=8882' 2024-09-11T06:25:55.982 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=106, tag_tid=20, entry_tid=8978' 2024-09-11T06:25:55.983 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:25:55.983 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:25:55.983 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:55.983 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:25:55.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=106, tag_tid=20, entry_tid=8978' 2024-09-11T06:25:55.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:25:55.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:55.985 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:25:55.986 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:25:55.986 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:25:55.986 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:55.987 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:25:55.987 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=106, tag_tid=20, entry_tid=8978' 2024-09-11T06:25:55.988 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:25:55.988 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:55.989 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=8978 2024-09-11T06:25:55.989 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 8978 2024-09-11T06:25:55.989 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:25:55.989 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:25:55.989 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:25:56.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:25:56.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:25:56.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:25:56.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:25:56.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:25:56.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:25:56.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:25:56.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:25:56.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:25:56.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:25:56.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:25:56.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:25:56.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:25:56.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:25:56.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:25:56.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:25:56.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:25:56.196 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:25:56.196 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:25:56.197 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:25:56.197 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:25:56.197 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:25:56.197 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:25:56.197 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:25:56.197 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:25:56.197 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:25:56.197 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:25:56.197 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:25:56.197 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:25:56.197 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:25:56.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:25:56.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:25:56.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:25:56.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:25:56.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:25:56.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:25:56.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:25:56.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:25:56.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:25:56.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:25:56.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:25:56.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:25:56.199 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:25:56.199 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:25:56.199 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:25:56.199 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:25:56.199 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:25:56.199 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:25:56.199 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:25:56.222 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:25:56.222 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:25:56.223 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:25:56.223 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:25:56.350 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:25:56.351 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:25:56.351 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:56.351 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:56.351 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:56.351 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:25:56.351 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:56.351 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:56.352 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:56.352 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:25:56.352 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:25:56.352 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:25:56.352 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:25:56.353 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:56.354 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:25:56.354 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:56.406 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 26 2024-09-11T06:25:56.406 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:25:56.406 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:25:56.407 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:25:56.407 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=104, tag_tid=20, entry_tid=9064], [object_number=107, tag_tid=20, entry_tid=9063], [object_number=106, tag_tid=20, entry_tid=9062], [object_number=105, tag_tid=20, entry_tid=9061]]], state=connected] 2024-09-11T06:25:56.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:56.411 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:25:56.412 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:25:56.413 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:56.413 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:56.413 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:56.413 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:25:56.413 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:56.413 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:56.413 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:56.413 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:25:56.413 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:25:56.413 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:25:56.413 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:25:56.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:56.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:25:56.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:56.461 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 26 2024-09-11T06:25:56.461 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:25:56.462 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:25:56.462 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:25:56.462 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=104, tag_tid=20, entry_tid=9064], [object_number=107, tag_tid=20, entry_tid=9063], [object_number=106, tag_tid=20, entry_tid=9062], [object_number=105, tag_tid=20, entry_tid=9061]]], state=connected] 2024-09-11T06:25:56.465 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:56.466 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=104, tag_tid=20, entry_tid=9064' 2024-09-11T06:25:56.466 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=104, tag_tid=20, entry_tid=9064' 2024-09-11T06:25:56.466 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=104, tag_tid=20, entry_tid=9064' '!=' 'object_number=106, tag_tid=20, entry_tid=8978' 2024-09-11T06:25:56.467 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:25:56.467 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=104, tag_tid=20, entry_tid=9064' = 'object_number=106, tag_tid=20, entry_tid=8978' 2024-09-11T06:25:56.467 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=104, tag_tid=20, entry_tid=9064' 2024-09-11T06:25:56.467 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:25:56.467 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:25:56.467 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:56.468 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:25:56.469 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=104, tag_tid=20, entry_tid=9064' 2024-09-11T06:25:56.469 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:25:56.469 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:56.470 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:25:56.470 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:25:56.470 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:25:56.471 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:56.472 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:25:56.472 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=104, tag_tid=20, entry_tid=9064' 2024-09-11T06:25:56.472 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:25:56.472 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:56.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=9064 2024-09-11T06:25:56.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 9064 2024-09-11T06:25:56.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:25:56.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:25:56.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:25:56.675 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:25:56.675 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:25:56.675 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:25:56.676 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:25:56.676 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:25:56.676 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:25:56.676 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:25:56.676 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:25:56.676 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:25:56.676 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:25:56.676 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:25:56.676 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:25:56.676 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:25:56.676 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:25:56.676 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:25:56.677 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:25:56.677 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:25:56.680 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:25:56.680 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:25:56.680 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:25:56.680 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:25:56.680 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:25:56.680 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:25:56.680 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:25:56.680 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:25:56.680 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:25:56.680 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:25:56.680 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:25:56.680 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:25:56.680 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:25:56.681 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:25:56.681 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:25:56.681 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:25:56.682 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:25:56.682 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:25:56.682 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:25:56.682 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:25:56.682 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:25:56.682 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:25:56.682 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:25:56.683 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:25:56.683 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:25:56.683 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:25:56.683 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:25:56.683 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:25:56.683 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:25:56.683 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:25:56.683 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:25:56.683 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:25:56.707 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:25:56.707 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:25:56.707 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:25:56.707 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:25:56.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:25:56.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:25:56.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:56.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:56.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:56.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:25:56.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:56.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:56.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:56.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:25:56.835 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:25:56.835 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:25:56.835 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:25:56.837 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:56.837 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:25:56.837 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:56.884 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 26 2024-09-11T06:25:56.884 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:25:56.884 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:25:56.884 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:25:56.884 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=107, tag_tid=20, entry_tid=9147], [object_number=106, tag_tid=20, entry_tid=9146], [object_number=105, tag_tid=20, entry_tid=9145], [object_number=104, tag_tid=20, entry_tid=9144]]], state=connected] 2024-09-11T06:25:56.887 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:56.889 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:25:56.893 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:25:56.893 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:56.893 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:56.893 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:56.893 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:25:56.893 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:56.894 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:56.894 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:56.894 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:25:56.894 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:25:56.894 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:25:56.894 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:25:56.894 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:56.894 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:25:56.894 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:56.938 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 26 2024-09-11T06:25:56.939 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:25:56.939 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:25:56.939 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:25:56.939 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=107, tag_tid=20, entry_tid=9147], [object_number=106, tag_tid=20, entry_tid=9146], [object_number=105, tag_tid=20, entry_tid=9145], [object_number=104, tag_tid=20, entry_tid=9144]]], state=connected] 2024-09-11T06:25:56.942 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:56.943 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=107, tag_tid=20, entry_tid=9147' 2024-09-11T06:25:56.944 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=107, tag_tid=20, entry_tid=9147' 2024-09-11T06:25:56.944 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=107, tag_tid=20, entry_tid=9147' '!=' 'object_number=104, tag_tid=20, entry_tid=9064' 2024-09-11T06:25:56.944 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:25:56.944 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=107, tag_tid=20, entry_tid=9147' = 'object_number=104, tag_tid=20, entry_tid=9064' 2024-09-11T06:25:56.944 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=107, tag_tid=20, entry_tid=9147' 2024-09-11T06:25:56.944 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:25:56.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:25:56.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:56.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:25:56.946 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=107, tag_tid=20, entry_tid=9147' 2024-09-11T06:25:56.946 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:25:56.946 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:56.947 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:25:56.947 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:25:56.947 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:25:56.947 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:56.948 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:25:56.949 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=107, tag_tid=20, entry_tid=9147' 2024-09-11T06:25:56.949 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:25:56.949 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:56.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=9147 2024-09-11T06:25:56.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 9147 2024-09-11T06:25:56.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:25:56.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:25:56.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:25:57.152 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:25:57.152 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:25:57.152 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:25:57.152 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:25:57.152 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:25:57.153 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:25:57.153 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:25:57.153 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:25:57.153 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:25:57.153 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:25:57.153 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:25:57.153 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:25:57.153 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:25:57.153 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:25:57.153 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:25:57.154 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:25:57.154 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:25:57.157 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:25:57.157 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:25:57.157 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:25:57.157 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:25:57.157 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:25:57.157 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:25:57.157 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:25:57.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:25:57.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:25:57.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:25:57.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:25:57.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:25:57.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:25:57.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:25:57.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:25:57.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:25:57.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:25:57.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:25:57.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:25:57.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:25:57.159 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:25:57.159 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:25:57.159 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:25:57.159 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:25:57.159 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:25:57.159 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:25:57.159 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:25:57.159 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:25:57.159 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:25:57.159 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:25:57.159 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:25:57.159 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:25:57.182 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:25:57.183 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:25:57.183 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:25:57.183 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:25:57.311 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:25:57.312 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:25:57.312 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:57.312 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:57.312 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:57.312 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:25:57.312 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:57.312 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:57.312 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:57.312 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:25:57.313 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:25:57.313 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:25:57.313 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:25:57.315 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:57.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:25:57.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:57.363 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 26 2024-09-11T06:25:57.364 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:25:57.364 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:25:57.364 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:25:57.364 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=106, tag_tid=20, entry_tid=9222], [object_number=105, tag_tid=20, entry_tid=9221], [object_number=104, tag_tid=20, entry_tid=9220], [object_number=107, tag_tid=20, entry_tid=9219]]], state=connected] 2024-09-11T06:25:57.367 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:57.368 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:25:57.369 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:25:57.369 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:57.369 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:57.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:57.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:25:57.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:57.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:57.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:57.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:25:57.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:25:57.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:25:57.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:25:57.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:57.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:25:57.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:57.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 26 2024-09-11T06:25:57.418 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:25:57.418 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:25:57.418 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:25:57.418 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=106, tag_tid=20, entry_tid=9222], [object_number=105, tag_tid=20, entry_tid=9221], [object_number=104, tag_tid=20, entry_tid=9220], [object_number=107, tag_tid=20, entry_tid=9219]]], state=connected] 2024-09-11T06:25:57.421 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:57.422 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=106, tag_tid=20, entry_tid=9222' 2024-09-11T06:25:57.422 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=106, tag_tid=20, entry_tid=9222' 2024-09-11T06:25:57.422 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=106, tag_tid=20, entry_tid=9222' '!=' 'object_number=107, tag_tid=20, entry_tid=9147' 2024-09-11T06:25:57.423 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:25:57.423 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=106, tag_tid=20, entry_tid=9222' = 'object_number=107, tag_tid=20, entry_tid=9147' 2024-09-11T06:25:57.423 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=106, tag_tid=20, entry_tid=9222' 2024-09-11T06:25:57.423 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:25:57.423 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:25:57.424 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:57.424 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:25:57.425 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=106, tag_tid=20, entry_tid=9222' 2024-09-11T06:25:57.425 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:25:57.425 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:57.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:25:57.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:25:57.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:25:57.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:57.427 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:25:57.428 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=106, tag_tid=20, entry_tid=9222' 2024-09-11T06:25:57.428 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:25:57.428 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:57.429 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=9222 2024-09-11T06:25:57.429 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 9222 2024-09-11T06:25:57.429 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:25:57.429 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:25:57.430 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:25:57.631 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:25:57.632 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:25:57.632 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:25:57.632 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:25:57.632 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:25:57.632 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:25:57.632 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:25:57.632 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:25:57.632 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:25:57.632 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:25:57.632 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:25:57.633 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:25:57.633 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:25:57.633 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:25:57.633 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:25:57.633 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:25:57.633 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:25:57.636 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:25:57.636 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:25:57.636 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:25:57.636 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:25:57.636 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:25:57.637 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:25:57.637 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:25:57.637 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:25:57.637 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:25:57.637 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:25:57.637 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:25:57.637 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:25:57.637 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:25:57.637 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:25:57.637 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:25:57.637 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:25:57.637 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:25:57.637 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:25:57.638 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:25:57.638 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:25:57.638 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:25:57.638 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:25:57.638 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:25:57.638 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:25:57.638 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:25:57.638 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:25:57.638 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:25:57.638 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:25:57.638 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:25:57.638 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:25:57.638 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:25:57.638 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:25:57.661 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:25:57.661 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:25:57.662 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:25:57.662 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:25:57.788 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:25:57.790 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:25:57.790 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:57.791 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:57.791 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:57.791 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:25:57.791 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:57.791 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:57.791 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:57.791 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:25:57.791 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:25:57.791 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:25:57.791 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:25:57.792 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:57.792 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:25:57.792 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:57.845 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 26 2024-09-11T06:25:57.845 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:25:57.845 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:25:57.845 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:25:57.845 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=104, tag_tid=20, entry_tid=9312], [object_number=107, tag_tid=20, entry_tid=9311], [object_number=106, tag_tid=20, entry_tid=9310], [object_number=105, tag_tid=20, entry_tid=9309]]], state=connected] 2024-09-11T06:25:57.848 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:57.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:25:57.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:25:57.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:57.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:57.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:57.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:25:57.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:57.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:57.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:57.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:25:57.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:25:57.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:25:57.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:25:57.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:57.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:25:57.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:57.899 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 26 2024-09-11T06:25:57.899 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:25:57.899 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:25:57.899 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:25:57.900 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=104, tag_tid=20, entry_tid=9312], [object_number=107, tag_tid=20, entry_tid=9311], [object_number=106, tag_tid=20, entry_tid=9310], [object_number=105, tag_tid=20, entry_tid=9309]]], state=connected] 2024-09-11T06:25:57.902 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:57.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=104, tag_tid=20, entry_tid=9312' 2024-09-11T06:25:57.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=104, tag_tid=20, entry_tid=9312' 2024-09-11T06:25:57.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=104, tag_tid=20, entry_tid=9312' '!=' 'object_number=106, tag_tid=20, entry_tid=9222' 2024-09-11T06:25:57.905 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:25:57.905 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=104, tag_tid=20, entry_tid=9312' = 'object_number=106, tag_tid=20, entry_tid=9222' 2024-09-11T06:25:57.905 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=104, tag_tid=20, entry_tid=9312' 2024-09-11T06:25:57.905 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:25:57.905 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:25:57.906 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:57.906 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:25:57.907 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=104, tag_tid=20, entry_tid=9312' 2024-09-11T06:25:57.907 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:25:57.907 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:57.907 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:25:57.908 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:25:57.908 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:25:57.908 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:57.909 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:25:57.910 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=104, tag_tid=20, entry_tid=9312' 2024-09-11T06:25:57.910 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:25:57.910 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:57.911 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=9312 2024-09-11T06:25:57.911 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 9312 2024-09-11T06:25:57.911 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:25:57.912 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:25:57.912 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:25:58.113 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:25:58.113 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:25:58.113 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:25:58.114 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:25:58.114 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:25:58.114 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:25:58.114 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:25:58.114 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:25:58.114 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:25:58.114 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:25:58.114 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:25:58.114 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:25:58.114 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:25:58.114 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:25:58.115 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:25:58.115 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:25:58.115 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:25:58.118 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:25:58.118 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:25:58.118 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:25:58.118 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:25:58.118 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:25:58.120 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:25:58.120 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:25:58.120 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:25:58.120 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:25:58.120 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:25:58.120 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:25:58.120 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:25:58.120 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:25:58.120 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:25:58.120 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:25:58.120 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:25:58.121 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:25:58.121 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:25:58.121 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:25:58.121 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:25:58.121 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:25:58.121 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:25:58.121 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:25:58.121 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:25:58.121 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:25:58.121 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:25:58.121 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:25:58.121 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:25:58.121 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:25:58.121 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:25:58.122 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:25:58.122 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:25:58.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:25:58.143 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:25:58.143 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:25:58.143 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:25:58.277 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:25:58.278 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:25:58.279 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:58.279 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:58.279 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:58.279 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:25:58.279 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:58.279 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:58.279 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:58.279 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:25:58.279 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:25:58.280 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:25:58.280 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:25:58.281 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:58.282 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:25:58.282 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:58.335 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 26 2024-09-11T06:25:58.335 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:25:58.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:25:58.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:25:58.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=104, tag_tid=20, entry_tid=9408], [object_number=107, tag_tid=20, entry_tid=9407], [object_number=106, tag_tid=20, entry_tid=9406], [object_number=105, tag_tid=20, entry_tid=9405]]], state=connected] 2024-09-11T06:25:58.339 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:58.340 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:25:58.341 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:25:58.341 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:58.341 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:58.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:58.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:25:58.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:58.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:58.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:58.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:25:58.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:25:58.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:25:58.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:25:58.343 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:58.343 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:25:58.343 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:58.390 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 26 2024-09-11T06:25:58.391 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:25:58.391 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:25:58.391 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:25:58.391 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=104, tag_tid=20, entry_tid=9408], [object_number=107, tag_tid=20, entry_tid=9407], [object_number=106, tag_tid=20, entry_tid=9406], [object_number=105, tag_tid=20, entry_tid=9405]]], state=connected] 2024-09-11T06:25:58.394 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:58.396 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=104, tag_tid=20, entry_tid=9408' 2024-09-11T06:25:58.396 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=104, tag_tid=20, entry_tid=9408' 2024-09-11T06:25:58.396 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=104, tag_tid=20, entry_tid=9408' '!=' 'object_number=104, tag_tid=20, entry_tid=9312' 2024-09-11T06:25:58.397 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:25:58.397 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=104, tag_tid=20, entry_tid=9408' = 'object_number=104, tag_tid=20, entry_tid=9312' 2024-09-11T06:25:58.397 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=104, tag_tid=20, entry_tid=9408' 2024-09-11T06:25:58.397 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:25:58.397 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:25:58.397 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:58.397 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:25:58.398 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=104, tag_tid=20, entry_tid=9408' 2024-09-11T06:25:58.398 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:25:58.398 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:58.399 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:25:58.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:25:58.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:25:58.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:58.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:25:58.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=104, tag_tid=20, entry_tid=9408' 2024-09-11T06:25:58.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:25:58.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:58.403 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=9408 2024-09-11T06:25:58.403 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 9408 2024-09-11T06:25:58.403 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:25:58.403 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:25:58.403 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:25:58.604 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:25:58.605 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:25:58.605 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:25:58.605 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:25:58.605 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:25:58.605 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:25:58.605 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:25:58.605 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:25:58.605 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:25:58.605 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:25:58.605 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:25:58.605 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:25:58.605 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:25:58.606 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:25:58.606 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:25:58.606 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:25:58.606 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:25:58.609 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:25:58.609 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:25:58.609 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:25:58.609 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:25:58.609 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:25:58.609 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:25:58.609 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:25:58.609 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:25:58.609 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:25:58.609 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:25:58.609 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:25:58.609 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:25:58.610 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:25:58.610 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:25:58.610 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:25:58.610 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:25:58.610 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:25:58.610 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:25:58.610 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:25:58.610 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:25:58.610 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:25:58.610 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:25:58.610 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:25:58.610 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:25:58.610 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:25:58.610 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:25:58.611 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:25:58.611 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:25:58.611 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:25:58.611 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:25:58.611 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:25:58.611 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:25:58.637 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:25:58.637 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:25:58.637 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:25:58.637 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:25:58.768 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:25:58.769 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:25:58.769 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:58.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:58.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:58.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:25:58.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:58.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:58.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:58.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:25:58.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:25:58.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:25:58.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:25:58.772 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:58.772 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:25:58.772 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:58.819 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 26 2024-09-11T06:25:58.819 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:25:58.820 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:25:58.820 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:25:58.820 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=107, tag_tid=20, entry_tid=9499], [object_number=106, tag_tid=20, entry_tid=9498], [object_number=105, tag_tid=20, entry_tid=9497], [object_number=104, tag_tid=20, entry_tid=9496]]], state=connected] 2024-09-11T06:25:58.823 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:58.825 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:25:58.826 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:25:58.826 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:58.826 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:58.826 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:58.826 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:25:58.826 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:58.826 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:58.826 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:58.826 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:25:58.826 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:25:58.826 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:25:58.827 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:25:58.827 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:58.827 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:25:58.827 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:58.874 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 26 2024-09-11T06:25:58.874 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:25:58.874 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:25:58.874 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:25:58.874 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=107, tag_tid=20, entry_tid=9499], [object_number=106, tag_tid=20, entry_tid=9498], [object_number=105, tag_tid=20, entry_tid=9497], [object_number=104, tag_tid=20, entry_tid=9496]]], state=connected] 2024-09-11T06:25:58.877 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:58.879 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=107, tag_tid=20, entry_tid=9499' 2024-09-11T06:25:58.879 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=107, tag_tid=20, entry_tid=9499' 2024-09-11T06:25:58.879 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=107, tag_tid=20, entry_tid=9499' '!=' 'object_number=104, tag_tid=20, entry_tid=9408' 2024-09-11T06:25:58.880 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:25:58.880 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=107, tag_tid=20, entry_tid=9499' = 'object_number=104, tag_tid=20, entry_tid=9408' 2024-09-11T06:25:58.880 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=107, tag_tid=20, entry_tid=9499' 2024-09-11T06:25:58.880 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:25:58.880 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:25:58.880 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:58.880 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:25:58.881 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=107, tag_tid=20, entry_tid=9499' 2024-09-11T06:25:58.881 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:25:58.881 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:58.882 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:25:58.883 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:25:58.883 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:25:58.883 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:58.884 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:25:58.884 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=107, tag_tid=20, entry_tid=9499' 2024-09-11T06:25:58.885 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:25:58.885 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:58.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=9499 2024-09-11T06:25:58.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 9499 2024-09-11T06:25:58.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:25:58.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:25:58.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:25:59.087 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:25:59.088 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:25:59.088 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:25:59.088 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:25:59.088 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:25:59.088 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:25:59.088 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:25:59.088 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:25:59.088 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:25:59.088 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:25:59.088 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:25:59.088 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:25:59.088 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:25:59.089 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:25:59.089 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:25:59.089 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:25:59.089 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:25:59.093 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:25:59.093 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:25:59.093 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:25:59.093 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:25:59.093 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:25:59.093 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:25:59.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:25:59.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:25:59.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:25:59.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:25:59.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:25:59.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:25:59.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:25:59.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:25:59.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:25:59.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:25:59.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:25:59.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:25:59.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:25:59.095 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:25:59.095 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:25:59.095 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:25:59.095 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:25:59.095 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:25:59.095 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:25:59.095 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:25:59.095 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:25:59.095 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:25:59.095 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:25:59.095 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:25:59.095 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:25:59.095 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:25:59.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:25:59.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:25:59.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:25:59.120 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:25:59.248 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:25:59.250 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:25:59.250 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:59.250 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:59.250 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:59.250 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:25:59.250 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:59.250 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:59.250 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:59.250 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:25:59.250 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:25:59.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:25:59.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:25:59.252 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:59.252 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:25:59.252 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:59.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 26 2024-09-11T06:25:59.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:25:59.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:25:59.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:25:59.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=106, tag_tid=20, entry_tid=9582], [object_number=105, tag_tid=20, entry_tid=9581], [object_number=104, tag_tid=20, entry_tid=9580], [object_number=107, tag_tid=20, entry_tid=9579]]], state=connected] 2024-09-11T06:25:59.313 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:59.315 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:25:59.315 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:25:59.315 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:59.315 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:59.315 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:59.315 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:25:59.315 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:59.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:59.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:59.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:25:59.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:25:59.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:25:59.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:25:59.317 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:59.317 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:25:59.317 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:59.366 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 26 2024-09-11T06:25:59.366 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:25:59.366 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:25:59.366 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:25:59.366 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=106, tag_tid=20, entry_tid=9582], [object_number=105, tag_tid=20, entry_tid=9581], [object_number=104, tag_tid=20, entry_tid=9580], [object_number=107, tag_tid=20, entry_tid=9579]]], state=connected] 2024-09-11T06:25:59.369 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:59.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=106, tag_tid=20, entry_tid=9582' 2024-09-11T06:25:59.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=106, tag_tid=20, entry_tid=9582' 2024-09-11T06:25:59.372 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=106, tag_tid=20, entry_tid=9582' '!=' 'object_number=107, tag_tid=20, entry_tid=9499' 2024-09-11T06:25:59.372 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:25:59.372 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=106, tag_tid=20, entry_tid=9582' = 'object_number=107, tag_tid=20, entry_tid=9499' 2024-09-11T06:25:59.372 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=106, tag_tid=20, entry_tid=9582' 2024-09-11T06:25:59.372 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:25:59.372 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:25:59.372 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:59.372 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:25:59.373 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=106, tag_tid=20, entry_tid=9582' 2024-09-11T06:25:59.373 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:25:59.373 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:59.374 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:25:59.375 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:25:59.375 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:25:59.375 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:59.376 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:25:59.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=106, tag_tid=20, entry_tid=9582' 2024-09-11T06:25:59.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:25:59.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:59.378 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=9582 2024-09-11T06:25:59.378 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 9582 2024-09-11T06:25:59.378 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:25:59.378 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:25:59.378 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:25:59.580 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:25:59.580 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:25:59.580 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:25:59.580 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:25:59.580 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:25:59.580 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:25:59.581 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:25:59.581 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:25:59.581 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:25:59.581 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:25:59.581 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:25:59.581 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:25:59.581 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:25:59.581 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:25:59.581 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:25:59.582 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:25:59.582 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:25:59.585 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:25:59.585 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:25:59.585 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:25:59.585 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:25:59.585 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:25:59.585 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:25:59.585 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:25:59.585 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:25:59.585 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:25:59.586 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:25:59.586 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:25:59.586 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:25:59.586 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:25:59.586 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:25:59.586 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:25:59.586 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:25:59.586 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:25:59.586 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:25:59.586 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:25:59.586 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:25:59.586 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:25:59.586 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:25:59.587 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:25:59.587 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:25:59.587 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:25:59.587 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:25:59.587 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:25:59.587 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:25:59.587 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:25:59.587 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:25:59.587 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:25:59.587 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:25:59.610 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:25:59.610 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:25:59.610 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:25:59.610 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:25:59.735 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:25:59.736 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:25:59.736 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:59.736 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:59.736 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:59.737 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:25:59.737 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:59.737 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:59.737 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:59.737 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:25:59.737 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:25:59.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:25:59.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:25:59.739 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:59.739 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:25:59.739 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:59.791 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 26 2024-09-11T06:25:59.791 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:25:59.791 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:25:59.791 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:25:59.791 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=104, tag_tid=20, entry_tid=9668], [object_number=107, tag_tid=20, entry_tid=9667], [object_number=106, tag_tid=20, entry_tid=9666], [object_number=105, tag_tid=20, entry_tid=9665]]], state=connected] 2024-09-11T06:25:59.794 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:59.795 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:25:59.796 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:25:59.797 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:59.797 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:59.797 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:59.797 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:25:59.797 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:25:59.797 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:25:59.797 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:25:59.797 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:25:59.797 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:25:59.797 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:25:59.797 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:25:59.798 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:59.798 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:25:59.798 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:59.846 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 26 2024-09-11T06:25:59.846 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:25:59.846 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:25:59.846 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:25:59.846 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=104, tag_tid=20, entry_tid=9668], [object_number=107, tag_tid=20, entry_tid=9667], [object_number=106, tag_tid=20, entry_tid=9666], [object_number=105, tag_tid=20, entry_tid=9665]]], state=connected] 2024-09-11T06:25:59.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:25:59.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=104, tag_tid=20, entry_tid=9668' 2024-09-11T06:25:59.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=104, tag_tid=20, entry_tid=9668' 2024-09-11T06:25:59.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=104, tag_tid=20, entry_tid=9668' '!=' 'object_number=106, tag_tid=20, entry_tid=9582' 2024-09-11T06:25:59.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:25:59.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=104, tag_tid=20, entry_tid=9668' = 'object_number=106, tag_tid=20, entry_tid=9582' 2024-09-11T06:25:59.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=104, tag_tid=20, entry_tid=9668' 2024-09-11T06:25:59.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:25:59.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:25:59.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:59.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:25:59.854 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=104, tag_tid=20, entry_tid=9668' 2024-09-11T06:25:59.854 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:25:59.854 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:59.854 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:25:59.855 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:25:59.855 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:25:59.855 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:59.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:25:59.857 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=104, tag_tid=20, entry_tid=9668' 2024-09-11T06:25:59.857 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:25:59.857 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:25:59.858 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=9668 2024-09-11T06:25:59.858 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 9668 2024-09-11T06:25:59.859 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:25:59.859 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:25:59.859 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:26:00.060 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:26:00.060 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:00.060 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:26:00.060 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:26:00.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:26:00.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:26:00.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:26:00.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:26:00.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:26:00.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:00.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:00.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:26:00.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:26:00.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:26:00.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:26:00.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:26:00.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:26:00.064 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:26:00.064 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:26:00.064 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:26:00.065 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:00.065 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:26:00.065 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:26:00.065 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:26:00.065 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:26:00.065 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:26:00.065 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:26:00.065 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:26:00.065 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:00.065 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:26:00.065 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:00.065 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:26:00.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:00.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:26:00.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:26:00.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:26:00.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:26:00.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:26:00.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:26:00.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:26:00.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:26:00.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:00.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:00.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:00.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:26:00.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:26:00.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:26:00.067 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:26:00.067 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:26:00.091 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:00.092 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:00.092 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:00.092 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:26:00.227 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:26:00.228 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:26:00.229 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:00.229 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:00.229 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:00.229 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:26:00.229 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:00.229 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:00.229 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:00.229 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:26:00.230 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:00.230 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:00.230 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:00.232 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:00.232 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:00.232 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:00.278 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 26 2024-09-11T06:26:00.278 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:00.278 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:00.279 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:00.279 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=106, tag_tid=20, entry_tid=9766], [object_number=105, tag_tid=20, entry_tid=9765], [object_number=104, tag_tid=20, entry_tid=9764], [object_number=107, tag_tid=20, entry_tid=9763]]], state=connected] 2024-09-11T06:26:00.282 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:00.283 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:00.284 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:26:00.284 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:00.284 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:00.284 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:00.285 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:26:00.285 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:00.285 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:00.285 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:00.285 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:26:00.285 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:00.285 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:00.285 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:00.286 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:00.286 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:00.286 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:00.335 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 26 2024-09-11T06:26:00.335 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:00.335 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:00.335 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:00.335 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=106, tag_tid=20, entry_tid=9766], [object_number=105, tag_tid=20, entry_tid=9765], [object_number=104, tag_tid=20, entry_tid=9764], [object_number=107, tag_tid=20, entry_tid=9763]]], state=connected] 2024-09-11T06:26:00.339 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:00.341 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=106, tag_tid=20, entry_tid=9766' 2024-09-11T06:26:00.341 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=106, tag_tid=20, entry_tid=9766' 2024-09-11T06:26:00.341 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=106, tag_tid=20, entry_tid=9766' '!=' 'object_number=104, tag_tid=20, entry_tid=9668' 2024-09-11T06:26:00.341 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:26:00.341 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=106, tag_tid=20, entry_tid=9766' = 'object_number=104, tag_tid=20, entry_tid=9668' 2024-09-11T06:26:00.341 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=106, tag_tid=20, entry_tid=9766' 2024-09-11T06:26:00.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:00.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:00.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:00.343 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:26:00.343 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=106, tag_tid=20, entry_tid=9766' 2024-09-11T06:26:00.343 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:00.343 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:00.344 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:26:00.344 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:00.344 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:00.344 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:00.345 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:26:00.345 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=106, tag_tid=20, entry_tid=9766' 2024-09-11T06:26:00.346 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:00.346 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:00.347 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=9766 2024-09-11T06:26:00.347 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 9766 2024-09-11T06:26:00.347 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:26:00.347 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:26:00.347 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:26:00.549 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:26:00.549 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:00.549 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:26:00.549 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:26:00.549 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:26:00.549 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:26:00.549 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:26:00.549 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:26:00.550 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:26:00.550 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:00.550 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:00.550 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:26:00.550 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:26:00.550 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:26:00.550 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:26:00.550 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:26:00.551 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:26:00.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:26:00.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:26:00.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:26:00.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:00.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:26:00.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:26:00.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:26:00.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:26:00.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:26:00.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:26:00.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:26:00.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:00.555 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:26:00.555 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:00.555 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:26:00.555 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:00.555 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:26:00.555 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:26:00.555 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:26:00.555 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:26:00.555 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:26:00.555 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:26:00.555 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:26:00.555 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:26:00.555 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:00.556 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:00.556 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:00.556 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:26:00.556 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:26:00.556 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:26:00.556 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:26:00.556 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:26:00.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:00.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:00.580 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:00.580 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:26:00.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:26:00.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:26:00.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:00.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:00.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:00.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:26:00.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:00.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:00.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:00.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:26:00.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:00.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:00.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:00.716 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:00.716 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:00.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:00.774 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 26 2024-09-11T06:26:00.774 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:00.774 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:00.774 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:00.774 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=105, tag_tid=20, entry_tid=9861], [object_number=104, tag_tid=20, entry_tid=9860], [object_number=107, tag_tid=20, entry_tid=9859], [object_number=106, tag_tid=20, entry_tid=9858]]], state=connected] 2024-09-11T06:26:00.777 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:00.779 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:00.780 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:26:00.780 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:00.780 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:00.780 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:00.780 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:26:00.780 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:00.780 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:00.780 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:00.780 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:26:00.780 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:00.780 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:00.780 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:00.781 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:00.781 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:00.781 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:00.829 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 26 2024-09-11T06:26:00.829 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:00.829 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:00.829 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:00.829 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=105, tag_tid=20, entry_tid=9861], [object_number=104, tag_tid=20, entry_tid=9860], [object_number=107, tag_tid=20, entry_tid=9859], [object_number=106, tag_tid=20, entry_tid=9858]]], state=connected] 2024-09-11T06:26:00.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:00.835 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=105, tag_tid=20, entry_tid=9861' 2024-09-11T06:26:00.836 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=105, tag_tid=20, entry_tid=9861' 2024-09-11T06:26:00.836 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=105, tag_tid=20, entry_tid=9861' '!=' 'object_number=106, tag_tid=20, entry_tid=9766' 2024-09-11T06:26:00.836 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:26:00.836 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=105, tag_tid=20, entry_tid=9861' = 'object_number=106, tag_tid=20, entry_tid=9766' 2024-09-11T06:26:00.836 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=105, tag_tid=20, entry_tid=9861' 2024-09-11T06:26:00.836 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:00.836 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:00.836 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:00.837 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:26:00.837 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=105, tag_tid=20, entry_tid=9861' 2024-09-11T06:26:00.837 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:00.837 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:00.838 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:26:00.839 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:00.839 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:00.839 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:00.840 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:26:00.841 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=105, tag_tid=20, entry_tid=9861' 2024-09-11T06:26:00.841 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:00.841 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:00.842 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=9861 2024-09-11T06:26:00.842 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 9861 2024-09-11T06:26:00.842 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:26:00.843 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:26:00.843 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:26:01.044 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:26:01.044 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:01.044 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:26:01.044 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:26:01.044 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:26:01.044 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:26:01.044 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:26:01.045 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:26:01.045 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:26:01.045 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:01.045 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:01.045 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:26:01.045 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:26:01.045 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:26:01.045 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:26:01.045 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:26:01.046 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:26:01.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:26:01.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:26:01.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:26:01.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:01.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:26:01.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:26:01.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:26:01.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:26:01.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:26:01.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:26:01.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:26:01.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:01.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:26:01.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:01.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:26:01.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:01.050 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:26:01.050 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:26:01.050 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:26:01.050 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:26:01.050 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:26:01.051 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:26:01.051 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:26:01.051 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:26:01.051 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:01.051 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:01.051 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:01.051 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:26:01.051 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:26:01.051 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:26:01.051 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:26:01.051 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:26:01.077 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:01.077 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:01.077 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:01.078 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:26:01.206 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:26:01.207 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:26:01.207 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:01.207 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:01.207 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:01.207 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:26:01.207 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:01.207 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:01.207 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:01.207 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:26:01.208 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:01.208 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:01.209 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:01.211 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:01.211 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:01.211 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:01.259 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 26 2024-09-11T06:26:01.260 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:01.260 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:01.260 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:01.260 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=106, tag_tid=20, entry_tid=9950], [object_number=105, tag_tid=20, entry_tid=9949], [object_number=104, tag_tid=20, entry_tid=9948], [object_number=107, tag_tid=20, entry_tid=9947]]], state=connected] 2024-09-11T06:26:01.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:01.264 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:01.265 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:26:01.265 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:01.265 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:01.266 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:01.266 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:26:01.266 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:01.266 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:01.266 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:01.266 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:26:01.266 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:01.266 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:01.266 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:01.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:01.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:01.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:01.313 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 26 2024-09-11T06:26:01.313 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:01.314 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:01.314 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:01.314 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=106, tag_tid=20, entry_tid=9950], [object_number=105, tag_tid=20, entry_tid=9949], [object_number=104, tag_tid=20, entry_tid=9948], [object_number=107, tag_tid=20, entry_tid=9947]]], state=connected] 2024-09-11T06:26:01.317 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:01.318 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=106, tag_tid=20, entry_tid=9950' 2024-09-11T06:26:01.318 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=106, tag_tid=20, entry_tid=9950' 2024-09-11T06:26:01.319 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=106, tag_tid=20, entry_tid=9950' '!=' 'object_number=105, tag_tid=20, entry_tid=9861' 2024-09-11T06:26:01.319 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:26:01.319 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=106, tag_tid=20, entry_tid=9950' = 'object_number=105, tag_tid=20, entry_tid=9861' 2024-09-11T06:26:01.319 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=106, tag_tid=20, entry_tid=9950' 2024-09-11T06:26:01.319 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:01.319 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:01.320 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:01.320 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:26:01.321 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=106, tag_tid=20, entry_tid=9950' 2024-09-11T06:26:01.321 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:01.321 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:01.322 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:26:01.322 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:01.322 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:01.323 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:01.323 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:26:01.324 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=106, tag_tid=20, entry_tid=9950' 2024-09-11T06:26:01.324 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:01.324 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:01.325 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=9950 2024-09-11T06:26:01.325 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 9950 2024-09-11T06:26:01.326 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:26:01.326 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:26:01.326 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:26:01.527 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:26:01.528 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:01.528 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:26:01.528 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:26:01.528 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:26:01.528 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:26:01.528 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:26:01.528 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:26:01.528 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:26:01.529 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:01.529 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:01.529 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:26:01.529 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:26:01.529 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:26:01.529 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:26:01.530 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:26:01.530 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:26:01.532 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:26:01.533 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:26:01.533 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:26:01.533 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:01.533 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:26:01.533 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:26:01.533 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:26:01.533 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:26:01.533 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:26:01.533 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:26:01.533 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:26:01.533 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:01.534 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:26:01.534 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:01.534 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:26:01.534 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:01.534 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:26:01.534 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:26:01.534 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:26:01.534 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:26:01.534 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:26:01.534 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:26:01.534 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:26:01.534 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:26:01.535 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:01.535 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:01.535 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:01.535 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:26:01.535 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:26:01.535 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:26:01.535 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:26:01.535 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:26:01.559 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:01.560 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:01.560 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:01.560 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:26:01.694 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:26:01.695 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:26:01.695 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:01.695 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:01.695 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:01.695 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:26:01.695 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:01.696 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:01.696 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:01.696 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:26:01.696 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:01.696 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:01.696 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:01.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:01.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:01.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:01.753 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 26 2024-09-11T06:26:01.753 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:01.753 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:01.753 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:01.753 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=106, tag_tid=20, entry_tid=10042], [object_number=105, tag_tid=20, entry_tid=10041], [object_number=104, tag_tid=20, entry_tid=10040], [object_number=107, tag_tid=20, entry_tid=10039]]], state=connected] 2024-09-11T06:26:01.756 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:01.758 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:01.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:26:01.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:01.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:01.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:01.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:26:01.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:01.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:01.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:01.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:26:01.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:01.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:01.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:01.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:01.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:01.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:01.808 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 26 2024-09-11T06:26:01.808 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:01.808 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:01.808 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:01.808 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=106, tag_tid=20, entry_tid=10042], [object_number=105, tag_tid=20, entry_tid=10041], [object_number=104, tag_tid=20, entry_tid=10040], [object_number=107, tag_tid=20, entry_tid=10039]]], state=connected] 2024-09-11T06:26:01.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:01.813 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=106, tag_tid=20, entry_tid=10042' 2024-09-11T06:26:01.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=106, tag_tid=20, entry_tid=10042' 2024-09-11T06:26:01.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=106, tag_tid=20, entry_tid=10042' '!=' 'object_number=106, tag_tid=20, entry_tid=9950' 2024-09-11T06:26:01.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:26:01.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=106, tag_tid=20, entry_tid=10042' = 'object_number=106, tag_tid=20, entry_tid=9950' 2024-09-11T06:26:01.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=106, tag_tid=20, entry_tid=10042' 2024-09-11T06:26:01.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:01.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:01.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:01.815 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:26:01.815 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=106, tag_tid=20, entry_tid=10042' 2024-09-11T06:26:01.815 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:01.816 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:01.816 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:26:01.817 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:01.817 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:01.817 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:01.818 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:26:01.819 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=106, tag_tid=20, entry_tid=10042' 2024-09-11T06:26:01.819 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:01.819 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:01.820 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=10042 2024-09-11T06:26:01.820 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 10042 2024-09-11T06:26:01.820 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:26:01.820 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:26:01.820 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:26:02.022 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:26:02.022 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:02.022 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:26:02.022 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:26:02.023 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:26:02.023 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:26:02.023 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:26:02.023 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:26:02.023 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:26:02.023 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:02.023 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:02.023 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:26:02.023 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:26:02.023 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:26:02.023 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:26:02.024 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:26:02.024 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:26:02.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:26:02.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:26:02.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:26:02.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:02.028 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:26:02.028 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:26:02.028 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:26:02.028 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:26:02.028 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:26:02.028 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:26:02.028 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:26:02.028 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:02.028 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:26:02.028 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:02.028 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:26:02.028 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:02.028 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:26:02.029 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:26:02.029 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:26:02.029 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:26:02.029 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:26:02.029 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:26:02.029 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:26:02.029 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:26:02.029 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:02.029 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:02.029 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:02.029 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:26:02.029 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:26:02.030 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:26:02.030 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:26:02.030 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:26:02.052 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:02.053 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:02.053 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:02.053 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:26:02.186 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:26:02.188 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:26:02.188 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:02.188 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:02.188 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:02.188 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:26:02.188 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:02.188 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:02.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:02.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:26:02.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:02.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:02.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:02.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:02.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:02.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:02.248 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 26 2024-09-11T06:26:02.248 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:02.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:02.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:02.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=104, tag_tid=20, entry_tid=10128], [object_number=107, tag_tid=20, entry_tid=10127], [object_number=106, tag_tid=20, entry_tid=10126], [object_number=105, tag_tid=20, entry_tid=10125]]], state=connected] 2024-09-11T06:26:02.252 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:02.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:02.254 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:26:02.254 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:02.254 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:02.254 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:02.255 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:26:02.255 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:02.255 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:02.255 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:02.255 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:26:02.255 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:02.255 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:02.255 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:02.256 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:02.256 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:02.256 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:02.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 26 2024-09-11T06:26:02.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:02.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:02.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:02.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=104, tag_tid=20, entry_tid=10128], [object_number=107, tag_tid=20, entry_tid=10127], [object_number=106, tag_tid=20, entry_tid=10126], [object_number=105, tag_tid=20, entry_tid=10125]]], state=connected] 2024-09-11T06:26:02.307 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:02.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=104, tag_tid=20, entry_tid=10128' 2024-09-11T06:26:02.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=104, tag_tid=20, entry_tid=10128' 2024-09-11T06:26:02.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=104, tag_tid=20, entry_tid=10128' '!=' 'object_number=106, tag_tid=20, entry_tid=10042' 2024-09-11T06:26:02.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:26:02.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=104, tag_tid=20, entry_tid=10128' = 'object_number=106, tag_tid=20, entry_tid=10042' 2024-09-11T06:26:02.310 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=104, tag_tid=20, entry_tid=10128' 2024-09-11T06:26:02.310 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:02.310 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:02.310 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:02.310 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:26:02.311 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=104, tag_tid=20, entry_tid=10128' 2024-09-11T06:26:02.311 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:02.312 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:02.313 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:26:02.313 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:02.313 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:02.313 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:02.313 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:26:02.314 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=104, tag_tid=20, entry_tid=10128' 2024-09-11T06:26:02.315 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:02.315 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:02.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=10128 2024-09-11T06:26:02.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 10128 2024-09-11T06:26:02.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:26:02.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:26:02.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:26:02.518 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:26:02.518 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:02.518 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:26:02.518 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:26:02.518 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:26:02.518 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:26:02.518 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:26:02.518 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:26:02.518 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:26:02.518 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:02.519 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:02.519 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:26:02.519 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:26:02.519 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:26:02.519 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:26:02.519 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:26:02.520 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:26:02.522 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:26:02.522 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:26:02.522 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:26:02.522 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:02.522 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:26:02.522 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:26:02.522 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:26:02.523 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:26:02.523 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:26:02.523 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:26:02.523 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:26:02.523 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:02.523 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:26:02.523 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:02.523 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:26:02.523 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:02.524 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:26:02.524 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:26:02.524 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:26:02.524 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:26:02.524 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:26:02.525 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:26:02.525 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:26:02.525 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:26:02.525 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:02.525 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:02.525 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:02.525 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:26:02.525 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:26:02.525 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:26:02.525 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:26:02.525 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:26:02.549 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:02.549 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:02.549 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:02.549 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:26:02.685 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:26:02.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:26:02.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:02.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:02.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:02.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:26:02.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:02.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:02.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:02.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:26:02.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:02.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:02.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:02.690 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:02.690 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:02.690 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:02.736 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 26 2024-09-11T06:26:02.736 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:02.737 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:02.737 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:02.737 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=105, tag_tid=20, entry_tid=10221], [object_number=104, tag_tid=20, entry_tid=10220], [object_number=107, tag_tid=20, entry_tid=10219], [object_number=106, tag_tid=20, entry_tid=10218]]], state=connected] 2024-09-11T06:26:02.740 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:02.741 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:02.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:26:02.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:02.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:02.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:02.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:26:02.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:02.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:02.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:02.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:26:02.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:02.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:02.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:02.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:02.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:02.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:02.791 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 26 2024-09-11T06:26:02.791 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:02.791 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:02.791 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:02.791 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=105, tag_tid=20, entry_tid=10221], [object_number=104, tag_tid=20, entry_tid=10220], [object_number=107, tag_tid=20, entry_tid=10219], [object_number=106, tag_tid=20, entry_tid=10218]]], state=connected] 2024-09-11T06:26:02.794 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:02.797 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=105, tag_tid=20, entry_tid=10221' 2024-09-11T06:26:02.797 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=105, tag_tid=20, entry_tid=10221' 2024-09-11T06:26:02.797 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=105, tag_tid=20, entry_tid=10221' '!=' 'object_number=104, tag_tid=20, entry_tid=10128' 2024-09-11T06:26:02.797 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:26:02.797 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=105, tag_tid=20, entry_tid=10221' = 'object_number=104, tag_tid=20, entry_tid=10128' 2024-09-11T06:26:02.797 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=105, tag_tid=20, entry_tid=10221' 2024-09-11T06:26:02.797 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:02.797 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:02.797 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:02.798 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:26:02.798 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=105, tag_tid=20, entry_tid=10221' 2024-09-11T06:26:02.799 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:02.799 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:02.799 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:26:02.800 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:02.800 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:02.800 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:02.801 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:26:02.802 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=105, tag_tid=20, entry_tid=10221' 2024-09-11T06:26:02.802 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:02.802 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:02.803 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=10221 2024-09-11T06:26:02.803 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 10221 2024-09-11T06:26:02.803 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:26:02.803 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:26:02.803 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:26:03.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:26:03.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:03.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:26:03.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:26:03.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:26:03.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:26:03.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:26:03.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:26:03.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:26:03.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:03.007 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:03.007 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:26:03.007 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:26:03.007 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:26:03.007 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:26:03.007 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:26:03.011 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:26:03.011 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:26:03.011 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:26:03.011 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:26:03.011 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:03.011 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:26:03.011 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:26:03.011 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:26:03.011 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:26:03.011 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:26:03.012 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:26:03.012 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:26:03.012 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:03.012 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:26:03.012 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:03.012 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:26:03.012 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:03.012 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:26:03.012 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:26:03.012 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:26:03.012 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:26:03.012 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:26:03.012 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:26:03.013 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:26:03.013 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:26:03.013 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:03.013 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:03.013 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:03.013 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:26:03.013 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:26:03.013 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:26:03.013 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:26:03.013 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:26:03.032 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:03.032 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:03.032 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:03.032 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:26:03.162 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:26:03.163 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:26:03.163 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:03.163 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:03.163 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:03.163 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:26:03.163 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:03.164 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:03.164 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:03.164 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:26:03.164 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:03.164 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:03.164 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:03.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:03.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:03.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:03.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 26 2024-09-11T06:26:03.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:03.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:03.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:03.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=105, tag_tid=20, entry_tid=10325], [object_number=104, tag_tid=20, entry_tid=10324], [object_number=107, tag_tid=20, entry_tid=10323], [object_number=106, tag_tid=20, entry_tid=10322]]], state=connected] 2024-09-11T06:26:03.216 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:03.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:03.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:26:03.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:03.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:03.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:03.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:26:03.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:03.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:03.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:03.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:26:03.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:03.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:03.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:03.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:03.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:03.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:03.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 26 2024-09-11T06:26:03.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:03.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:03.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:03.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=105, tag_tid=20, entry_tid=10325], [object_number=104, tag_tid=20, entry_tid=10324], [object_number=107, tag_tid=20, entry_tid=10323], [object_number=106, tag_tid=20, entry_tid=10322]]], state=connected] 2024-09-11T06:26:03.270 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:03.272 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=105, tag_tid=20, entry_tid=10325' 2024-09-11T06:26:03.272 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=105, tag_tid=20, entry_tid=10325' 2024-09-11T06:26:03.272 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=105, tag_tid=20, entry_tid=10325' '!=' 'object_number=105, tag_tid=20, entry_tid=10221' 2024-09-11T06:26:03.272 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:26:03.272 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=105, tag_tid=20, entry_tid=10325' = 'object_number=105, tag_tid=20, entry_tid=10221' 2024-09-11T06:26:03.272 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=105, tag_tid=20, entry_tid=10325' 2024-09-11T06:26:03.273 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:03.273 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:03.273 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:03.273 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:26:03.274 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=105, tag_tid=20, entry_tid=10325' 2024-09-11T06:26:03.274 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:03.274 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:03.275 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:26:03.276 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:03.276 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:03.276 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:03.277 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:26:03.278 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=105, tag_tid=20, entry_tid=10325' 2024-09-11T06:26:03.278 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:03.278 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:03.279 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=10325 2024-09-11T06:26:03.279 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 10325 2024-09-11T06:26:03.279 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:26:03.280 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:26:03.280 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:26:03.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:26:03.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:03.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:26:03.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:26:03.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:26:03.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:26:03.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:26:03.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:26:03.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:26:03.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:03.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:03.483 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:26:03.483 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:26:03.483 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:26:03.483 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:26:03.485 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:26:03.485 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:26:03.485 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:26:03.485 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:26:03.486 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:26:03.486 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:03.486 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:26:03.486 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:26:03.486 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:26:03.486 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:26:03.486 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:26:03.486 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:26:03.486 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:26:03.486 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:03.486 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:26:03.486 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:03.486 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:26:03.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:03.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:26:03.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:26:03.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:26:03.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:26:03.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:26:03.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:26:03.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:26:03.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:26:03.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:03.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:03.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:03.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:26:03.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:26:03.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:26:03.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:26:03.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:26:03.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:03.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:03.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:03.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:26:03.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:26:03.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:26:03.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:03.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:03.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:03.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:26:03.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:03.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:03.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:03.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:26:03.649 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:03.649 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:03.649 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:03.650 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:03.650 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:03.650 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:03.708 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 26 2024-09-11T06:26:03.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:03.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:03.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:03.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=105, tag_tid=20, entry_tid=10425], [object_number=104, tag_tid=20, entry_tid=10424], [object_number=107, tag_tid=20, entry_tid=10423], [object_number=106, tag_tid=20, entry_tid=10422]]], state=connected] 2024-09-11T06:26:03.712 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:03.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:03.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:26:03.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:03.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:03.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:03.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:26:03.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:03.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:03.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:03.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:26:03.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:03.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:03.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:03.716 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:03.716 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:03.716 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:03.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 26 2024-09-11T06:26:03.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:03.764 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:03.764 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:03.764 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=105, tag_tid=20, entry_tid=10425], [object_number=104, tag_tid=20, entry_tid=10424], [object_number=107, tag_tid=20, entry_tid=10423], [object_number=106, tag_tid=20, entry_tid=10422]]], state=connected] 2024-09-11T06:26:03.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:03.769 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=105, tag_tid=20, entry_tid=10425' 2024-09-11T06:26:03.769 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=105, tag_tid=20, entry_tid=10425' 2024-09-11T06:26:03.769 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=105, tag_tid=20, entry_tid=10425' '!=' 'object_number=105, tag_tid=20, entry_tid=10325' 2024-09-11T06:26:03.769 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:26:03.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=105, tag_tid=20, entry_tid=10425' = 'object_number=105, tag_tid=20, entry_tid=10325' 2024-09-11T06:26:03.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=105, tag_tid=20, entry_tid=10425' 2024-09-11T06:26:03.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:03.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:03.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:03.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:26:03.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=105, tag_tid=20, entry_tid=10425' 2024-09-11T06:26:03.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:03.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:03.772 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:26:03.773 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:03.773 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:03.773 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:03.774 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:26:03.774 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=105, tag_tid=20, entry_tid=10425' 2024-09-11T06:26:03.774 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:03.774 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:03.775 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=10425 2024-09-11T06:26:03.776 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 10425 2024-09-11T06:26:03.776 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:26:03.776 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:26:03.776 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:26:03.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:26:03.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:03.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:26:03.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:26:03.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:26:03.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:26:03.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:26:03.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:26:03.979 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:26:03.979 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:03.979 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:03.979 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:26:03.979 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:26:03.979 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:26:03.979 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:26:03.979 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:26:03.981 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:26:03.981 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:26:03.982 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:26:03.982 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:26:03.982 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:03.982 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:26:03.982 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:26:03.982 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:26:03.982 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:26:03.982 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:26:03.982 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:26:03.982 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:26:03.982 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:03.982 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:26:03.982 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:03.983 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:26:03.983 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:03.983 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:26:03.983 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:26:03.983 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:26:03.983 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:26:03.983 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:26:03.983 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:26:03.983 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:26:03.983 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:26:03.983 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:03.983 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:03.983 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:03.983 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:26:03.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:26:03.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:26:03.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:26:03.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:26:04.007 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:04.007 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:04.007 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:04.007 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:26:04.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:26:04.139 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:26:04.139 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:04.139 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:04.139 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:04.139 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:26:04.139 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:04.139 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:04.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:04.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:26:04.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:04.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:04.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:04.143 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:04.143 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:04.143 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:04.196 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 26 2024-09-11T06:26:04.196 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:04.196 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:04.196 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:04.197 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=105, tag_tid=20, entry_tid=10525], [object_number=104, tag_tid=20, entry_tid=10524], [object_number=107, tag_tid=20, entry_tid=10523], [object_number=106, tag_tid=20, entry_tid=10522]]], state=connected] 2024-09-11T06:26:04.200 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:04.201 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:04.202 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:26:04.203 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:04.203 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:04.203 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:04.203 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:26:04.203 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:04.203 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:04.203 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:04.203 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:26:04.203 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:04.203 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:04.203 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:04.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:04.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:04.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:04.252 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 26 2024-09-11T06:26:04.252 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:04.252 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:04.252 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:04.252 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=105, tag_tid=20, entry_tid=10525], [object_number=104, tag_tid=20, entry_tid=10524], [object_number=107, tag_tid=20, entry_tid=10523], [object_number=106, tag_tid=20, entry_tid=10522]]], state=connected] 2024-09-11T06:26:04.256 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:04.258 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=105, tag_tid=20, entry_tid=10525' 2024-09-11T06:26:04.258 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=105, tag_tid=20, entry_tid=10525' 2024-09-11T06:26:04.258 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=105, tag_tid=20, entry_tid=10525' '!=' 'object_number=105, tag_tid=20, entry_tid=10425' 2024-09-11T06:26:04.258 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:26:04.258 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=105, tag_tid=20, entry_tid=10525' = 'object_number=105, tag_tid=20, entry_tid=10425' 2024-09-11T06:26:04.258 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=105, tag_tid=20, entry_tid=10525' 2024-09-11T06:26:04.258 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:04.258 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:04.258 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:04.259 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:26:04.260 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=105, tag_tid=20, entry_tid=10525' 2024-09-11T06:26:04.260 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:04.260 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:04.261 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:26:04.261 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:04.261 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:04.262 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:04.262 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:26:04.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=105, tag_tid=20, entry_tid=10525' 2024-09-11T06:26:04.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:04.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:04.264 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=10525 2024-09-11T06:26:04.265 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 10525 2024-09-11T06:26:04.265 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:26:04.265 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:26:04.265 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:26:04.466 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:26:04.467 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:04.467 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:26:04.467 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:26:04.467 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:26:04.467 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:26:04.467 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:26:04.467 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:26:04.467 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:26:04.467 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:04.467 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:04.467 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:26:04.468 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:26:04.468 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:26:04.468 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:26:04.468 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:26:04.468 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:26:04.473 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:26:04.473 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:26:04.473 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:26:04.473 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:04.473 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:26:04.473 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:26:04.473 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:26:04.473 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:26:04.473 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:26:04.473 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:26:04.473 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:26:04.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:04.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:26:04.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:04.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:26:04.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:04.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:26:04.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:26:04.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:26:04.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:26:04.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:26:04.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:26:04.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:26:04.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:26:04.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:04.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:04.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:04.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:26:04.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:26:04.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:26:04.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:26:04.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:26:04.498 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:04.499 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:04.499 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:04.499 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:26:04.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:26:04.631 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:26:04.631 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:04.631 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:04.631 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:04.631 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:26:04.631 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:04.631 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:04.631 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:04.631 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:26:04.632 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:04.632 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:04.632 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:04.633 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:04.634 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:04.634 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:04.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 26 2024-09-11T06:26:04.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:04.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:04.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:04.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=105, tag_tid=20, entry_tid=10617], [object_number=104, tag_tid=20, entry_tid=10616], [object_number=107, tag_tid=20, entry_tid=10615], [object_number=106, tag_tid=20, entry_tid=10614]]], state=connected] 2024-09-11T06:26:04.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:04.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:04.693 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:26:04.694 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:04.694 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:04.694 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:04.694 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:26:04.694 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:04.694 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:04.694 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:04.694 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:26:04.694 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:04.694 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:04.695 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:04.695 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:04.695 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:04.695 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:04.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 26 2024-09-11T06:26:04.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:04.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:04.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:04.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=105, tag_tid=20, entry_tid=10617], [object_number=104, tag_tid=20, entry_tid=10616], [object_number=107, tag_tid=20, entry_tid=10615], [object_number=106, tag_tid=20, entry_tid=10614]]], state=connected] 2024-09-11T06:26:04.747 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:04.749 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=105, tag_tid=20, entry_tid=10617' 2024-09-11T06:26:04.749 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=105, tag_tid=20, entry_tid=10617' 2024-09-11T06:26:04.749 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=105, tag_tid=20, entry_tid=10617' '!=' 'object_number=105, tag_tid=20, entry_tid=10525' 2024-09-11T06:26:04.750 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:26:04.750 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=105, tag_tid=20, entry_tid=10617' = 'object_number=105, tag_tid=20, entry_tid=10525' 2024-09-11T06:26:04.750 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=105, tag_tid=20, entry_tid=10617' 2024-09-11T06:26:04.750 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:04.750 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:04.750 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:04.750 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:26:04.751 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=105, tag_tid=20, entry_tid=10617' 2024-09-11T06:26:04.751 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:04.751 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:04.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:26:04.753 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:04.753 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:04.753 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:04.754 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:26:04.754 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=105, tag_tid=20, entry_tid=10617' 2024-09-11T06:26:04.754 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:04.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:04.756 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=10617 2024-09-11T06:26:04.756 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 10617 2024-09-11T06:26:04.756 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:26:04.756 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:26:04.756 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:26:04.957 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:26:04.957 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:04.957 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:26:04.957 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:26:04.957 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:26:04.957 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:26:04.957 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:26:04.958 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:26:04.958 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:26:04.958 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:04.958 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:04.958 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:26:04.958 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:26:04.958 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:26:04.958 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:26:04.959 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:26:04.959 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:26:04.962 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:26:04.962 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:26:04.962 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:26:04.962 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:04.962 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:26:04.962 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:26:04.962 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:26:04.962 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:26:04.962 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:26:04.963 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:26:04.963 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:26:04.963 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:04.963 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:26:04.963 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:04.963 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:26:04.963 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:04.963 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:26:04.963 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:26:04.963 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:26:04.963 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:26:04.963 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:26:04.963 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:26:04.963 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:26:04.964 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:26:04.964 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:04.964 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:04.964 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:04.964 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:26:04.964 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:26:04.964 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:26:04.964 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:26:04.964 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:26:04.987 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:04.988 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:04.988 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:04.988 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:26:05.121 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:26:05.123 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:26:05.123 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:05.123 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:05.123 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:05.123 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:26:05.123 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:05.123 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:05.123 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:05.123 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:26:05.123 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:05.124 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:05.124 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:05.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:05.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:05.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:05.173 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 26 2024-09-11T06:26:05.173 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:05.174 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:05.174 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:05.174 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=107, tag_tid=20, entry_tid=10707], [object_number=106, tag_tid=20, entry_tid=10706], [object_number=105, tag_tid=20, entry_tid=10705], [object_number=104, tag_tid=20, entry_tid=10704]]], state=connected] 2024-09-11T06:26:05.177 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:05.178 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:05.179 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:26:05.179 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:05.180 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:05.180 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:05.180 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:26:05.180 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:05.180 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:05.180 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:05.180 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:26:05.180 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:05.180 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:05.180 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:05.181 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:05.181 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:05.181 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:05.227 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 26 2024-09-11T06:26:05.228 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:05.228 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:05.228 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:05.228 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=107, tag_tid=20, entry_tid=10707], [object_number=106, tag_tid=20, entry_tid=10706], [object_number=105, tag_tid=20, entry_tid=10705], [object_number=104, tag_tid=20, entry_tid=10704]]], state=connected] 2024-09-11T06:26:05.231 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:05.233 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=107, tag_tid=20, entry_tid=10707' 2024-09-11T06:26:05.233 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=107, tag_tid=20, entry_tid=10707' 2024-09-11T06:26:05.233 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=107, tag_tid=20, entry_tid=10707' '!=' 'object_number=105, tag_tid=20, entry_tid=10617' 2024-09-11T06:26:05.233 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:26:05.233 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=107, tag_tid=20, entry_tid=10707' = 'object_number=105, tag_tid=20, entry_tid=10617' 2024-09-11T06:26:05.234 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=107, tag_tid=20, entry_tid=10707' 2024-09-11T06:26:05.234 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:05.234 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:05.234 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:05.234 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:26:05.235 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=107, tag_tid=20, entry_tid=10707' 2024-09-11T06:26:05.235 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:05.235 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:05.236 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:26:05.237 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:05.237 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:05.237 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:05.238 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:26:05.238 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=107, tag_tid=20, entry_tid=10707' 2024-09-11T06:26:05.238 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:05.239 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:05.240 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=10707 2024-09-11T06:26:05.240 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 10707 2024-09-11T06:26:05.240 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:26:05.240 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:26:05.240 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:26:05.441 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:26:05.442 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:05.442 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:26:05.442 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:26:05.442 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:26:05.442 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:26:05.442 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:26:05.442 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:26:05.442 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:26:05.442 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:05.442 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:05.442 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:26:05.443 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:26:05.443 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:26:05.443 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:26:05.443 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:26:05.443 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:26:05.446 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:26:05.446 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:26:05.447 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:26:05.447 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:05.447 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:26:05.447 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:26:05.447 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:26:05.447 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:26:05.447 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:26:05.447 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:26:05.447 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:26:05.447 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:05.447 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:26:05.447 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:05.447 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:26:05.448 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:05.448 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:26:05.448 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:26:05.448 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:26:05.448 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:26:05.448 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:26:05.448 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:26:05.448 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:26:05.448 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:26:05.448 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:05.448 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:05.448 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:05.448 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:26:05.449 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:26:05.449 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:26:05.449 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:26:05.449 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:26:05.472 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:05.472 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:05.472 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:05.472 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:26:05.599 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:26:05.600 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:26:05.600 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:05.601 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:05.601 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:05.601 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:26:05.601 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:05.601 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:05.601 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:05.601 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:26:05.601 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:05.602 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:05.602 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:05.603 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:05.603 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:05.603 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:05.651 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 26 2024-09-11T06:26:05.651 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:05.651 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:05.651 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:05.651 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=107, tag_tid=20, entry_tid=10799], [object_number=106, tag_tid=20, entry_tid=10798], [object_number=105, tag_tid=20, entry_tid=10797], [object_number=104, tag_tid=20, entry_tid=10796]]], state=connected] 2024-09-11T06:26:05.654 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:05.656 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:05.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:26:05.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:05.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:05.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:05.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:26:05.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:05.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:05.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:05.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:26:05.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:05.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:05.658 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:05.658 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:05.658 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:05.658 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:05.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 26 2024-09-11T06:26:05.706 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:05.706 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:05.706 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:05.706 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=107, tag_tid=20, entry_tid=10799], [object_number=106, tag_tid=20, entry_tid=10798], [object_number=105, tag_tid=20, entry_tid=10797], [object_number=104, tag_tid=20, entry_tid=10796]]], state=connected] 2024-09-11T06:26:05.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:05.711 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=107, tag_tid=20, entry_tid=10799' 2024-09-11T06:26:05.711 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=107, tag_tid=20, entry_tid=10799' 2024-09-11T06:26:05.711 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=107, tag_tid=20, entry_tid=10799' '!=' 'object_number=107, tag_tid=20, entry_tid=10707' 2024-09-11T06:26:05.711 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:26:05.711 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=107, tag_tid=20, entry_tid=10799' = 'object_number=107, tag_tid=20, entry_tid=10707' 2024-09-11T06:26:05.711 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=107, tag_tid=20, entry_tid=10799' 2024-09-11T06:26:05.712 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:05.712 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:05.712 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:05.712 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:26:05.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=107, tag_tid=20, entry_tid=10799' 2024-09-11T06:26:05.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:05.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:05.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:26:05.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:05.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:05.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:05.716 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:26:05.716 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=107, tag_tid=20, entry_tid=10799' 2024-09-11T06:26:05.716 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:05.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:05.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=10799 2024-09-11T06:26:05.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 10799 2024-09-11T06:26:05.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:26:05.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:26:05.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:26:05.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:26:05.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:05.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:26:05.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:26:05.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:26:05.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:26:05.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:26:05.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:26:05.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:26:05.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:05.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:05.921 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:26:05.921 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:26:05.921 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:26:05.921 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:26:05.921 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:26:05.921 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:26:05.924 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:26:05.924 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:26:05.924 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:26:05.924 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:05.925 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:26:05.925 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:26:05.925 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:26:05.925 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:26:05.925 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:26:05.925 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:26:05.925 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:26:05.925 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:05.925 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:26:05.925 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:05.925 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:26:05.925 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:05.925 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:26:05.926 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:26:05.926 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:26:05.926 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:26:05.926 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:26:05.926 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:26:05.926 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:26:05.926 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:26:05.926 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:05.926 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:05.926 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:05.926 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:26:05.926 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:26:05.926 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:26:05.926 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:26:05.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:26:05.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:05.952 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:05.952 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:05.952 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:26:06.082 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:26:06.084 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:26:06.084 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:06.084 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:06.084 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:06.085 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:26:06.085 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:06.085 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:06.085 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:06.085 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:26:06.085 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:06.085 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:06.085 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:06.086 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:06.087 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:06.087 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:06.139 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 26 2024-09-11T06:26:06.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:06.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:06.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:06.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=105, tag_tid=20, entry_tid=10885], [object_number=104, tag_tid=20, entry_tid=10884], [object_number=107, tag_tid=20, entry_tid=10883], [object_number=106, tag_tid=20, entry_tid=10882]]], state=connected] 2024-09-11T06:26:06.143 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:06.147 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:06.147 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:26:06.147 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:06.147 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:06.147 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:06.147 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:26:06.148 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:06.148 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:06.148 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:06.148 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:26:06.148 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:06.148 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:06.149 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:06.149 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:06.149 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:06.149 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:06.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 26 2024-09-11T06:26:06.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:06.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:06.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:06.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=105, tag_tid=20, entry_tid=10885], [object_number=104, tag_tid=20, entry_tid=10884], [object_number=107, tag_tid=20, entry_tid=10883], [object_number=106, tag_tid=20, entry_tid=10882]]], state=connected] 2024-09-11T06:26:06.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:06.200 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=105, tag_tid=20, entry_tid=10885' 2024-09-11T06:26:06.201 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=105, tag_tid=20, entry_tid=10885' 2024-09-11T06:26:06.201 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=105, tag_tid=20, entry_tid=10885' '!=' 'object_number=107, tag_tid=20, entry_tid=10799' 2024-09-11T06:26:06.201 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:26:06.201 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=105, tag_tid=20, entry_tid=10885' = 'object_number=107, tag_tid=20, entry_tid=10799' 2024-09-11T06:26:06.201 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=105, tag_tid=20, entry_tid=10885' 2024-09-11T06:26:06.201 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:06.202 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:06.202 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:06.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:26:06.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=105, tag_tid=20, entry_tid=10885' 2024-09-11T06:26:06.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:06.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:06.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:26:06.205 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:06.205 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:06.205 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:06.207 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:26:06.207 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=105, tag_tid=20, entry_tid=10885' 2024-09-11T06:26:06.207 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:06.208 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:06.208 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=10885 2024-09-11T06:26:06.208 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 10885 2024-09-11T06:26:06.208 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:26:06.208 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:26:06.209 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:26:06.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:26:06.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:06.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:26:06.411 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:26:06.411 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:26:06.411 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:26:06.411 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:26:06.411 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:26:06.411 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:26:06.412 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:06.412 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:06.412 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:26:06.412 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:26:06.412 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:26:06.412 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:26:06.413 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:26:06.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:26:06.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:26:06.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:26:06.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:26:06.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:06.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:26:06.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:26:06.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:26:06.418 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:26:06.418 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:26:06.418 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:26:06.418 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:26:06.418 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:06.418 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:26:06.418 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:06.418 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:26:06.418 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:06.418 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:26:06.418 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:26:06.418 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:26:06.418 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:26:06.418 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:26:06.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:26:06.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:26:06.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:26:06.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:06.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:06.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:06.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:26:06.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:26:06.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:26:06.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:26:06.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:26:06.439 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:06.439 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:06.439 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:06.439 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:26:06.565 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:26:06.566 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:26:06.566 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:06.566 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:06.566 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:06.566 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:26:06.566 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:06.566 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:06.566 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:06.566 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:26:06.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:06.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:06.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:06.569 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:06.569 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:06.569 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:06.626 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 26 2024-09-11T06:26:06.626 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:06.626 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:06.626 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:06.626 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=106, tag_tid=20, entry_tid=10974], [object_number=105, tag_tid=20, entry_tid=10973], [object_number=104, tag_tid=20, entry_tid=10972], [object_number=107, tag_tid=20, entry_tid=10971]]], state=connected] 2024-09-11T06:26:06.629 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:06.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:06.632 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:26:06.632 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:06.632 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:06.632 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:06.632 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:26:06.632 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:06.632 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:06.632 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:06.632 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:26:06.633 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:06.633 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:06.633 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:06.633 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:06.633 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:06.633 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:06.683 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 26 2024-09-11T06:26:06.683 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:06.684 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:06.684 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:06.684 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=106, tag_tid=20, entry_tid=10974], [object_number=105, tag_tid=20, entry_tid=10973], [object_number=104, tag_tid=20, entry_tid=10972], [object_number=107, tag_tid=20, entry_tid=10971]]], state=connected] 2024-09-11T06:26:06.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:06.689 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=106, tag_tid=20, entry_tid=10974' 2024-09-11T06:26:06.689 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=106, tag_tid=20, entry_tid=10974' 2024-09-11T06:26:06.689 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=106, tag_tid=20, entry_tid=10974' '!=' 'object_number=105, tag_tid=20, entry_tid=10885' 2024-09-11T06:26:06.689 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:26:06.689 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=106, tag_tid=20, entry_tid=10974' = 'object_number=105, tag_tid=20, entry_tid=10885' 2024-09-11T06:26:06.690 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=106, tag_tid=20, entry_tid=10974' 2024-09-11T06:26:06.690 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:06.690 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:06.690 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:06.690 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:26:06.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=106, tag_tid=20, entry_tid=10974' 2024-09-11T06:26:06.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:06.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:06.693 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:26:06.693 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:06.693 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:06.694 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:06.694 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:26:06.694 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=106, tag_tid=20, entry_tid=10974' 2024-09-11T06:26:06.694 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:06.694 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:06.696 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=10974 2024-09-11T06:26:06.696 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 10974 2024-09-11T06:26:06.696 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:26:06.696 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:26:06.696 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:26:06.898 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:26:06.898 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:06.898 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:26:06.898 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:26:06.898 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:26:06.898 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:26:06.898 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:26:06.899 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:26:06.899 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:26:06.899 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:06.899 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:06.899 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:26:06.899 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:26:06.899 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:26:06.899 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:26:06.899 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:26:06.900 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:26:06.902 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:26:06.902 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:26:06.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:26:06.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:06.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:26:06.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:26:06.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:26:06.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:26:06.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:26:06.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:26:06.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:26:06.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:06.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:26:06.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:06.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:26:06.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:06.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:26:06.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:26:06.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:26:06.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:26:06.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:26:06.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:26:06.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:26:06.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:26:06.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:06.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:06.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:06.905 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:26:06.905 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:26:06.905 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:26:06.905 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:26:06.905 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:26:06.928 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:06.929 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:06.929 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:06.929 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:26:07.064 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:26:07.065 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:26:07.065 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:07.065 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:07.065 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:07.065 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:26:07.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:07.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:07.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:07.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:26:07.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:07.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:07.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:07.068 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:07.068 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:07.068 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:07.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 26 2024-09-11T06:26:07.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:07.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:07.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:07.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=107, tag_tid=20, entry_tid=11079], [object_number=106, tag_tid=20, entry_tid=11078], [object_number=105, tag_tid=20, entry_tid=11077], [object_number=104, tag_tid=20, entry_tid=11076]]], state=connected] 2024-09-11T06:26:07.130 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:07.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:07.133 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:26:07.133 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:07.133 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:07.133 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:07.133 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:26:07.133 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:07.133 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:07.133 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:07.133 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:26:07.133 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:07.133 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:07.133 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:07.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:07.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:07.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:07.182 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 26 2024-09-11T06:26:07.182 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:07.183 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:07.183 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:07.183 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=107, tag_tid=20, entry_tid=11079], [object_number=106, tag_tid=20, entry_tid=11078], [object_number=105, tag_tid=20, entry_tid=11077], [object_number=104, tag_tid=20, entry_tid=11076]]], state=connected] 2024-09-11T06:26:07.186 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:07.188 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=107, tag_tid=20, entry_tid=11079' 2024-09-11T06:26:07.188 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=107, tag_tid=20, entry_tid=11079' 2024-09-11T06:26:07.188 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=107, tag_tid=20, entry_tid=11079' '!=' 'object_number=106, tag_tid=20, entry_tid=10974' 2024-09-11T06:26:07.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:26:07.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=107, tag_tid=20, entry_tid=11079' = 'object_number=106, tag_tid=20, entry_tid=10974' 2024-09-11T06:26:07.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=107, tag_tid=20, entry_tid=11079' 2024-09-11T06:26:07.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:07.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:07.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:07.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:26:07.190 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=107, tag_tid=20, entry_tid=11079' 2024-09-11T06:26:07.190 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:07.190 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:07.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:26:07.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:07.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:07.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:07.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:26:07.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=107, tag_tid=20, entry_tid=11079' 2024-09-11T06:26:07.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:07.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:07.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=11079 2024-09-11T06:26:07.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 11079 2024-09-11T06:26:07.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:26:07.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:26:07.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:26:07.397 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:26:07.398 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:07.398 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:26:07.398 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:26:07.398 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:26:07.398 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:26:07.398 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:26:07.398 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:26:07.398 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:26:07.398 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:07.398 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:07.398 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:26:07.399 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:26:07.399 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:26:07.399 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:26:07.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:26:07.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:26:07.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:26:07.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:26:07.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:26:07.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:07.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:26:07.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:26:07.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:26:07.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:26:07.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:26:07.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:26:07.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:26:07.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:07.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:26:07.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:07.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:26:07.403 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:07.403 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:26:07.403 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:26:07.403 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:26:07.403 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:26:07.403 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:26:07.403 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:26:07.403 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:26:07.403 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:26:07.403 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:07.403 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:07.403 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:07.403 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:26:07.403 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:26:07.404 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:26:07.404 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:26:07.404 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:26:07.427 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:07.428 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:07.428 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:07.428 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:26:07.585 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:26:07.587 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:26:07.587 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:07.587 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:07.587 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:07.587 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:26:07.587 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:07.587 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:07.587 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:07.588 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:26:07.588 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:07.588 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:07.588 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:07.590 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:07.591 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:07.591 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:07.638 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 26 2024-09-11T06:26:07.638 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:07.638 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:07.638 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:07.638 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=104, tag_tid=20, entry_tid=11152], [object_number=107, tag_tid=20, entry_tid=11151], [object_number=106, tag_tid=20, entry_tid=11150], [object_number=105, tag_tid=20, entry_tid=11149]]], state=connected] 2024-09-11T06:26:07.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:07.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:07.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:26:07.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:07.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:07.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:07.645 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:26:07.645 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:07.645 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:07.645 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:07.645 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:26:07.645 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:07.645 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:07.645 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:07.646 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:07.646 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:07.646 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:07.693 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 26 2024-09-11T06:26:07.693 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:07.694 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:07.694 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:07.694 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=104, tag_tid=20, entry_tid=11152], [object_number=107, tag_tid=20, entry_tid=11151], [object_number=106, tag_tid=20, entry_tid=11150], [object_number=105, tag_tid=20, entry_tid=11149]]], state=connected] 2024-09-11T06:26:07.696 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:07.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=104, tag_tid=20, entry_tid=11152' 2024-09-11T06:26:07.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=104, tag_tid=20, entry_tid=11152' 2024-09-11T06:26:07.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=104, tag_tid=20, entry_tid=11152' '!=' 'object_number=107, tag_tid=20, entry_tid=11079' 2024-09-11T06:26:07.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:26:07.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=104, tag_tid=20, entry_tid=11152' = 'object_number=107, tag_tid=20, entry_tid=11079' 2024-09-11T06:26:07.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=104, tag_tid=20, entry_tid=11152' 2024-09-11T06:26:07.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:07.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:07.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:07.700 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:26:07.700 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=104, tag_tid=20, entry_tid=11152' 2024-09-11T06:26:07.701 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:07.701 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:07.701 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:26:07.702 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:07.702 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:07.702 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:07.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:26:07.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=104, tag_tid=20, entry_tid=11152' 2024-09-11T06:26:07.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:07.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:07.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=11152 2024-09-11T06:26:07.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 11152 2024-09-11T06:26:07.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:26:07.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:26:07.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:26:07.907 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:26:07.907 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:07.907 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:26:07.907 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:26:07.907 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:26:07.907 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:26:07.907 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:26:07.907 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:26:07.907 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:26:07.908 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:07.908 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:07.908 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:26:07.908 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:26:07.908 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:26:07.908 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:26:07.908 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:26:07.909 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:26:07.911 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:26:07.911 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:26:07.911 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:26:07.912 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:07.912 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:26:07.912 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:26:07.912 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:26:07.912 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:26:07.912 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:26:07.912 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:26:07.912 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:26:07.912 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:07.912 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:26:07.912 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:07.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:26:07.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:07.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:26:07.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:26:07.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:26:07.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:26:07.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:26:07.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:26:07.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:26:07.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:26:07.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:07.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:07.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:07.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:26:07.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:26:07.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:26:07.915 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:26:07.915 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:26:07.937 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:07.938 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:07.938 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:07.938 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:26:08.078 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:26:08.079 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:26:08.079 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:08.079 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:08.079 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:08.079 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:26:08.079 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:08.079 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:08.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:08.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:26:08.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:08.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:08.081 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:08.082 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:08.083 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:08.084 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:08.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 27 2024-09-11T06:26:08.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:08.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:08.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:08.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=108, tag_tid=20, entry_tid=11184], [object_number=111, tag_tid=20, entry_tid=11183], [object_number=110, tag_tid=20, entry_tid=11182], [object_number=109, tag_tid=20, entry_tid=11181]]], state=connected] 2024-09-11T06:26:08.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:08.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:08.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:26:08.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:08.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:08.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:08.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:26:08.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:08.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:08.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:08.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:26:08.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:08.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:08.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:08.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:08.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:08.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:08.188 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 27 2024-09-11T06:26:08.188 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:08.188 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:08.188 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:08.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=108, tag_tid=20, entry_tid=11184], [object_number=111, tag_tid=20, entry_tid=11183], [object_number=110, tag_tid=20, entry_tid=11182], [object_number=109, tag_tid=20, entry_tid=11181]]], state=connected] 2024-09-11T06:26:08.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:08.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=108, tag_tid=20, entry_tid=11184' 2024-09-11T06:26:08.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=108, tag_tid=20, entry_tid=11184' 2024-09-11T06:26:08.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=108, tag_tid=20, entry_tid=11184' '!=' 'object_number=104, tag_tid=20, entry_tid=11152' 2024-09-11T06:26:08.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:26:08.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=108, tag_tid=20, entry_tid=11184' = 'object_number=104, tag_tid=20, entry_tid=11152' 2024-09-11T06:26:08.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=108, tag_tid=20, entry_tid=11184' 2024-09-11T06:26:08.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:08.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:08.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:08.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:26:08.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=108, tag_tid=20, entry_tid=11184' 2024-09-11T06:26:08.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:08.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:08.196 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:26:08.197 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:08.197 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:08.197 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:08.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:26:08.199 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=108, tag_tid=20, entry_tid=11184' 2024-09-11T06:26:08.199 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:08.199 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:08.200 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=11184 2024-09-11T06:26:08.200 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 11184 2024-09-11T06:26:08.200 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:26:08.200 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:26:08.200 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:26:08.403 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:26:08.403 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:08.404 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:26:08.404 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:26:08.404 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:26:08.404 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:26:08.404 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:26:08.405 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:26:08.405 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:26:08.405 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:08.405 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:08.405 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:26:08.405 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:26:08.406 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:26:08.406 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:26:08.406 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:26:08.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:26:08.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:26:08.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:26:08.411 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:26:08.411 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:08.411 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:26:08.411 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:26:08.411 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:26:08.411 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:26:08.412 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:26:08.412 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:26:08.412 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:26:08.412 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:08.412 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:26:08.413 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:08.413 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:26:08.413 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:08.413 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:26:08.413 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:26:08.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:26:08.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:26:08.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:26:08.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:26:08.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:26:08.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:26:08.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:08.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:08.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:08.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:26:08.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:26:08.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:26:08.416 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:26:08.416 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:26:08.434 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:08.434 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:08.434 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:08.434 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:26:08.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:26:08.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:26:08.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:08.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:08.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:08.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:26:08.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:08.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:08.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:08.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:26:08.572 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:08.572 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:08.572 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:08.574 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:08.575 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:08.575 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:08.628 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 27 2024-09-11T06:26:08.628 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:08.628 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:08.628 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:08.628 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=108, tag_tid=20, entry_tid=11280], [object_number=111, tag_tid=20, entry_tid=11279], [object_number=110, tag_tid=20, entry_tid=11278], [object_number=109, tag_tid=20, entry_tid=11277]]], state=connected] 2024-09-11T06:26:08.632 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:08.633 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:08.634 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:26:08.634 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:08.634 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:08.634 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:08.634 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:26:08.634 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:08.634 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:08.634 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:08.634 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:26:08.634 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:08.635 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:08.635 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:08.636 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:08.636 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:08.636 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:08.683 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 27 2024-09-11T06:26:08.683 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:08.683 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:08.683 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:08.683 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=108, tag_tid=20, entry_tid=11280], [object_number=111, tag_tid=20, entry_tid=11279], [object_number=110, tag_tid=20, entry_tid=11278], [object_number=109, tag_tid=20, entry_tid=11277]]], state=connected] 2024-09-11T06:26:08.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:08.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=108, tag_tid=20, entry_tid=11280' 2024-09-11T06:26:08.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=108, tag_tid=20, entry_tid=11280' 2024-09-11T06:26:08.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=108, tag_tid=20, entry_tid=11280' '!=' 'object_number=108, tag_tid=20, entry_tid=11184' 2024-09-11T06:26:08.689 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:26:08.689 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=108, tag_tid=20, entry_tid=11280' = 'object_number=108, tag_tid=20, entry_tid=11184' 2024-09-11T06:26:08.689 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=108, tag_tid=20, entry_tid=11280' 2024-09-11T06:26:08.689 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:08.689 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:08.690 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:08.690 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:26:08.690 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=108, tag_tid=20, entry_tid=11280' 2024-09-11T06:26:08.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:08.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:08.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:26:08.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:08.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:08.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:08.693 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:26:08.694 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=108, tag_tid=20, entry_tid=11280' 2024-09-11T06:26:08.694 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:08.694 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:08.696 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=11280 2024-09-11T06:26:08.696 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 11280 2024-09-11T06:26:08.696 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:26:08.696 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:26:08.696 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:26:08.897 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:26:08.897 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:08.898 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:26:08.898 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:26:08.898 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:26:08.898 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:26:08.898 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:26:08.898 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:26:08.898 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:26:08.898 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:08.898 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:08.898 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:26:08.899 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:26:08.899 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:26:08.899 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:26:08.899 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:26:08.899 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:26:08.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:26:08.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:26:08.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:26:08.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:08.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:26:08.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:26:08.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:26:08.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:26:08.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:26:08.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:26:08.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:26:08.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:08.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:26:08.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:08.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:26:08.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:08.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:26:08.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:26:08.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:26:08.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:26:08.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:26:08.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:26:08.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:26:08.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:26:08.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:08.905 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:08.905 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:08.905 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:26:08.905 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:26:08.905 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:26:08.905 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:26:08.905 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:26:08.931 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:08.931 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:08.931 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:08.931 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:26:09.063 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:26:09.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:26:09.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:09.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:09.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:09.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:26:09.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:09.067 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:09.067 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:09.067 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:26:09.067 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:09.067 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:09.067 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:09.068 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:09.069 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:09.069 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:09.118 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 27 2024-09-11T06:26:09.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:09.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:09.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:09.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=111, tag_tid=20, entry_tid=11375], [object_number=110, tag_tid=20, entry_tid=11374], [object_number=109, tag_tid=20, entry_tid=11373], [object_number=108, tag_tid=20, entry_tid=11372]]], state=connected] 2024-09-11T06:26:09.122 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:09.123 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:09.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:26:09.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:09.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:09.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:09.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:26:09.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:09.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:09.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:09.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:26:09.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:09.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:09.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:09.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:09.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:09.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:09.174 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 27 2024-09-11T06:26:09.174 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:09.174 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:09.174 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:09.174 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=111, tag_tid=20, entry_tid=11375], [object_number=110, tag_tid=20, entry_tid=11374], [object_number=109, tag_tid=20, entry_tid=11373], [object_number=108, tag_tid=20, entry_tid=11372]]], state=connected] 2024-09-11T06:26:09.177 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:09.179 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=111, tag_tid=20, entry_tid=11375' 2024-09-11T06:26:09.179 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=111, tag_tid=20, entry_tid=11375' 2024-09-11T06:26:09.180 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=111, tag_tid=20, entry_tid=11375' '!=' 'object_number=108, tag_tid=20, entry_tid=11280' 2024-09-11T06:26:09.180 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:26:09.180 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=111, tag_tid=20, entry_tid=11375' = 'object_number=108, tag_tid=20, entry_tid=11280' 2024-09-11T06:26:09.180 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=111, tag_tid=20, entry_tid=11375' 2024-09-11T06:26:09.180 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:09.180 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:09.180 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:09.181 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:26:09.181 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=111, tag_tid=20, entry_tid=11375' 2024-09-11T06:26:09.181 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:09.182 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:09.182 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:26:09.183 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:09.183 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:09.183 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:09.184 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:26:09.185 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=111, tag_tid=20, entry_tid=11375' 2024-09-11T06:26:09.185 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:09.185 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:09.186 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=11375 2024-09-11T06:26:09.186 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 11375 2024-09-11T06:26:09.186 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:26:09.186 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:26:09.186 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:26:09.388 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:26:09.388 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:09.388 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:26:09.388 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:26:09.388 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:26:09.388 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:26:09.389 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:26:09.389 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:26:09.389 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:26:09.389 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:09.389 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:09.389 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:26:09.389 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:26:09.389 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:26:09.389 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:26:09.390 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:26:09.390 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:26:09.394 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:26:09.394 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:26:09.394 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:26:09.394 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:09.394 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:26:09.394 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:26:09.394 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:26:09.394 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:26:09.395 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:26:09.395 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:26:09.395 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:26:09.395 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:09.395 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:26:09.395 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:09.395 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:26:09.395 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:09.395 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:26:09.395 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:26:09.395 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:26:09.395 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:26:09.395 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:26:09.396 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:26:09.396 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:26:09.396 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:26:09.396 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:09.396 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:09.396 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:09.396 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:26:09.396 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:26:09.396 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:26:09.396 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:26:09.396 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:26:09.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:09.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:09.420 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:09.420 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:26:09.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:26:09.553 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:26:09.553 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:09.553 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:09.553 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:09.553 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:26:09.553 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:09.553 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:09.553 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:09.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:26:09.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:09.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:09.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:09.556 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:09.556 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:09.556 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:09.603 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 27 2024-09-11T06:26:09.604 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:09.604 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:09.604 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:09.604 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=110, tag_tid=20, entry_tid=11466], [object_number=109, tag_tid=20, entry_tid=11465], [object_number=108, tag_tid=20, entry_tid=11464], [object_number=111, tag_tid=20, entry_tid=11463]]], state=connected] 2024-09-11T06:26:09.607 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:09.608 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:09.609 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:26:09.609 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:09.609 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:09.610 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:09.610 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:26:09.610 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:09.610 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:09.610 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:09.610 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:26:09.610 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:09.610 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:09.610 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:09.611 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:09.611 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:09.611 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:09.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 27 2024-09-11T06:26:09.658 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:09.658 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:09.658 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:09.658 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=110, tag_tid=20, entry_tid=11466], [object_number=109, tag_tid=20, entry_tid=11465], [object_number=108, tag_tid=20, entry_tid=11464], [object_number=111, tag_tid=20, entry_tid=11463]]], state=connected] 2024-09-11T06:26:09.661 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:09.663 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=110, tag_tid=20, entry_tid=11466' 2024-09-11T06:26:09.663 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=110, tag_tid=20, entry_tid=11466' 2024-09-11T06:26:09.663 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=110, tag_tid=20, entry_tid=11466' '!=' 'object_number=111, tag_tid=20, entry_tid=11375' 2024-09-11T06:26:09.663 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:26:09.664 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=110, tag_tid=20, entry_tid=11466' = 'object_number=111, tag_tid=20, entry_tid=11375' 2024-09-11T06:26:09.664 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=110, tag_tid=20, entry_tid=11466' 2024-09-11T06:26:09.664 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:09.664 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:09.664 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:09.664 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:26:09.665 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=110, tag_tid=20, entry_tid=11466' 2024-09-11T06:26:09.665 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:09.665 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:09.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:26:09.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:09.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:09.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:09.668 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:26:09.668 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=110, tag_tid=20, entry_tid=11466' 2024-09-11T06:26:09.668 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:09.669 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:09.670 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=11466 2024-09-11T06:26:09.670 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 11466 2024-09-11T06:26:09.670 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:26:09.670 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:26:09.670 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:26:09.872 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:26:09.872 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:09.872 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:26:09.872 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:26:09.872 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:26:09.872 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:26:09.872 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:26:09.872 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:26:09.872 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:26:09.873 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:09.873 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:09.873 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:26:09.873 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:26:09.873 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:26:09.873 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:26:09.873 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:26:09.874 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:26:09.875 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:26:09.875 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:26:09.875 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:26:09.875 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:09.875 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:26:09.875 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:26:09.876 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:26:09.876 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:26:09.876 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:26:09.876 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:26:09.876 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:26:09.876 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:09.876 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:26:09.876 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:09.876 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:26:09.876 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:09.878 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:26:09.878 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:26:09.878 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:26:09.878 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:26:09.878 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:26:09.878 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:26:09.878 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:26:09.878 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:26:09.878 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:09.878 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:09.878 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:09.879 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:26:09.879 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:26:09.879 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:26:09.879 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:26:09.879 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:26:09.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:09.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:09.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:09.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:26:10.032 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:26:10.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:26:10.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:10.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:10.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:10.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:26:10.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:10.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:10.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:10.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:26:10.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:10.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:10.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:10.036 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:10.036 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:10.036 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:10.082 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 27 2024-09-11T06:26:10.082 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:10.083 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:10.083 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:10.083 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=111, tag_tid=20, entry_tid=11555], [object_number=110, tag_tid=20, entry_tid=11554], [object_number=109, tag_tid=20, entry_tid=11553], [object_number=108, tag_tid=20, entry_tid=11552]]], state=connected] 2024-09-11T06:26:10.086 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:10.088 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:10.088 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:26:10.088 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:10.089 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:10.089 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:10.089 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:26:10.089 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:10.089 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:10.089 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:10.089 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:26:10.089 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:10.089 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:10.089 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:10.090 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:10.090 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:10.090 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:10.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 27 2024-09-11T06:26:10.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:10.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:10.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:10.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=111, tag_tid=20, entry_tid=11555], [object_number=110, tag_tid=20, entry_tid=11554], [object_number=109, tag_tid=20, entry_tid=11553], [object_number=108, tag_tid=20, entry_tid=11552]]], state=connected] 2024-09-11T06:26:10.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:10.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=111, tag_tid=20, entry_tid=11555' 2024-09-11T06:26:10.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=111, tag_tid=20, entry_tid=11555' 2024-09-11T06:26:10.143 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=111, tag_tid=20, entry_tid=11555' '!=' 'object_number=110, tag_tid=20, entry_tid=11466' 2024-09-11T06:26:10.143 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:26:10.143 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=111, tag_tid=20, entry_tid=11555' = 'object_number=110, tag_tid=20, entry_tid=11466' 2024-09-11T06:26:10.143 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=111, tag_tid=20, entry_tid=11555' 2024-09-11T06:26:10.144 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:10.144 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:10.144 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:10.144 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:26:10.145 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=111, tag_tid=20, entry_tid=11555' 2024-09-11T06:26:10.145 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:10.145 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:10.145 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:26:10.146 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:10.146 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:10.146 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:10.147 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:26:10.148 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=111, tag_tid=20, entry_tid=11555' 2024-09-11T06:26:10.148 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:10.148 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:10.149 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=11555 2024-09-11T06:26:10.149 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 11555 2024-09-11T06:26:10.149 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:26:10.149 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:26:10.149 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:26:10.352 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:26:10.352 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:10.352 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:26:10.352 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:26:10.352 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:26:10.352 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:26:10.352 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:26:10.352 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:26:10.352 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:26:10.352 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:10.353 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:10.353 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:26:10.353 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:26:10.353 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:26:10.353 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:26:10.353 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:26:10.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:26:10.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:26:10.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:26:10.356 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:26:10.356 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:10.356 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:26:10.356 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:26:10.356 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:26:10.356 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:26:10.356 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:26:10.356 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:26:10.356 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:26:10.356 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:10.356 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:26:10.356 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:10.356 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:26:10.357 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:10.357 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:26:10.357 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:26:10.357 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:26:10.357 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:26:10.357 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:26:10.357 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:26:10.357 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:26:10.357 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:26:10.357 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:10.357 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:10.357 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:10.357 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:26:10.358 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:26:10.358 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:26:10.358 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:26:10.358 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:26:10.381 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:10.381 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:10.381 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:10.381 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:26:10.517 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:26:10.518 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:26:10.518 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:10.518 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:10.518 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:10.518 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:26:10.519 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:10.519 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:10.519 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:10.519 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:26:10.519 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:10.519 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:10.520 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:10.522 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:10.522 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:10.523 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:10.576 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 27 2024-09-11T06:26:10.576 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:10.576 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:10.576 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:10.576 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=108, tag_tid=20, entry_tid=11644], [object_number=111, tag_tid=20, entry_tid=11643], [object_number=110, tag_tid=20, entry_tid=11642], [object_number=109, tag_tid=20, entry_tid=11641]]], state=connected] 2024-09-11T06:26:10.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:10.580 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:10.581 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:26:10.582 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:10.582 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:10.582 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:10.582 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:26:10.582 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:10.582 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:10.582 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:10.582 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:26:10.582 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:10.582 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:10.582 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:10.583 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:10.583 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:10.583 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:10.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 27 2024-09-11T06:26:10.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:10.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:10.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:10.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=108, tag_tid=20, entry_tid=11644], [object_number=111, tag_tid=20, entry_tid=11643], [object_number=110, tag_tid=20, entry_tid=11642], [object_number=109, tag_tid=20, entry_tid=11641]]], state=connected] 2024-09-11T06:26:10.634 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:10.635 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=108, tag_tid=20, entry_tid=11644' 2024-09-11T06:26:10.635 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=108, tag_tid=20, entry_tid=11644' 2024-09-11T06:26:10.636 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=108, tag_tid=20, entry_tid=11644' '!=' 'object_number=111, tag_tid=20, entry_tid=11555' 2024-09-11T06:26:10.636 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:26:10.636 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=108, tag_tid=20, entry_tid=11644' = 'object_number=111, tag_tid=20, entry_tid=11555' 2024-09-11T06:26:10.636 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=108, tag_tid=20, entry_tid=11644' 2024-09-11T06:26:10.636 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:10.636 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:10.637 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:10.637 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:26:10.638 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=108, tag_tid=20, entry_tid=11644' 2024-09-11T06:26:10.638 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:10.638 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:10.638 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:26:10.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:10.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:10.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:10.640 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:26:10.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=108, tag_tid=20, entry_tid=11644' 2024-09-11T06:26:10.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:10.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:10.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=11644 2024-09-11T06:26:10.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 11644 2024-09-11T06:26:10.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:26:10.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:26:10.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:26:10.844 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:26:10.844 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:10.845 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:26:10.845 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:26:10.845 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:26:10.845 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:26:10.845 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:26:10.845 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:26:10.845 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:26:10.845 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:10.845 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:10.845 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:26:10.845 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:26:10.845 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:26:10.846 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:26:10.846 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:26:10.846 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:26:10.848 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:26:10.848 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:26:10.848 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:26:10.848 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:10.848 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:26:10.848 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:26:10.848 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:26:10.848 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:26:10.848 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:26:10.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:26:10.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:26:10.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:10.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:26:10.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:10.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:26:10.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:10.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:26:10.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:26:10.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:26:10.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:26:10.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:26:10.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:26:10.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:26:10.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:26:10.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:10.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:10.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:10.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:26:10.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:26:10.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:26:10.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:26:10.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:26:10.874 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:10.875 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:10.875 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:10.875 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:26:11.007 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:26:11.009 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:26:11.009 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:11.009 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:11.009 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:11.009 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:26:11.009 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:11.009 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:11.009 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:11.009 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:26:11.009 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:11.013 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:11.013 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:11.013 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:11.013 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:11.013 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:11.058 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 27 2024-09-11T06:26:11.058 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:11.058 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:11.058 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:11.058 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=108, tag_tid=20, entry_tid=11728], [object_number=111, tag_tid=20, entry_tid=11727], [object_number=110, tag_tid=20, entry_tid=11726], [object_number=109, tag_tid=20, entry_tid=11725]]], state=connected] 2024-09-11T06:26:11.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:11.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:11.063 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:26:11.064 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:11.064 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:11.064 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:11.064 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:26:11.064 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:11.064 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:11.064 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:11.064 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:26:11.064 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:11.064 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:11.064 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:11.065 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:11.065 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:11.065 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:11.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 27 2024-09-11T06:26:11.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:11.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:11.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:11.112 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=108, tag_tid=20, entry_tid=11728], [object_number=111, tag_tid=20, entry_tid=11727], [object_number=110, tag_tid=20, entry_tid=11726], [object_number=109, tag_tid=20, entry_tid=11725]]], state=connected] 2024-09-11T06:26:11.115 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:11.116 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=108, tag_tid=20, entry_tid=11728' 2024-09-11T06:26:11.116 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=108, tag_tid=20, entry_tid=11728' 2024-09-11T06:26:11.116 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=108, tag_tid=20, entry_tid=11728' '!=' 'object_number=108, tag_tid=20, entry_tid=11644' 2024-09-11T06:26:11.117 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:26:11.117 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=108, tag_tid=20, entry_tid=11728' = 'object_number=108, tag_tid=20, entry_tid=11644' 2024-09-11T06:26:11.117 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=108, tag_tid=20, entry_tid=11728' 2024-09-11T06:26:11.117 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:11.117 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:11.117 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:11.118 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:26:11.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=108, tag_tid=20, entry_tid=11728' 2024-09-11T06:26:11.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:11.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:11.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:26:11.120 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:11.120 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:11.120 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:11.121 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:26:11.122 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=108, tag_tid=20, entry_tid=11728' 2024-09-11T06:26:11.122 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:11.122 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:11.123 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=11728 2024-09-11T06:26:11.123 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 11728 2024-09-11T06:26:11.124 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:26:11.124 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:26:11.124 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:26:11.325 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:26:11.325 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:11.325 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:26:11.325 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:26:11.326 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:26:11.326 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:26:11.326 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:26:11.326 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:26:11.326 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:26:11.326 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:11.326 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:11.326 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:26:11.326 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:26:11.326 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:26:11.326 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:26:11.327 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:26:11.327 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:26:11.331 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:26:11.331 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:26:11.331 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:26:11.331 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:11.331 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:26:11.332 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:26:11.332 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:26:11.332 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:26:11.332 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:26:11.332 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:26:11.332 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:26:11.332 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:11.332 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:26:11.332 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:11.332 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:26:11.332 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:11.332 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:26:11.332 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:26:11.332 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:26:11.333 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:26:11.333 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:26:11.333 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:26:11.333 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:26:11.333 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:26:11.333 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:11.333 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:11.333 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:11.333 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:26:11.333 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:26:11.333 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:26:11.333 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:26:11.333 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:26:11.358 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:11.358 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:11.358 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:11.358 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:26:11.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:26:11.490 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:26:11.490 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:11.490 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:11.490 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:11.491 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:26:11.491 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:11.491 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:11.491 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:11.491 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:26:11.491 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:11.491 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:11.491 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:11.493 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:11.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:11.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:11.546 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 27 2024-09-11T06:26:11.546 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:11.546 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:11.546 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:11.546 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=111, tag_tid=20, entry_tid=11811], [object_number=110, tag_tid=20, entry_tid=11810], [object_number=109, tag_tid=20, entry_tid=11809], [object_number=108, tag_tid=20, entry_tid=11808]]], state=connected] 2024-09-11T06:26:11.549 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:11.551 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:11.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:26:11.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:11.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:11.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:11.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:26:11.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:11.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:11.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:11.553 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:26:11.553 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:11.553 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:11.553 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:11.553 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:11.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:11.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:11.808 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 27 2024-09-11T06:26:11.808 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:11.808 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:11.808 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:11.808 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=111, tag_tid=20, entry_tid=11811], [object_number=110, tag_tid=20, entry_tid=11810], [object_number=109, tag_tid=20, entry_tid=11809], [object_number=108, tag_tid=20, entry_tid=11808]]], state=connected] 2024-09-11T06:26:11.813 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:11.815 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=111, tag_tid=20, entry_tid=11811' 2024-09-11T06:26:11.815 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=111, tag_tid=20, entry_tid=11811' 2024-09-11T06:26:11.815 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=111, tag_tid=20, entry_tid=11811' '!=' 'object_number=108, tag_tid=20, entry_tid=11728' 2024-09-11T06:26:11.815 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:26:11.815 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=111, tag_tid=20, entry_tid=11811' = 'object_number=108, tag_tid=20, entry_tid=11728' 2024-09-11T06:26:11.815 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=111, tag_tid=20, entry_tid=11811' 2024-09-11T06:26:11.816 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:11.816 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:11.816 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:11.817 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:26:11.818 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=111, tag_tid=20, entry_tid=11811' 2024-09-11T06:26:11.818 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:11.818 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:11.819 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:26:11.820 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:11.820 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:11.820 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:11.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:26:11.822 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=111, tag_tid=20, entry_tid=11811' 2024-09-11T06:26:11.822 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:11.822 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:11.824 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=11811 2024-09-11T06:26:11.824 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 11811 2024-09-11T06:26:11.824 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:26:11.824 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:26:11.825 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:26:12.026 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:26:12.026 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:12.026 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:26:12.026 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:26:12.026 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:26:12.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:26:12.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:26:12.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:26:12.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:26:12.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:12.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:12.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:26:12.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:26:12.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:26:12.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:26:12.028 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:26:12.028 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:26:12.031 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:26:12.031 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:26:12.031 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:26:12.032 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:12.032 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:26:12.032 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:26:12.032 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:26:12.032 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:26:12.032 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:26:12.032 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:26:12.032 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:26:12.032 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:12.032 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:26:12.032 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:12.032 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:26:12.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:12.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:26:12.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:26:12.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:26:12.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:26:12.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:26:12.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:26:12.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:26:12.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:26:12.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:12.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:12.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:12.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:26:12.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:26:12.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:26:12.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:26:12.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:26:12.056 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:12.057 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:12.057 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:12.057 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:26:12.185 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:26:12.186 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:26:12.186 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:12.186 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:12.186 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:12.186 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:26:12.187 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:12.187 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:12.187 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:12.187 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:26:12.187 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:12.187 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:12.187 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:12.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:12.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:12.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:12.236 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 27 2024-09-11T06:26:12.236 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:12.236 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:12.236 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:12.236 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=110, tag_tid=20, entry_tid=11926], [object_number=109, tag_tid=20, entry_tid=11925], [object_number=108, tag_tid=20, entry_tid=11924], [object_number=111, tag_tid=20, entry_tid=11923]]], state=connected] 2024-09-11T06:26:12.239 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:12.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:12.245 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:26:12.245 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:12.245 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:12.245 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:12.245 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:26:12.245 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:12.246 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:12.246 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:12.246 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:26:12.246 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:12.246 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:12.246 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:12.246 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:12.246 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:12.246 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:12.291 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 27 2024-09-11T06:26:12.291 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:12.291 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:12.291 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:12.291 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=110, tag_tid=20, entry_tid=11926], [object_number=109, tag_tid=20, entry_tid=11925], [object_number=108, tag_tid=20, entry_tid=11924], [object_number=111, tag_tid=20, entry_tid=11923]]], state=connected] 2024-09-11T06:26:12.294 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:12.296 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=110, tag_tid=20, entry_tid=11926' 2024-09-11T06:26:12.296 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=110, tag_tid=20, entry_tid=11926' 2024-09-11T06:26:12.296 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=110, tag_tid=20, entry_tid=11926' '!=' 'object_number=111, tag_tid=20, entry_tid=11811' 2024-09-11T06:26:12.296 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:26:12.296 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=110, tag_tid=20, entry_tid=11926' = 'object_number=111, tag_tid=20, entry_tid=11811' 2024-09-11T06:26:12.296 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=110, tag_tid=20, entry_tid=11926' 2024-09-11T06:26:12.296 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:12.296 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:12.297 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:12.297 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:26:12.298 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=110, tag_tid=20, entry_tid=11926' 2024-09-11T06:26:12.298 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:12.298 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:12.298 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:26:12.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:12.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:12.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:12.300 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:26:12.301 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=110, tag_tid=20, entry_tid=11926' 2024-09-11T06:26:12.301 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:12.301 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:12.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=11926 2024-09-11T06:26:12.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 11926 2024-09-11T06:26:12.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:26:12.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:26:12.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:26:12.504 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:26:12.504 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:12.504 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:26:12.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:26:12.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:26:12.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:26:12.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:26:12.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:26:12.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:26:12.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:12.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:12.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:26:12.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:26:12.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:26:12.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:26:12.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:26:12.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:26:12.509 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:26:12.509 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:26:12.509 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:26:12.510 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:12.510 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:26:12.510 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:26:12.510 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:26:12.510 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:26:12.510 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:26:12.510 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:26:12.510 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:26:12.510 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:12.510 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:26:12.511 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:12.511 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:26:12.511 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:12.511 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:26:12.511 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:26:12.511 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:26:12.511 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:26:12.511 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:26:12.511 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:26:12.511 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:26:12.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:26:12.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:12.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:12.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:12.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:26:12.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:26:12.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:26:12.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:26:12.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:26:12.535 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:12.535 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:12.535 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:12.536 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:26:12.669 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:26:12.671 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:26:12.671 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:12.671 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:12.671 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:12.672 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:26:12.672 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:12.672 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:12.672 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:12.672 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:26:12.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:12.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:12.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:12.674 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:12.674 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:12.674 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:12.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 27 2024-09-11T06:26:12.721 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:12.721 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:12.721 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:12.721 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=108, tag_tid=20, entry_tid=12016], [object_number=111, tag_tid=20, entry_tid=12015], [object_number=110, tag_tid=20, entry_tid=12014], [object_number=109, tag_tid=20, entry_tid=12013]]], state=connected] 2024-09-11T06:26:12.724 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:12.725 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:12.726 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:26:12.726 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:12.726 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:12.726 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:12.726 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:26:12.726 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:12.726 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:12.727 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:12.727 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:26:12.727 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:12.727 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:12.727 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:12.727 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:12.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:12.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:12.774 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 27 2024-09-11T06:26:12.774 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:12.774 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:12.774 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:12.774 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=108, tag_tid=20, entry_tid=12016], [object_number=111, tag_tid=20, entry_tid=12015], [object_number=110, tag_tid=20, entry_tid=12014], [object_number=109, tag_tid=20, entry_tid=12013]]], state=connected] 2024-09-11T06:26:12.778 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:12.780 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=108, tag_tid=20, entry_tid=12016' 2024-09-11T06:26:12.780 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=108, tag_tid=20, entry_tid=12016' 2024-09-11T06:26:12.780 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=108, tag_tid=20, entry_tid=12016' '!=' 'object_number=110, tag_tid=20, entry_tid=11926' 2024-09-11T06:26:12.780 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:26:12.780 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=108, tag_tid=20, entry_tid=12016' = 'object_number=110, tag_tid=20, entry_tid=11926' 2024-09-11T06:26:12.780 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=108, tag_tid=20, entry_tid=12016' 2024-09-11T06:26:12.780 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:12.781 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:12.781 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:12.782 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:26:12.782 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=108, tag_tid=20, entry_tid=12016' 2024-09-11T06:26:12.782 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:12.782 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:12.783 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:26:12.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:12.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:12.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:12.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:26:12.785 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=108, tag_tid=20, entry_tid=12016' 2024-09-11T06:26:12.785 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:12.785 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:12.787 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=12016 2024-09-11T06:26:12.787 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 12016 2024-09-11T06:26:12.787 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:26:12.787 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:26:12.787 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:26:12.989 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:26:12.989 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:12.989 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:26:12.989 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:26:12.989 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:26:12.989 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:26:12.990 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:26:12.990 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:26:12.990 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:26:12.990 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:12.990 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:12.990 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:26:12.990 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:26:12.990 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:26:12.990 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:26:12.991 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:26:12.991 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:26:12.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:26:12.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:26:12.994 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:26:12.994 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:12.994 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:26:12.994 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:26:12.994 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:26:12.994 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:26:12.994 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:26:12.994 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:26:12.994 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:26:12.994 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:12.994 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:26:12.994 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:12.994 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:26:12.995 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:12.995 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:26:12.995 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:26:12.995 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:26:12.995 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:26:12.995 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:26:12.995 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:26:12.995 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:26:12.995 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:26:12.995 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:12.995 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:12.995 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:12.995 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:26:12.995 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:26:12.996 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:26:12.996 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:26:12.996 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:26:13.018 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:13.019 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:13.019 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:13.019 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:26:13.146 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:26:13.147 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:26:13.147 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:13.147 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:13.147 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:13.148 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:26:13.148 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:13.148 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:13.148 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:13.148 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:26:13.148 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:13.149 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:13.149 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:13.150 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:13.150 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:13.150 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:13.202 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 27 2024-09-11T06:26:13.203 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:13.203 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:13.203 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:13.203 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=110, tag_tid=20, entry_tid=12130], [object_number=109, tag_tid=20, entry_tid=12129], [object_number=108, tag_tid=20, entry_tid=12128], [object_number=111, tag_tid=20, entry_tid=12127]]], state=connected] 2024-09-11T06:26:13.206 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:13.208 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:13.209 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:26:13.209 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:13.209 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:13.209 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:13.210 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:26:13.210 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:13.210 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:13.210 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:13.210 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:26:13.210 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:13.210 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:13.210 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:13.211 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:13.211 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:13.211 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:13.259 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 27 2024-09-11T06:26:13.259 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:13.259 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:13.260 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:13.260 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=110, tag_tid=20, entry_tid=12130], [object_number=109, tag_tid=20, entry_tid=12129], [object_number=108, tag_tid=20, entry_tid=12128], [object_number=111, tag_tid=20, entry_tid=12127]]], state=connected] 2024-09-11T06:26:13.262 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:13.265 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=110, tag_tid=20, entry_tid=12130' 2024-09-11T06:26:13.265 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=110, tag_tid=20, entry_tid=12130' 2024-09-11T06:26:13.265 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=110, tag_tid=20, entry_tid=12130' '!=' 'object_number=108, tag_tid=20, entry_tid=12016' 2024-09-11T06:26:13.266 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:26:13.266 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=110, tag_tid=20, entry_tid=12130' = 'object_number=108, tag_tid=20, entry_tid=12016' 2024-09-11T06:26:13.266 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=110, tag_tid=20, entry_tid=12130' 2024-09-11T06:26:13.266 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:13.266 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:13.266 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:13.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:26:13.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=110, tag_tid=20, entry_tid=12130' 2024-09-11T06:26:13.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:13.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:13.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:26:13.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:13.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:13.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:13.269 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:26:13.270 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=110, tag_tid=20, entry_tid=12130' 2024-09-11T06:26:13.270 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:13.270 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:13.271 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=12130 2024-09-11T06:26:13.271 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 12130 2024-09-11T06:26:13.271 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:26:13.271 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:26:13.271 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:26:13.473 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:26:13.473 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:13.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:26:13.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:26:13.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:26:13.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:26:13.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:26:13.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:26:13.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:26:13.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:13.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:13.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:26:13.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:26:13.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:26:13.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:26:13.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:26:13.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:26:13.478 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:26:13.478 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:26:13.478 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:26:13.478 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:13.478 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:26:13.479 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:26:13.479 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:26:13.479 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:26:13.479 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:26:13.479 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:26:13.479 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:26:13.479 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:13.479 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:26:13.479 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:13.479 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:26:13.479 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:13.479 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:26:13.479 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:26:13.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:26:13.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:26:13.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:26:13.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:26:13.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:26:13.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:26:13.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:13.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:13.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:13.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:26:13.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:26:13.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:26:13.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:26:13.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:26:13.504 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:13.504 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:13.504 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:13.504 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:26:13.635 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:26:13.636 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:26:13.636 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:13.636 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:13.636 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:13.636 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:26:13.636 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:13.636 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:13.636 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:13.637 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:26:13.637 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:13.637 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:13.637 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:13.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:13.640 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:13.640 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:13.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 27 2024-09-11T06:26:13.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:13.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:13.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:13.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=108, tag_tid=20, entry_tid=12236], [object_number=111, tag_tid=20, entry_tid=12235], [object_number=110, tag_tid=20, entry_tid=12234], [object_number=109, tag_tid=20, entry_tid=12233]]], state=connected] 2024-09-11T06:26:13.731 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:13.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:13.734 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:26:13.734 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:13.734 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:13.734 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:13.734 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:26:13.734 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:13.734 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:13.734 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:13.735 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:26:13.735 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:13.735 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:13.735 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:13.735 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:13.736 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:13.736 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:13.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 27 2024-09-11T06:26:13.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:13.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:13.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:13.785 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=108, tag_tid=20, entry_tid=12236], [object_number=111, tag_tid=20, entry_tid=12235], [object_number=110, tag_tid=20, entry_tid=12234], [object_number=109, tag_tid=20, entry_tid=12233]]], state=connected] 2024-09-11T06:26:13.787 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:13.790 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=108, tag_tid=20, entry_tid=12236' 2024-09-11T06:26:13.790 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=108, tag_tid=20, entry_tid=12236' 2024-09-11T06:26:13.790 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=108, tag_tid=20, entry_tid=12236' '!=' 'object_number=110, tag_tid=20, entry_tid=12130' 2024-09-11T06:26:13.790 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:26:13.790 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=108, tag_tid=20, entry_tid=12236' = 'object_number=110, tag_tid=20, entry_tid=12130' 2024-09-11T06:26:13.790 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=108, tag_tid=20, entry_tid=12236' 2024-09-11T06:26:13.790 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:13.790 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:13.790 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:13.791 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:26:13.792 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=108, tag_tid=20, entry_tid=12236' 2024-09-11T06:26:13.792 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:13.792 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:13.793 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:26:13.793 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:13.793 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:13.793 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:13.794 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:26:13.795 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=108, tag_tid=20, entry_tid=12236' 2024-09-11T06:26:13.795 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:13.795 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:13.797 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=12236 2024-09-11T06:26:13.797 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 12236 2024-09-11T06:26:13.797 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:26:13.797 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:26:13.797 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:26:13.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:26:13.999 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:13.999 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:26:13.999 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:26:13.999 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:26:13.999 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:26:13.999 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:26:13.999 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:26:13.999 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:26:13.999 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:13.999 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:13.999 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:26:13.999 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:26:14.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:26:14.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:26:14.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:26:14.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:26:14.004 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:26:14.004 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:26:14.004 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:26:14.004 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:14.004 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:26:14.004 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:26:14.004 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:26:14.004 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:26:14.004 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:26:14.004 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:26:14.005 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:26:14.005 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:14.005 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:26:14.005 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:14.005 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:26:14.005 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:14.005 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:26:14.005 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:26:14.005 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:26:14.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:26:14.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:26:14.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:26:14.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:26:14.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:26:14.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:14.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:14.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:14.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:26:14.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:26:14.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:26:14.007 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:26:14.007 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:26:14.030 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:14.031 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:14.031 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:14.031 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:26:14.163 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:26:14.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:26:14.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:14.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:14.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:14.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:26:14.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:14.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:14.166 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:14.166 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:26:14.166 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:14.166 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:14.166 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:14.167 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:14.168 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:14.168 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:14.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 27 2024-09-11T06:26:14.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:14.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:14.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:14.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=110, tag_tid=20, entry_tid=12334], [object_number=109, tag_tid=20, entry_tid=12333], [object_number=108, tag_tid=20, entry_tid=12332], [object_number=111, tag_tid=20, entry_tid=12331]]], state=connected] 2024-09-11T06:26:14.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:14.222 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:14.223 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:26:14.223 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:14.223 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:14.223 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:14.223 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:26:14.223 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:14.223 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:14.223 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:14.224 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:26:14.224 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:14.224 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:14.224 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:14.225 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:14.225 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:14.225 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:14.273 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 27 2024-09-11T06:26:14.273 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:14.273 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:14.273 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:14.273 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=110, tag_tid=20, entry_tid=12334], [object_number=109, tag_tid=20, entry_tid=12333], [object_number=108, tag_tid=20, entry_tid=12332], [object_number=111, tag_tid=20, entry_tid=12331]]], state=connected] 2024-09-11T06:26:14.276 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:14.279 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=110, tag_tid=20, entry_tid=12334' 2024-09-11T06:26:14.279 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=110, tag_tid=20, entry_tid=12334' 2024-09-11T06:26:14.279 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=110, tag_tid=20, entry_tid=12334' '!=' 'object_number=108, tag_tid=20, entry_tid=12236' 2024-09-11T06:26:14.279 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:26:14.279 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=110, tag_tid=20, entry_tid=12334' = 'object_number=108, tag_tid=20, entry_tid=12236' 2024-09-11T06:26:14.279 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=110, tag_tid=20, entry_tid=12334' 2024-09-11T06:26:14.279 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:14.279 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:14.279 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:14.280 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:26:14.281 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=110, tag_tid=20, entry_tid=12334' 2024-09-11T06:26:14.281 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:14.281 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:14.281 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:26:14.282 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:14.282 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:14.282 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:14.283 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:26:14.284 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=110, tag_tid=20, entry_tid=12334' 2024-09-11T06:26:14.284 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:14.284 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:14.285 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=12334 2024-09-11T06:26:14.285 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 12334 2024-09-11T06:26:14.285 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:26:14.286 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:26:14.286 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:26:14.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:26:14.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:14.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:26:14.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:26:14.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:26:14.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:26:14.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:26:14.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:26:14.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:26:14.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:14.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:14.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:26:14.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:26:14.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:26:14.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:26:14.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:26:14.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:26:14.493 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:26:14.493 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:26:14.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:26:14.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:14.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:26:14.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:26:14.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:26:14.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:26:14.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:26:14.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:26:14.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:26:14.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:14.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:26:14.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:14.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:26:14.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:14.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:26:14.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:26:14.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:26:14.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:26:14.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:26:14.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:26:14.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:26:14.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:26:14.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:14.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:14.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:14.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:26:14.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:26:14.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:26:14.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:26:14.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:26:14.519 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:14.519 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:14.519 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:14.519 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:26:14.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:26:14.654 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:26:14.654 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:14.655 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:14.655 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:14.655 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:26:14.655 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:14.655 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:14.655 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:14.655 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:26:14.655 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:14.656 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:14.656 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:14.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:14.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:14.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:14.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 27 2024-09-11T06:26:14.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:14.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:14.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:14.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=110, tag_tid=20, entry_tid=12422], [object_number=109, tag_tid=20, entry_tid=12421], [object_number=108, tag_tid=20, entry_tid=12420], [object_number=111, tag_tid=20, entry_tid=12419]]], state=connected] 2024-09-11T06:26:14.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:14.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:14.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:26:14.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:14.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:14.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:14.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:26:14.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:14.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:14.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:14.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:26:14.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:14.721 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:14.721 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:14.721 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:14.721 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:14.721 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:14.769 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 27 2024-09-11T06:26:14.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:14.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:14.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:14.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=110, tag_tid=20, entry_tid=12422], [object_number=109, tag_tid=20, entry_tid=12421], [object_number=108, tag_tid=20, entry_tid=12420], [object_number=111, tag_tid=20, entry_tid=12419]]], state=connected] 2024-09-11T06:26:14.773 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:14.775 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=110, tag_tid=20, entry_tid=12422' 2024-09-11T06:26:14.775 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=110, tag_tid=20, entry_tid=12422' 2024-09-11T06:26:14.775 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=110, tag_tid=20, entry_tid=12422' '!=' 'object_number=110, tag_tid=20, entry_tid=12334' 2024-09-11T06:26:14.775 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:26:14.776 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=110, tag_tid=20, entry_tid=12422' = 'object_number=110, tag_tid=20, entry_tid=12334' 2024-09-11T06:26:14.776 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=110, tag_tid=20, entry_tid=12422' 2024-09-11T06:26:14.776 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:14.776 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:14.776 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:14.776 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:26:14.777 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=110, tag_tid=20, entry_tid=12422' 2024-09-11T06:26:14.777 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:14.777 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:14.778 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:26:14.779 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:14.779 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:14.779 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:14.780 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:26:14.781 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=110, tag_tid=20, entry_tid=12422' 2024-09-11T06:26:14.781 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:14.781 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:14.782 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=12422 2024-09-11T06:26:14.782 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 12422 2024-09-11T06:26:14.782 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:26:14.782 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:26:14.782 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:26:14.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:26:14.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:14.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:26:14.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:26:14.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:26:14.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:26:14.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:26:14.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:26:14.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:26:14.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:14.985 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:14.985 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:26:14.985 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:26:14.985 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:26:14.985 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:26:14.985 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:26:14.986 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:26:14.990 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:26:14.990 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:26:14.990 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:26:14.990 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:14.991 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:26:14.991 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:26:14.991 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:26:14.991 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:26:14.991 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:26:14.991 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:26:14.992 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:26:14.992 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:14.992 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:26:14.992 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:14.992 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:26:14.992 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:14.992 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:26:14.992 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:26:14.992 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:26:14.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:26:14.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:26:14.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:26:14.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:26:14.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:26:14.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:14.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:14.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:14.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:26:14.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:26:14.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:26:14.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:26:14.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:26:15.015 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:15.015 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:15.016 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:15.016 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:26:15.146 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:26:15.148 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:26:15.148 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:15.148 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:15.148 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:15.148 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:26:15.148 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:15.148 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:15.148 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:15.148 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:26:15.148 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:15.149 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:15.149 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:15.150 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:15.151 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:15.151 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:15.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 27 2024-09-11T06:26:15.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:15.199 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:15.199 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:15.199 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=109, tag_tid=20, entry_tid=12509], [object_number=108, tag_tid=20, entry_tid=12508], [object_number=111, tag_tid=20, entry_tid=12507], [object_number=110, tag_tid=20, entry_tid=12506]]], state=connected] 2024-09-11T06:26:15.202 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:15.203 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:15.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:26:15.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:15.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:15.205 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:15.205 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:26:15.205 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:15.205 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:15.205 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:15.205 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:26:15.205 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:15.205 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:15.205 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:15.206 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:15.206 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:15.206 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:15.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 27 2024-09-11T06:26:15.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:15.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:15.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:15.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=109, tag_tid=20, entry_tid=12509], [object_number=108, tag_tid=20, entry_tid=12508], [object_number=111, tag_tid=20, entry_tid=12507], [object_number=110, tag_tid=20, entry_tid=12506]]], state=connected] 2024-09-11T06:26:15.257 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:15.258 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=109, tag_tid=20, entry_tid=12509' 2024-09-11T06:26:15.258 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=109, tag_tid=20, entry_tid=12509' 2024-09-11T06:26:15.258 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=109, tag_tid=20, entry_tid=12509' '!=' 'object_number=110, tag_tid=20, entry_tid=12422' 2024-09-11T06:26:15.259 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:26:15.259 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=109, tag_tid=20, entry_tid=12509' = 'object_number=110, tag_tid=20, entry_tid=12422' 2024-09-11T06:26:15.259 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=109, tag_tid=20, entry_tid=12509' 2024-09-11T06:26:15.259 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:15.259 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:15.259 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:15.260 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:26:15.260 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=109, tag_tid=20, entry_tid=12509' 2024-09-11T06:26:15.261 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:15.261 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:15.261 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:26:15.262 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:15.262 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:15.262 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:15.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:26:15.264 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=109, tag_tid=20, entry_tid=12509' 2024-09-11T06:26:15.264 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:15.264 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:15.265 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=12509 2024-09-11T06:26:15.265 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 12509 2024-09-11T06:26:15.265 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:26:15.266 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:26:15.266 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:26:15.467 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:26:15.468 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:15.468 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:26:15.468 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:26:15.468 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:26:15.468 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:26:15.468 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:26:15.468 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:26:15.468 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:26:15.468 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:15.468 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:15.469 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:26:15.469 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:26:15.469 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:26:15.469 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:26:15.469 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:26:15.470 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:26:15.473 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:26:15.473 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:26:15.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:26:15.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:15.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:26:15.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:26:15.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:26:15.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:26:15.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:26:15.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:26:15.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:26:15.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:15.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:26:15.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:15.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:26:15.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:15.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:26:15.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:26:15.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:26:15.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:26:15.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:26:15.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:26:15.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:26:15.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:26:15.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:15.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:15.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:15.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:26:15.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:26:15.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:26:15.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:26:15.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:26:15.498 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:15.499 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:15.499 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:15.499 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:26:15.633 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:26:15.634 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:26:15.634 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:15.635 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:15.635 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:15.635 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:26:15.635 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:15.635 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:15.635 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:15.635 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:26:15.635 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:15.636 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:15.636 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:15.637 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:15.638 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:15.638 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:15.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 27 2024-09-11T06:26:15.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:15.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:15.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:15.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=110, tag_tid=20, entry_tid=12606], [object_number=109, tag_tid=20, entry_tid=12605], [object_number=108, tag_tid=20, entry_tid=12604], [object_number=111, tag_tid=20, entry_tid=12603]]], state=connected] 2024-09-11T06:26:15.694 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:15.696 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:15.697 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:26:15.697 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:15.697 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:15.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:15.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:26:15.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:15.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:15.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:15.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:26:15.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:15.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:15.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:15.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:15.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:15.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:15.746 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 27 2024-09-11T06:26:15.746 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:15.746 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:15.746 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:15.746 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=110, tag_tid=20, entry_tid=12606], [object_number=109, tag_tid=20, entry_tid=12605], [object_number=108, tag_tid=20, entry_tid=12604], [object_number=111, tag_tid=20, entry_tid=12603]]], state=connected] 2024-09-11T06:26:15.750 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:15.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=110, tag_tid=20, entry_tid=12606' 2024-09-11T06:26:15.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=110, tag_tid=20, entry_tid=12606' 2024-09-11T06:26:15.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=110, tag_tid=20, entry_tid=12606' '!=' 'object_number=109, tag_tid=20, entry_tid=12509' 2024-09-11T06:26:15.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:26:15.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=110, tag_tid=20, entry_tid=12606' = 'object_number=109, tag_tid=20, entry_tid=12509' 2024-09-11T06:26:15.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=110, tag_tid=20, entry_tid=12606' 2024-09-11T06:26:15.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:15.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:15.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:15.753 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:26:15.753 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=110, tag_tid=20, entry_tid=12606' 2024-09-11T06:26:15.753 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:15.754 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:15.754 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:26:15.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:15.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:15.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:15.756 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:26:15.757 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=110, tag_tid=20, entry_tid=12606' 2024-09-11T06:26:15.757 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:15.757 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:15.758 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=12606 2024-09-11T06:26:15.758 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 12606 2024-09-11T06:26:15.758 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:26:15.758 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:26:15.758 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:26:15.960 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:26:15.960 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:15.960 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:26:15.960 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:26:15.961 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:26:15.961 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:26:15.961 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:26:15.961 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:26:15.961 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:26:15.961 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:15.961 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:15.961 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:26:15.961 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:26:15.961 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:26:15.961 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:26:15.962 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:26:15.962 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:26:15.965 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:26:15.965 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:26:15.965 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:26:15.965 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:15.965 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:26:15.965 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:26:15.965 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:26:15.965 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:26:15.965 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:26:15.965 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:26:15.965 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:26:15.965 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:15.965 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:26:15.965 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:15.966 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:26:15.966 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:15.966 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:26:15.966 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:26:15.966 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:26:15.966 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:26:15.966 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:26:15.966 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:26:15.966 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:26:15.966 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:26:15.966 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:15.966 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:15.966 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:15.966 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:26:15.966 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:26:15.967 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:26:15.967 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:26:15.967 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:26:15.991 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:15.991 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:15.991 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:15.991 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:26:16.122 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:26:16.123 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:26:16.123 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:16.123 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:16.123 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:16.123 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:26:16.123 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:16.123 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:16.124 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:16.124 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:26:16.124 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:16.124 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:16.124 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:16.129 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:16.129 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:16.129 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:16.173 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 27 2024-09-11T06:26:16.173 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:16.173 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:16.173 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:16.174 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=109, tag_tid=20, entry_tid=12701], [object_number=108, tag_tid=20, entry_tid=12700], [object_number=111, tag_tid=20, entry_tid=12699], [object_number=110, tag_tid=20, entry_tid=12698]]], state=connected] 2024-09-11T06:26:16.177 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:16.178 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:16.179 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:26:16.179 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:16.180 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:16.180 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:16.180 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:26:16.180 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:16.180 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:16.180 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:16.180 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:26:16.180 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:16.180 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:16.181 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:16.182 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:16.182 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:16.182 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:16.227 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 27 2024-09-11T06:26:16.227 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:16.227 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:16.228 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:16.228 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=109, tag_tid=20, entry_tid=12701], [object_number=108, tag_tid=20, entry_tid=12700], [object_number=111, tag_tid=20, entry_tid=12699], [object_number=110, tag_tid=20, entry_tid=12698]]], state=connected] 2024-09-11T06:26:16.231 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:16.232 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=109, tag_tid=20, entry_tid=12701' 2024-09-11T06:26:16.232 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=109, tag_tid=20, entry_tid=12701' 2024-09-11T06:26:16.233 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=109, tag_tid=20, entry_tid=12701' '!=' 'object_number=110, tag_tid=20, entry_tid=12606' 2024-09-11T06:26:16.233 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:26:16.233 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=109, tag_tid=20, entry_tid=12701' = 'object_number=110, tag_tid=20, entry_tid=12606' 2024-09-11T06:26:16.233 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=109, tag_tid=20, entry_tid=12701' 2024-09-11T06:26:16.233 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:16.234 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:16.234 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:16.234 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:26:16.235 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=109, tag_tid=20, entry_tid=12701' 2024-09-11T06:26:16.235 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:16.235 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:16.235 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:26:16.236 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:16.236 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:16.236 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:16.237 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:26:16.238 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=109, tag_tid=20, entry_tid=12701' 2024-09-11T06:26:16.238 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:16.238 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:16.239 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=12701 2024-09-11T06:26:16.239 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 12701 2024-09-11T06:26:16.239 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:26:16.240 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:26:16.240 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:26:16.441 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:26:16.442 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:16.442 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:26:16.442 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:26:16.442 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:26:16.442 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:26:16.442 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:26:16.442 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:26:16.442 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:26:16.442 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:16.442 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:16.442 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:26:16.442 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:26:16.443 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:26:16.443 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:26:16.446 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:26:16.446 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:26:16.446 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:26:16.446 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:26:16.446 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:26:16.446 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:16.446 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:26:16.446 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:26:16.446 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:26:16.446 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:26:16.446 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:26:16.446 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:26:16.447 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:26:16.447 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:16.447 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:26:16.447 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:16.447 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:26:16.447 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:16.447 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:26:16.447 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:26:16.447 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:26:16.447 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:26:16.447 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:26:16.447 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:26:16.447 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:26:16.448 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:26:16.448 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:16.448 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:16.448 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:16.448 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:26:16.448 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:26:16.448 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:26:16.448 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:26:16.448 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:26:16.471 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:16.471 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:16.471 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:16.471 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:26:16.603 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:26:16.604 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:26:16.605 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:16.605 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:16.605 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:16.605 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:26:16.605 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:16.605 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:16.605 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:16.605 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:26:16.606 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:16.606 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:16.606 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:16.608 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:16.609 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:16.609 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:16.865 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 27 2024-09-11T06:26:16.865 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:16.865 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:16.865 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:16.865 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=111, tag_tid=20, entry_tid=12799], [object_number=110, tag_tid=20, entry_tid=12798], [object_number=109, tag_tid=20, entry_tid=12797], [object_number=108, tag_tid=20, entry_tid=12796]]], state=connected] 2024-09-11T06:26:16.871 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:16.873 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:16.874 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:26:16.874 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:16.874 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:16.874 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:16.874 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:26:16.875 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:16.875 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:16.875 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:16.875 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:26:16.875 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:16.875 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:16.875 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:16.877 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:16.877 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:16.877 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:16.934 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 27 2024-09-11T06:26:16.934 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:16.935 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:16.935 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:16.935 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=111, tag_tid=20, entry_tid=12799], [object_number=110, tag_tid=20, entry_tid=12798], [object_number=109, tag_tid=20, entry_tid=12797], [object_number=108, tag_tid=20, entry_tid=12796]]], state=connected] 2024-09-11T06:26:16.938 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:16.940 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=111, tag_tid=20, entry_tid=12799' 2024-09-11T06:26:16.940 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=111, tag_tid=20, entry_tid=12799' 2024-09-11T06:26:16.940 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=111, tag_tid=20, entry_tid=12799' '!=' 'object_number=109, tag_tid=20, entry_tid=12701' 2024-09-11T06:26:16.940 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:26:16.940 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=111, tag_tid=20, entry_tid=12799' = 'object_number=109, tag_tid=20, entry_tid=12701' 2024-09-11T06:26:16.941 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=111, tag_tid=20, entry_tid=12799' 2024-09-11T06:26:16.941 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:16.941 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:16.941 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:16.941 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:26:16.942 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=111, tag_tid=20, entry_tid=12799' 2024-09-11T06:26:16.942 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:16.942 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:16.943 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:26:16.944 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:16.944 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:16.944 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:16.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:26:16.946 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=111, tag_tid=20, entry_tid=12799' 2024-09-11T06:26:16.946 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:16.946 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:16.947 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=12799 2024-09-11T06:26:16.947 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 12799 2024-09-11T06:26:16.947 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:26:16.948 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:26:16.948 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:26:17.150 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:26:17.150 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:17.150 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:26:17.150 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:26:17.150 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:26:17.150 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:26:17.150 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:26:17.150 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:26:17.150 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:26:17.151 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:17.151 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:17.151 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:26:17.151 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:26:17.151 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:26:17.151 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:26:17.151 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:26:17.152 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:26:17.157 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:26:17.157 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:26:17.157 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:26:17.157 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:17.157 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:26:17.157 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:26:17.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:26:17.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:26:17.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:26:17.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:26:17.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:26:17.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:17.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:26:17.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:17.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:26:17.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:17.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:26:17.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:26:17.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:26:17.159 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:26:17.159 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:26:17.159 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:26:17.159 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:26:17.159 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:26:17.159 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:17.159 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:17.159 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:17.159 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:26:17.159 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:26:17.159 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:26:17.159 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:26:17.159 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:26:17.180 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:17.181 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:17.181 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:17.181 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:26:17.317 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:26:17.318 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:26:17.318 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:17.318 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:17.318 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:17.318 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:26:17.318 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:17.318 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:17.318 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:17.318 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:26:17.319 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:17.319 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:17.319 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:17.321 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:17.322 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:17.322 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:17.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 27 2024-09-11T06:26:17.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:17.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:17.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:17.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=109, tag_tid=20, entry_tid=12921], [object_number=108, tag_tid=20, entry_tid=12920], [object_number=111, tag_tid=20, entry_tid=12919], [object_number=110, tag_tid=20, entry_tid=12918]]], state=connected] 2024-09-11T06:26:17.373 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:17.376 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:17.376 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:26:17.376 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:17.376 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:17.376 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:17.376 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:26:17.376 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:17.376 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:17.376 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:17.376 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:26:17.376 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:17.376 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:17.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:17.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:17.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:17.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:17.424 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 27 2024-09-11T06:26:17.424 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:17.424 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:17.424 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:17.424 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=109, tag_tid=20, entry_tid=12921], [object_number=108, tag_tid=20, entry_tid=12920], [object_number=111, tag_tid=20, entry_tid=12919], [object_number=110, tag_tid=20, entry_tid=12918]]], state=connected] 2024-09-11T06:26:17.427 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:17.429 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=109, tag_tid=20, entry_tid=12921' 2024-09-11T06:26:17.429 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=109, tag_tid=20, entry_tid=12921' 2024-09-11T06:26:17.429 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=109, tag_tid=20, entry_tid=12921' '!=' 'object_number=111, tag_tid=20, entry_tid=12799' 2024-09-11T06:26:17.429 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:26:17.429 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=109, tag_tid=20, entry_tid=12921' = 'object_number=111, tag_tid=20, entry_tid=12799' 2024-09-11T06:26:17.429 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=109, tag_tid=20, entry_tid=12921' 2024-09-11T06:26:17.430 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:17.430 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:17.430 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:17.431 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:26:17.431 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=109, tag_tid=20, entry_tid=12921' 2024-09-11T06:26:17.432 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:17.432 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:17.433 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:26:17.433 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:17.433 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:17.433 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:17.434 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:26:17.435 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=109, tag_tid=20, entry_tid=12921' 2024-09-11T06:26:17.435 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:17.435 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:17.436 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=12921 2024-09-11T06:26:17.436 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 12921 2024-09-11T06:26:17.436 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:26:17.437 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:26:17.437 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:26:17.638 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:26:17.638 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:17.638 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:26:17.638 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:26:17.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:26:17.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:26:17.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:26:17.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:26:17.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:26:17.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:17.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:17.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:26:17.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:26:17.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:26:17.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:26:17.640 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:26:17.640 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:26:17.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:26:17.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:26:17.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:26:17.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:17.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:26:17.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:26:17.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:26:17.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:26:17.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:26:17.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:26:17.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:26:17.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:17.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:26:17.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:17.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:26:17.645 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:17.645 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:26:17.645 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:26:17.645 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:26:17.645 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:26:17.645 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:26:17.645 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:26:17.645 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:26:17.646 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:26:17.646 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:17.646 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:17.646 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:17.646 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:26:17.646 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:26:17.646 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:26:17.646 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:26:17.646 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:26:17.669 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:17.669 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:17.669 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:17.669 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:26:17.797 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:26:17.798 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:26:17.798 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:17.798 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:17.798 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:17.798 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:26:17.798 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:17.799 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:17.799 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:17.799 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:26:17.799 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:17.799 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:17.799 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:17.802 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:17.802 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:17.802 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:17.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 27 2024-09-11T06:26:17.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:17.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:17.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:17.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=109, tag_tid=20, entry_tid=13013], [object_number=108, tag_tid=20, entry_tid=13012], [object_number=111, tag_tid=20, entry_tid=13011], [object_number=110, tag_tid=20, entry_tid=13010]]], state=connected] 2024-09-11T06:26:17.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:17.854 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:17.855 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:26:17.855 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:17.855 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:17.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:17.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:26:17.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:17.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:17.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:17.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:26:17.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:17.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:17.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:17.857 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:17.857 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:17.857 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:17.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 27 2024-09-11T06:26:17.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:17.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:17.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:17.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=109, tag_tid=20, entry_tid=13013], [object_number=108, tag_tid=20, entry_tid=13012], [object_number=111, tag_tid=20, entry_tid=13011], [object_number=110, tag_tid=20, entry_tid=13010]]], state=connected] 2024-09-11T06:26:17.907 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:17.909 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=109, tag_tid=20, entry_tid=13013' 2024-09-11T06:26:17.909 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=109, tag_tid=20, entry_tid=13013' 2024-09-11T06:26:17.910 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=109, tag_tid=20, entry_tid=13013' '!=' 'object_number=109, tag_tid=20, entry_tid=12921' 2024-09-11T06:26:17.910 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:26:17.910 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=109, tag_tid=20, entry_tid=13013' = 'object_number=109, tag_tid=20, entry_tid=12921' 2024-09-11T06:26:17.910 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=109, tag_tid=20, entry_tid=13013' 2024-09-11T06:26:17.910 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:17.910 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:17.910 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:17.910 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:26:17.911 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=109, tag_tid=20, entry_tid=13013' 2024-09-11T06:26:17.911 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:17.911 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:17.912 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:26:17.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:17.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:17.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:17.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:26:17.915 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=109, tag_tid=20, entry_tid=13013' 2024-09-11T06:26:17.915 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:17.915 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:17.916 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=13013 2024-09-11T06:26:17.916 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 13013 2024-09-11T06:26:17.916 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:26:17.916 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:26:17.916 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:26:18.118 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:26:18.118 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:18.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:26:18.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:26:18.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:26:18.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:26:18.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:26:18.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:26:18.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:26:18.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:18.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:18.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:26:18.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:26:18.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:26:18.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:26:18.120 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:26:18.120 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:26:18.123 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:26:18.123 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:26:18.123 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:26:18.123 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:18.123 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:26:18.123 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:26:18.123 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:26:18.123 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:26:18.123 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:26:18.123 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:26:18.124 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:26:18.124 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:18.124 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:26:18.124 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:18.124 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:26:18.124 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:18.124 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:26:18.124 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:26:18.124 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:26:18.124 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:26:18.124 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:26:18.124 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:26:18.124 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:26:18.124 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:26:18.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:18.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:18.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:18.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:26:18.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:26:18.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:26:18.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:26:18.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:26:18.148 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:18.148 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:18.148 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:18.148 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:26:18.285 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:26:18.286 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:26:18.286 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:18.286 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:18.286 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:18.286 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:26:18.286 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:18.287 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:18.287 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:18.287 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:26:18.288 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:18.288 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:18.288 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:18.289 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:18.290 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:18.290 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:18.345 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 27 2024-09-11T06:26:18.345 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:18.345 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:18.345 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:18.345 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=111, tag_tid=20, entry_tid=13111], [object_number=110, tag_tid=20, entry_tid=13110], [object_number=109, tag_tid=20, entry_tid=13109], [object_number=108, tag_tid=20, entry_tid=13108]]], state=connected] 2024-09-11T06:26:18.348 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:18.349 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:18.351 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:26:18.351 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:18.351 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:18.351 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:18.351 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:26:18.351 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:18.351 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:18.351 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:18.351 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:26:18.351 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:18.352 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:18.352 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:18.352 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:18.352 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:18.352 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:18.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 27 2024-09-11T06:26:18.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:18.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:18.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:18.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=111, tag_tid=20, entry_tid=13111], [object_number=110, tag_tid=20, entry_tid=13110], [object_number=109, tag_tid=20, entry_tid=13109], [object_number=108, tag_tid=20, entry_tid=13108]]], state=connected] 2024-09-11T06:26:18.403 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:18.405 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=111, tag_tid=20, entry_tid=13111' 2024-09-11T06:26:18.405 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=111, tag_tid=20, entry_tid=13111' 2024-09-11T06:26:18.405 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=111, tag_tid=20, entry_tid=13111' '!=' 'object_number=109, tag_tid=20, entry_tid=13013' 2024-09-11T06:26:18.405 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:26:18.405 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=111, tag_tid=20, entry_tid=13111' = 'object_number=109, tag_tid=20, entry_tid=13013' 2024-09-11T06:26:18.406 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=111, tag_tid=20, entry_tid=13111' 2024-09-11T06:26:18.406 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:18.406 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:18.406 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:18.406 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:26:18.407 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=111, tag_tid=20, entry_tid=13111' 2024-09-11T06:26:18.407 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:18.407 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:18.408 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:26:18.409 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:18.409 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:18.409 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:18.409 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:26:18.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=111, tag_tid=20, entry_tid=13111' 2024-09-11T06:26:18.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:18.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:18.412 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=13111 2024-09-11T06:26:18.412 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 13111 2024-09-11T06:26:18.412 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:26:18.412 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:26:18.412 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:26:18.614 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:26:18.614 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:18.614 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:26:18.614 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:26:18.614 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:26:18.614 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:26:18.614 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:26:18.614 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:26:18.614 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:26:18.614 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:18.615 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:18.615 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:26:18.615 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:26:18.615 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:26:18.615 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:26:18.615 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:26:18.615 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:26:18.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:26:18.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:26:18.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:26:18.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:18.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:26:18.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:26:18.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:26:18.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:26:18.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:26:18.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:26:18.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:26:18.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:18.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:26:18.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:18.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:26:18.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:18.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:26:18.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:26:18.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:26:18.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:26:18.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:26:18.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:26:18.621 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:26:18.621 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:26:18.621 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:18.621 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:18.621 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:18.621 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:26:18.621 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:26:18.621 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:26:18.621 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:26:18.621 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:26:18.645 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:18.645 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:18.645 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:18.645 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:26:18.773 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:26:18.774 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:26:18.774 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:18.774 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:18.774 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:18.774 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:26:18.775 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:18.775 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:18.775 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:18.775 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:26:18.775 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:18.775 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:18.775 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:18.777 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:18.777 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:18.777 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:18.831 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 27 2024-09-11T06:26:18.831 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:18.831 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:18.831 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:18.831 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=110, tag_tid=20, entry_tid=13214], [object_number=109, tag_tid=20, entry_tid=13213], [object_number=108, tag_tid=20, entry_tid=13212], [object_number=111, tag_tid=20, entry_tid=13211]]], state=connected] 2024-09-11T06:26:18.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:18.836 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:18.837 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:26:18.837 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:18.837 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:18.837 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:18.837 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:26:18.837 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:18.837 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:18.837 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:18.837 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:26:18.838 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:18.838 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:18.838 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:18.838 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:18.838 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:18.838 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:18.885 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 27 2024-09-11T06:26:18.885 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:18.885 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:18.885 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:18.885 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=110, tag_tid=20, entry_tid=13214], [object_number=109, tag_tid=20, entry_tid=13213], [object_number=108, tag_tid=20, entry_tid=13212], [object_number=111, tag_tid=20, entry_tid=13211]]], state=connected] 2024-09-11T06:26:18.889 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:18.891 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=110, tag_tid=20, entry_tid=13214' 2024-09-11T06:26:18.891 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=110, tag_tid=20, entry_tid=13214' 2024-09-11T06:26:18.891 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=110, tag_tid=20, entry_tid=13214' '!=' 'object_number=111, tag_tid=20, entry_tid=13111' 2024-09-11T06:26:18.891 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:26:18.891 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=110, tag_tid=20, entry_tid=13214' = 'object_number=111, tag_tid=20, entry_tid=13111' 2024-09-11T06:26:18.891 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=110, tag_tid=20, entry_tid=13214' 2024-09-11T06:26:18.891 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:18.892 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:18.892 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:18.892 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:26:18.893 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=110, tag_tid=20, entry_tid=13214' 2024-09-11T06:26:18.893 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:18.893 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:18.894 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:26:18.894 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:18.895 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:18.895 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:18.895 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:26:18.896 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=110, tag_tid=20, entry_tid=13214' 2024-09-11T06:26:18.896 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:18.896 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:18.897 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=13214 2024-09-11T06:26:18.898 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 13214 2024-09-11T06:26:18.898 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:26:18.898 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:26:18.898 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:26:19.099 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:26:19.099 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:19.099 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:26:19.100 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:26:19.100 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:26:19.100 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:26:19.100 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:26:19.100 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:26:19.100 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:26:19.100 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:19.100 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:19.100 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:26:19.100 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:26:19.100 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:26:19.101 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:26:19.101 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:26:19.101 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:26:19.105 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:26:19.105 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:26:19.105 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:26:19.105 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:19.105 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:26:19.105 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:26:19.105 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:26:19.106 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:26:19.106 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:26:19.106 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:26:19.106 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:26:19.106 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:19.106 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:26:19.106 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:19.106 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:26:19.106 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:19.106 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:26:19.106 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:26:19.106 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:26:19.107 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:26:19.107 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:26:19.107 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:26:19.107 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:26:19.107 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:26:19.107 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:19.107 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:19.107 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:19.107 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:26:19.107 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:26:19.107 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:26:19.108 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:26:19.108 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:26:19.130 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:19.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:19.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:19.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:26:19.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:26:19.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:26:19.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:19.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:19.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:19.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:26:19.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:19.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:19.269 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:19.269 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:26:19.269 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:19.269 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:19.269 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:19.271 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:19.272 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:19.272 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:19.320 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 27 2024-09-11T06:26:19.320 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:19.320 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:19.320 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:19.320 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=109, tag_tid=20, entry_tid=13313], [object_number=108, tag_tid=20, entry_tid=13312], [object_number=111, tag_tid=20, entry_tid=13311], [object_number=110, tag_tid=20, entry_tid=13310]]], state=connected] 2024-09-11T06:26:19.323 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:19.325 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:19.326 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:26:19.326 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:19.326 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:19.326 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:19.326 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:26:19.326 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:19.326 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:19.326 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:19.327 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:26:19.327 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:19.327 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:19.327 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:19.327 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:19.327 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:19.328 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:19.375 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 27 2024-09-11T06:26:19.376 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:19.376 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:19.376 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:19.376 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=109, tag_tid=20, entry_tid=13313], [object_number=108, tag_tid=20, entry_tid=13312], [object_number=111, tag_tid=20, entry_tid=13311], [object_number=110, tag_tid=20, entry_tid=13310]]], state=connected] 2024-09-11T06:26:19.379 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:19.380 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=109, tag_tid=20, entry_tid=13313' 2024-09-11T06:26:19.381 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=109, tag_tid=20, entry_tid=13313' 2024-09-11T06:26:19.381 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=109, tag_tid=20, entry_tid=13313' '!=' 'object_number=110, tag_tid=20, entry_tid=13214' 2024-09-11T06:26:19.381 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:26:19.381 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=109, tag_tid=20, entry_tid=13313' = 'object_number=110, tag_tid=20, entry_tid=13214' 2024-09-11T06:26:19.381 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=109, tag_tid=20, entry_tid=13313' 2024-09-11T06:26:19.381 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:19.382 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:19.382 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:19.382 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:26:19.383 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=109, tag_tid=20, entry_tid=13313' 2024-09-11T06:26:19.383 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:19.383 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:19.384 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:26:19.384 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:19.384 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:19.384 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:19.385 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:26:19.386 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=109, tag_tid=20, entry_tid=13313' 2024-09-11T06:26:19.386 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:19.386 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:19.387 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=13313 2024-09-11T06:26:19.387 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 13313 2024-09-11T06:26:19.388 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:26:19.388 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:26:19.388 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:26:19.589 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:26:19.590 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:19.590 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:26:19.590 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:26:19.590 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:26:19.590 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:26:19.590 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:26:19.590 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:26:19.590 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:26:19.590 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:19.590 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:19.590 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:26:19.591 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:26:19.591 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:26:19.591 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:26:19.591 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:26:19.591 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:26:19.594 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:26:19.594 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:26:19.594 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:26:19.594 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:19.594 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:26:19.594 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:26:19.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:26:19.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:26:19.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:26:19.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:26:19.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:26:19.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:19.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:26:19.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:19.596 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:26:19.596 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:19.596 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:26:19.596 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:26:19.596 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:26:19.596 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:26:19.596 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:26:19.596 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:26:19.596 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:26:19.596 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:26:19.596 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:19.597 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:19.597 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:19.597 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:26:19.597 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:26:19.597 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:26:19.597 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:26:19.598 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:26:19.625 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:19.625 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:19.625 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:19.625 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:26:19.754 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:26:19.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:26:19.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:19.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:19.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:19.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:26:19.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:19.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:19.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:19.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:26:19.756 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:19.756 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:19.756 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:19.758 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:19.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:19.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:19.810 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 27 2024-09-11T06:26:19.810 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:19.810 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:19.810 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:19.810 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=110, tag_tid=20, entry_tid=13410], [object_number=109, tag_tid=20, entry_tid=13409], [object_number=108, tag_tid=20, entry_tid=13408], [object_number=111, tag_tid=20, entry_tid=13407]]], state=connected] 2024-09-11T06:26:19.813 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:19.815 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:19.816 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:26:19.816 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:19.816 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:19.816 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:19.816 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:26:19.816 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:19.816 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:19.816 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:19.816 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:26:19.816 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:19.817 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:19.817 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:19.817 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:19.817 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:19.817 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:19.864 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 27 2024-09-11T06:26:19.865 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:19.865 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:19.865 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:19.865 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=110, tag_tid=20, entry_tid=13410], [object_number=109, tag_tid=20, entry_tid=13409], [object_number=108, tag_tid=20, entry_tid=13408], [object_number=111, tag_tid=20, entry_tid=13407]]], state=connected] 2024-09-11T06:26:19.867 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:19.869 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=110, tag_tid=20, entry_tid=13410' 2024-09-11T06:26:19.869 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=110, tag_tid=20, entry_tid=13410' 2024-09-11T06:26:19.869 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=110, tag_tid=20, entry_tid=13410' '!=' 'object_number=109, tag_tid=20, entry_tid=13313' 2024-09-11T06:26:19.869 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:26:19.870 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=110, tag_tid=20, entry_tid=13410' = 'object_number=109, tag_tid=20, entry_tid=13313' 2024-09-11T06:26:19.870 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=110, tag_tid=20, entry_tid=13410' 2024-09-11T06:26:19.870 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:19.870 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:19.870 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:19.871 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:26:19.871 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=110, tag_tid=20, entry_tid=13410' 2024-09-11T06:26:19.872 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:19.872 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:19.873 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:26:19.873 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:19.873 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:19.873 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:19.874 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:26:19.875 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=110, tag_tid=20, entry_tid=13410' 2024-09-11T06:26:19.875 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:19.875 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:19.876 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=13410 2024-09-11T06:26:19.876 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 13410 2024-09-11T06:26:19.876 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:26:19.877 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:26:19.877 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:26:20.078 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:26:20.078 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:20.078 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:26:20.078 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:26:20.078 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:26:20.078 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:26:20.079 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:26:20.079 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:26:20.079 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:26:20.079 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:20.079 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:20.079 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:26:20.079 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:26:20.079 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:26:20.079 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:26:20.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:26:20.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:26:20.082 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:26:20.082 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:26:20.082 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:26:20.082 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:20.083 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:26:20.083 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:26:20.083 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:26:20.083 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:26:20.083 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:26:20.083 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:26:20.083 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:26:20.083 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:20.083 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:26:20.083 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:20.083 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:26:20.083 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:20.085 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:26:20.085 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:26:20.085 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:26:20.085 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:26:20.085 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:26:20.085 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:26:20.085 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:26:20.085 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:26:20.085 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:20.086 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:20.086 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:20.086 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:26:20.086 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:26:20.086 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:26:20.086 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:26:20.086 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:26:20.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:20.110 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:20.110 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:20.110 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:26:20.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:26:20.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:26:20.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:20.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:20.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:20.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:26:20.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:20.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:20.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:20.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:26:20.244 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:20.244 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:20.244 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:20.245 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:20.245 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:20.245 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:20.301 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 27 2024-09-11T06:26:20.301 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:20.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:20.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:20.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=109, tag_tid=20, entry_tid=13501], [object_number=108, tag_tid=20, entry_tid=13500], [object_number=111, tag_tid=20, entry_tid=13499], [object_number=110, tag_tid=20, entry_tid=13498]]], state=connected] 2024-09-11T06:26:20.305 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:20.306 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:20.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:26:20.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:20.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:20.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:20.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:26:20.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:20.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:20.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:20.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:26:20.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:20.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:20.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:20.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:20.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:20.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:20.358 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 27 2024-09-11T06:26:20.359 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:20.359 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:20.359 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:20.359 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=109, tag_tid=20, entry_tid=13501], [object_number=108, tag_tid=20, entry_tid=13500], [object_number=111, tag_tid=20, entry_tid=13499], [object_number=110, tag_tid=20, entry_tid=13498]]], state=connected] 2024-09-11T06:26:20.362 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:20.363 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=109, tag_tid=20, entry_tid=13501' 2024-09-11T06:26:20.364 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=109, tag_tid=20, entry_tid=13501' 2024-09-11T06:26:20.364 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=109, tag_tid=20, entry_tid=13501' '!=' 'object_number=110, tag_tid=20, entry_tid=13410' 2024-09-11T06:26:20.364 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:26:20.364 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=109, tag_tid=20, entry_tid=13501' = 'object_number=110, tag_tid=20, entry_tid=13410' 2024-09-11T06:26:20.364 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=109, tag_tid=20, entry_tid=13501' 2024-09-11T06:26:20.364 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:20.365 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:20.365 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:20.365 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:26:20.366 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=109, tag_tid=20, entry_tid=13501' 2024-09-11T06:26:20.366 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:20.366 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:20.367 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:26:20.367 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:20.367 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:20.368 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:20.368 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:26:20.369 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=109, tag_tid=20, entry_tid=13501' 2024-09-11T06:26:20.369 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:20.369 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:20.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=13501 2024-09-11T06:26:20.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 13501 2024-09-11T06:26:20.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:26:20.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:26:20.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:26:20.573 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:26:20.573 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:20.573 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:26:20.573 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:26:20.573 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:26:20.573 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:26:20.573 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:26:20.573 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:26:20.573 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:26:20.573 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:20.574 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:20.574 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:26:20.574 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:26:20.574 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:26:20.574 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:26:20.574 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:26:20.574 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:26:20.577 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:26:20.577 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:26:20.577 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:26:20.577 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:20.578 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:26:20.578 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:26:20.578 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:26:20.578 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:26:20.578 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:26:20.578 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:26:20.578 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:26:20.578 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:20.578 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:26:20.578 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:20.578 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:26:20.578 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:20.578 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:26:20.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:26:20.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:26:20.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:26:20.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:26:20.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:26:20.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:26:20.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:26:20.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:20.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:20.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:20.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:26:20.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:26:20.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:26:20.580 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:26:20.580 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:26:20.602 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:20.603 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:20.603 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:20.603 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:26:20.731 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:26:20.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:26:20.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:20.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:20.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:20.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:26:20.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:20.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:20.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:20.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:26:20.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:20.734 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:20.734 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:20.735 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:20.736 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:20.736 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:20.788 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 27 2024-09-11T06:26:20.788 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:20.788 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:20.788 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:20.788 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=108, tag_tid=20, entry_tid=13584], [object_number=111, tag_tid=20, entry_tid=13583], [object_number=110, tag_tid=20, entry_tid=13582], [object_number=109, tag_tid=20, entry_tid=13581]]], state=connected] 2024-09-11T06:26:20.791 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:20.793 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:20.794 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:26:20.794 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:20.794 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:20.794 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:20.794 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:26:20.795 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:20.795 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:20.795 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:20.795 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:26:20.795 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:20.795 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:20.795 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:20.796 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:20.796 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:20.796 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:20.843 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 27 2024-09-11T06:26:20.844 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:20.844 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:20.844 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:20.844 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=108, tag_tid=20, entry_tid=13584], [object_number=111, tag_tid=20, entry_tid=13583], [object_number=110, tag_tid=20, entry_tid=13582], [object_number=109, tag_tid=20, entry_tid=13581]]], state=connected] 2024-09-11T06:26:20.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:20.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=108, tag_tid=20, entry_tid=13584' 2024-09-11T06:26:20.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=108, tag_tid=20, entry_tid=13584' 2024-09-11T06:26:20.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=108, tag_tid=20, entry_tid=13584' '!=' 'object_number=109, tag_tid=20, entry_tid=13501' 2024-09-11T06:26:20.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:26:20.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=108, tag_tid=20, entry_tid=13584' = 'object_number=109, tag_tid=20, entry_tid=13501' 2024-09-11T06:26:20.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=108, tag_tid=20, entry_tid=13584' 2024-09-11T06:26:20.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:20.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:20.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:20.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:26:20.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=108, tag_tid=20, entry_tid=13584' 2024-09-11T06:26:20.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:20.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:20.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:26:20.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:20.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:20.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:20.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:26:20.854 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=108, tag_tid=20, entry_tid=13584' 2024-09-11T06:26:20.854 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:20.854 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:20.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=13584 2024-09-11T06:26:20.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 13584 2024-09-11T06:26:20.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:26:20.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:26:20.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:26:21.057 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:26:21.058 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:21.058 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:26:21.058 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:26:21.058 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:26:21.058 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:26:21.058 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:26:21.058 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:26:21.058 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:26:21.058 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:21.058 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:21.058 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:26:21.058 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:26:21.058 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:26:21.059 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:26:21.059 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:26:21.059 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:26:21.063 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:26:21.063 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:26:21.064 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:26:21.064 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:21.064 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:26:21.064 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:26:21.064 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:26:21.064 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:26:21.064 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:26:21.065 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:26:21.065 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:26:21.065 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:21.065 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:26:21.065 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:21.065 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:26:21.065 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:21.065 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:26:21.065 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:26:21.065 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:26:21.065 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:26:21.065 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:26:21.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:26:21.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:26:21.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:26:21.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:21.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:21.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:21.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:26:21.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:26:21.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:26:21.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:26:21.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:26:21.088 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:21.088 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:21.089 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:21.089 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:26:21.226 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:26:21.228 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:26:21.228 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:21.228 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:21.228 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:21.228 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:26:21.228 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:21.228 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:21.228 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:21.228 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:26:21.229 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:21.229 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:21.229 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:21.230 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:21.231 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:21.231 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:21.278 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 27 2024-09-11T06:26:21.278 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:21.278 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:21.278 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:21.278 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=110, tag_tid=20, entry_tid=13666], [object_number=109, tag_tid=20, entry_tid=13665], [object_number=108, tag_tid=20, entry_tid=13664], [object_number=111, tag_tid=20, entry_tid=13663]]], state=connected] 2024-09-11T06:26:21.281 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:21.282 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:21.283 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:26:21.283 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:21.284 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:21.284 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:21.284 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:26:21.284 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:21.284 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:21.284 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:21.284 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:26:21.284 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:21.284 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:21.284 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:21.285 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:21.285 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:21.285 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:21.332 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 27 2024-09-11T06:26:21.332 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:21.332 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:21.332 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:21.332 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=110, tag_tid=20, entry_tid=13666], [object_number=109, tag_tid=20, entry_tid=13665], [object_number=108, tag_tid=20, entry_tid=13664], [object_number=111, tag_tid=20, entry_tid=13663]]], state=connected] 2024-09-11T06:26:21.335 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:21.337 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=110, tag_tid=20, entry_tid=13666' 2024-09-11T06:26:21.337 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=110, tag_tid=20, entry_tid=13666' 2024-09-11T06:26:21.337 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=110, tag_tid=20, entry_tid=13666' '!=' 'object_number=108, tag_tid=20, entry_tid=13584' 2024-09-11T06:26:21.337 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:26:21.337 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=110, tag_tid=20, entry_tid=13666' = 'object_number=108, tag_tid=20, entry_tid=13584' 2024-09-11T06:26:21.337 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=110, tag_tid=20, entry_tid=13666' 2024-09-11T06:26:21.338 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:21.338 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:21.338 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:21.338 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:26:21.339 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=110, tag_tid=20, entry_tid=13666' 2024-09-11T06:26:21.339 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:21.339 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:21.340 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:26:21.341 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:21.341 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:21.341 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:21.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:26:21.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=110, tag_tid=20, entry_tid=13666' 2024-09-11T06:26:21.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:21.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:21.344 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=13666 2024-09-11T06:26:21.344 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 13666 2024-09-11T06:26:21.344 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:26:21.344 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:26:21.344 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:26:21.546 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:26:21.546 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:21.546 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:26:21.546 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:26:21.546 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:26:21.546 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:26:21.546 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:26:21.546 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:26:21.547 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:26:21.547 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:21.547 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:21.547 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:26:21.547 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:26:21.547 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:26:21.547 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:26:21.548 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:26:21.548 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:26:21.551 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:26:21.551 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:26:21.551 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:26:21.551 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:21.551 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:26:21.551 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:26:21.551 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:26:21.551 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:26:21.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:26:21.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:26:21.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:26:21.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:21.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:26:21.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:21.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:26:21.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:21.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:26:21.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:26:21.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:26:21.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:26:21.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:26:21.553 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:26:21.553 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:26:21.553 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:26:21.553 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:21.553 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:21.553 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:21.553 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:26:21.553 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:26:21.553 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:26:21.553 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:26:21.553 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:26:21.578 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:21.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:21.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:21.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:26:21.707 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:26:21.708 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:26:21.708 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:21.708 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:21.708 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:21.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:26:21.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:21.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:21.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:21.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:26:21.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:21.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:21.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:21.711 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:21.711 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:21.711 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:21.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 27 2024-09-11T06:26:21.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:21.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:21.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:21.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=111, tag_tid=20, entry_tid=13743], [object_number=110, tag_tid=20, entry_tid=13742], [object_number=109, tag_tid=20, entry_tid=13741], [object_number=108, tag_tid=20, entry_tid=13740]]], state=connected] 2024-09-11T06:26:21.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:21.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:21.765 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:26:21.765 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:21.765 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:21.765 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:21.765 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:26:21.765 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:21.765 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:21.765 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:21.765 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:26:21.765 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:21.765 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:21.765 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:21.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:21.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:21.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:21.813 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 27 2024-09-11T06:26:21.813 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:21.813 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:21.813 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:21.813 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=111, tag_tid=20, entry_tid=13743], [object_number=110, tag_tid=20, entry_tid=13742], [object_number=109, tag_tid=20, entry_tid=13741], [object_number=108, tag_tid=20, entry_tid=13740]]], state=connected] 2024-09-11T06:26:21.816 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:21.818 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=111, tag_tid=20, entry_tid=13743' 2024-09-11T06:26:21.818 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=111, tag_tid=20, entry_tid=13743' 2024-09-11T06:26:21.818 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=111, tag_tid=20, entry_tid=13743' '!=' 'object_number=110, tag_tid=20, entry_tid=13666' 2024-09-11T06:26:21.818 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:26:21.818 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=111, tag_tid=20, entry_tid=13743' = 'object_number=110, tag_tid=20, entry_tid=13666' 2024-09-11T06:26:21.819 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=111, tag_tid=20, entry_tid=13743' 2024-09-11T06:26:21.819 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:21.819 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:21.819 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:21.819 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:26:21.820 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=111, tag_tid=20, entry_tid=13743' 2024-09-11T06:26:21.820 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:21.820 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:21.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:26:21.822 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:21.822 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:21.822 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:21.823 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:26:21.823 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=111, tag_tid=20, entry_tid=13743' 2024-09-11T06:26:21.823 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:21.823 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:21.825 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=13743 2024-09-11T06:26:21.825 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 13743 2024-09-11T06:26:21.825 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:26:21.825 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:26:21.825 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:26:22.026 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:26:22.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:22.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:26:22.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:26:22.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:26:22.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:26:22.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:26:22.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:26:22.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:26:22.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:22.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:22.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:26:22.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:26:22.028 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:26:22.028 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:26:22.028 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:26:22.028 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:26:22.032 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:26:22.032 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:26:22.032 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:26:22.032 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:22.032 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:26:22.032 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:26:22.032 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:26:22.032 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:26:22.032 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:26:22.032 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:26:22.032 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:26:22.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:22.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:26:22.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:22.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:26:22.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:22.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:26:22.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:26:22.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:26:22.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:26:22.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:26:22.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:26:22.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:26:22.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:26:22.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:22.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:22.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:22.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:26:22.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:26:22.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:26:22.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:26:22.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:26:22.056 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:22.057 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:22.057 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:22.057 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:26:22.188 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:26:22.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:26:22.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:22.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:22.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:22.190 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:26:22.190 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:22.190 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:22.190 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:22.190 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:26:22.190 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:22.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:22.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:22.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:22.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:22.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:22.239 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 27 2024-09-11T06:26:22.240 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:22.240 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:22.240 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:22.240 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=110, tag_tid=20, entry_tid=13834], [object_number=109, tag_tid=20, entry_tid=13833], [object_number=108, tag_tid=20, entry_tid=13832], [object_number=111, tag_tid=20, entry_tid=13831]]], state=connected] 2024-09-11T06:26:22.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:22.244 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:22.245 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:26:22.245 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:22.246 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:22.246 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:22.246 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:26:22.246 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:22.246 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:22.246 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:22.246 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:26:22.246 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:22.246 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:22.246 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:22.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:22.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:22.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:22.293 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 27 2024-09-11T06:26:22.293 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:22.293 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:22.294 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:22.294 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=110, tag_tid=20, entry_tid=13834], [object_number=109, tag_tid=20, entry_tid=13833], [object_number=108, tag_tid=20, entry_tid=13832], [object_number=111, tag_tid=20, entry_tid=13831]]], state=connected] 2024-09-11T06:26:22.297 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:22.298 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=110, tag_tid=20, entry_tid=13834' 2024-09-11T06:26:22.298 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=110, tag_tid=20, entry_tid=13834' 2024-09-11T06:26:22.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=110, tag_tid=20, entry_tid=13834' '!=' 'object_number=111, tag_tid=20, entry_tid=13743' 2024-09-11T06:26:22.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:26:22.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=110, tag_tid=20, entry_tid=13834' = 'object_number=111, tag_tid=20, entry_tid=13743' 2024-09-11T06:26:22.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=110, tag_tid=20, entry_tid=13834' 2024-09-11T06:26:22.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:22.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:22.300 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:22.300 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:26:22.300 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=110, tag_tid=20, entry_tid=13834' 2024-09-11T06:26:22.301 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:22.301 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:22.301 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:26:22.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:22.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:22.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:22.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:26:22.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=110, tag_tid=20, entry_tid=13834' 2024-09-11T06:26:22.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:22.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:22.306 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=13834 2024-09-11T06:26:22.306 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 13834 2024-09-11T06:26:22.306 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:26:22.306 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:26:22.306 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:26:22.507 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:26:22.507 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:22.507 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:26:22.507 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:26:22.507 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:26:22.507 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:26:22.507 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:26:22.507 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:26:22.508 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:26:22.508 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:22.508 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:22.508 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:26:22.508 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:26:22.508 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:26:22.508 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:26:22.509 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:26:22.509 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:26:22.511 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:26:22.511 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:26:22.511 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:26:22.511 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:22.511 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:26:22.511 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:26:22.511 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:26:22.511 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:26:22.511 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:26:22.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:26:22.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:26:22.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:22.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:26:22.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:22.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:26:22.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:22.514 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:26:22.514 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:26:22.514 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:26:22.514 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:26:22.514 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:26:22.514 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:26:22.514 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:26:22.515 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:26:22.515 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:22.515 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:22.515 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:22.515 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:26:22.515 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:26:22.515 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:26:22.515 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:26:22.515 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:26:22.538 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:22.539 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:22.539 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:22.539 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:26:22.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:26:22.668 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:26:22.668 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:22.668 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:22.668 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:22.668 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:26:22.668 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:22.668 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:22.669 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:22.669 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:26:22.669 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:22.669 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:22.669 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:22.672 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:22.672 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:22.672 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:22.726 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 27 2024-09-11T06:26:22.726 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:22.726 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:22.727 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:22.727 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=109, tag_tid=20, entry_tid=13917], [object_number=108, tag_tid=20, entry_tid=13916], [object_number=111, tag_tid=20, entry_tid=13915], [object_number=110, tag_tid=20, entry_tid=13914]]], state=connected] 2024-09-11T06:26:22.730 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:22.731 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:22.732 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:26:22.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:22.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:22.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:22.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:26:22.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:22.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:22.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:22.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:26:22.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:22.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:22.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:22.734 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:22.734 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:22.734 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:22.781 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 27 2024-09-11T06:26:22.781 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:22.781 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:22.781 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:22.781 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=109, tag_tid=20, entry_tid=13917], [object_number=108, tag_tid=20, entry_tid=13916], [object_number=111, tag_tid=20, entry_tid=13915], [object_number=110, tag_tid=20, entry_tid=13914]]], state=connected] 2024-09-11T06:26:22.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:22.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=109, tag_tid=20, entry_tid=13917' 2024-09-11T06:26:22.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=109, tag_tid=20, entry_tid=13917' 2024-09-11T06:26:22.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=109, tag_tid=20, entry_tid=13917' '!=' 'object_number=110, tag_tid=20, entry_tid=13834' 2024-09-11T06:26:22.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:26:22.787 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=109, tag_tid=20, entry_tid=13917' = 'object_number=110, tag_tid=20, entry_tid=13834' 2024-09-11T06:26:22.787 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=109, tag_tid=20, entry_tid=13917' 2024-09-11T06:26:22.787 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:22.787 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:22.787 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:22.788 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:26:22.789 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=109, tag_tid=20, entry_tid=13917' 2024-09-11T06:26:22.789 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:22.789 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:22.790 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:26:22.791 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:22.791 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:22.791 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:22.792 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:26:22.793 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=109, tag_tid=20, entry_tid=13917' 2024-09-11T06:26:22.793 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:22.793 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:22.794 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=13917 2024-09-11T06:26:22.795 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 13917 2024-09-11T06:26:22.795 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:26:22.795 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:26:22.795 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:26:22.995 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:26:22.996 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:22.996 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:26:22.996 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:26:22.996 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:26:22.996 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:26:22.996 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:26:22.996 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:26:22.996 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:26:22.996 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:22.996 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:22.997 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:26:22.997 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:26:22.997 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:26:22.997 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:26:22.999 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:26:22.999 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:26:23.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:26:23.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:26:23.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:26:23.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:23.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:26:23.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:26:23.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:26:23.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:26:23.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:26:23.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:26:23.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:26:23.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:23.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:26:23.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:23.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:26:23.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:23.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:26:23.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:26:23.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:26:23.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:26:23.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:26:23.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:26:23.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:26:23.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:26:23.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:23.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:23.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:23.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:26:23.002 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:26:23.002 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:26:23.002 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:26:23.002 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:26:23.026 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:23.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:23.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:23.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:26:23.154 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:26:23.155 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:26:23.155 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:23.155 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:23.156 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:23.156 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:26:23.156 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:23.156 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:23.156 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:23.156 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:26:23.157 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:23.157 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:23.157 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:23.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:23.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:23.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:23.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 27 2024-09-11T06:26:23.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:23.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:23.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:23.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=111, tag_tid=20, entry_tid=13995], [object_number=110, tag_tid=20, entry_tid=13994], [object_number=109, tag_tid=20, entry_tid=13993], [object_number=108, tag_tid=20, entry_tid=13992]]], state=connected] 2024-09-11T06:26:23.215 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:23.216 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:23.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:26:23.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:23.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:23.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:23.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:26:23.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:23.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:23.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:23.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:26:23.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:23.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:23.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:23.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:23.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:23.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:23.266 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 27 2024-09-11T06:26:23.266 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:23.266 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:23.266 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:23.266 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=111, tag_tid=20, entry_tid=13995], [object_number=110, tag_tid=20, entry_tid=13994], [object_number=109, tag_tid=20, entry_tid=13993], [object_number=108, tag_tid=20, entry_tid=13992]]], state=connected] 2024-09-11T06:26:23.270 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:23.271 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=111, tag_tid=20, entry_tid=13995' 2024-09-11T06:26:23.272 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=111, tag_tid=20, entry_tid=13995' 2024-09-11T06:26:23.272 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=111, tag_tid=20, entry_tid=13995' '!=' 'object_number=109, tag_tid=20, entry_tid=13917' 2024-09-11T06:26:23.272 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:26:23.272 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=111, tag_tid=20, entry_tid=13995' = 'object_number=109, tag_tid=20, entry_tid=13917' 2024-09-11T06:26:23.272 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=111, tag_tid=20, entry_tid=13995' 2024-09-11T06:26:23.272 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:23.272 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:23.272 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:23.273 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:26:23.273 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=111, tag_tid=20, entry_tid=13995' 2024-09-11T06:26:23.274 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:23.274 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:23.274 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:26:23.275 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:23.275 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:23.275 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:23.276 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:26:23.277 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=111, tag_tid=20, entry_tid=13995' 2024-09-11T06:26:23.277 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:23.277 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:23.278 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=13995 2024-09-11T06:26:23.278 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 13995 2024-09-11T06:26:23.278 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:26:23.278 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:26:23.278 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:26:23.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:26:23.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:23.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:26:23.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:26:23.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:26:23.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:26:23.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:26:23.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:26:23.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:26:23.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:23.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:23.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:26:23.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:26:23.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:26:23.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:26:23.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:26:23.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:26:23.485 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:26:23.485 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:26:23.485 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:26:23.485 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:23.485 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:26:23.485 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:26:23.485 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:26:23.485 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:26:23.486 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:26:23.486 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:26:23.486 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:26:23.486 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:23.486 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:26:23.486 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:23.486 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:26:23.486 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:23.486 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:26:23.486 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:26:23.486 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:26:23.486 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:26:23.486 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:26:23.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:26:23.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:26:23.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:26:23.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:23.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:23.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:23.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:26:23.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:26:23.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:26:23.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:26:23.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:26:23.513 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:23.513 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:23.513 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:23.513 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:26:23.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:26:23.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:26:23.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:23.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:23.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:23.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:26:23.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:23.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:23.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:23.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:26:23.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:23.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:23.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:23.645 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:23.646 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:23.646 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:23.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 27 2024-09-11T06:26:23.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:23.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:23.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:23.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=111, tag_tid=20, entry_tid=14079], [object_number=110, tag_tid=20, entry_tid=14078], [object_number=109, tag_tid=20, entry_tid=14077], [object_number=108, tag_tid=20, entry_tid=14076]]], state=connected] 2024-09-11T06:26:23.701 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:23.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:23.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:26:23.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:23.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:23.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:23.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:26:23.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:23.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:23.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:23.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:26:23.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:23.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:23.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:23.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:23.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:23.706 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:23.753 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 27 2024-09-11T06:26:23.753 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:23.753 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:23.753 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:23.753 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=111, tag_tid=20, entry_tid=14079], [object_number=110, tag_tid=20, entry_tid=14078], [object_number=109, tag_tid=20, entry_tid=14077], [object_number=108, tag_tid=20, entry_tid=14076]]], state=connected] 2024-09-11T06:26:23.756 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:23.758 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=111, tag_tid=20, entry_tid=14079' 2024-09-11T06:26:23.758 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=111, tag_tid=20, entry_tid=14079' 2024-09-11T06:26:23.758 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=111, tag_tid=20, entry_tid=14079' '!=' 'object_number=111, tag_tid=20, entry_tid=13995' 2024-09-11T06:26:23.758 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:26:23.758 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=111, tag_tid=20, entry_tid=14079' = 'object_number=111, tag_tid=20, entry_tid=13995' 2024-09-11T06:26:23.758 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=111, tag_tid=20, entry_tid=14079' 2024-09-11T06:26:23.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:23.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:23.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:23.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:26:23.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=111, tag_tid=20, entry_tid=14079' 2024-09-11T06:26:23.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:23.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:23.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:26:23.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:23.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:23.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:23.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:26:23.764 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=111, tag_tid=20, entry_tid=14079' 2024-09-11T06:26:23.764 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:23.764 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:23.765 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=14079 2024-09-11T06:26:23.765 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 14079 2024-09-11T06:26:23.765 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:26:23.765 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:26:23.765 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:26:23.967 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:26:23.967 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:23.967 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:26:23.967 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:26:23.968 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:26:23.968 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:26:23.968 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:26:23.968 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:26:23.968 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:26:23.968 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:23.968 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:23.968 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:26:23.968 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:26:23.968 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:26:23.968 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:26:23.969 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:26:23.969 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:26:23.972 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:26:23.972 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:26:23.972 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:26:23.972 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:23.972 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:26:23.972 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:26:23.973 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:26:23.973 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:26:23.973 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:26:23.973 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:26:23.973 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:26:23.973 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:23.973 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:26:23.973 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:23.973 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:26:23.973 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:23.973 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:26:23.973 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:26:23.973 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:26:23.974 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:26:23.974 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:26:23.974 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:26:23.974 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:26:23.974 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:26:23.974 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:23.974 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:23.974 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:23.974 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:26:23.975 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:26:23.975 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:26:23.975 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:26:23.975 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:26:23.997 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:23.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:23.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:23.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:26:24.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:26:24.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:26:24.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:24.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:24.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:24.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:26:24.128 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:24.128 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:24.128 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:24.128 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:26:24.128 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:24.128 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:24.129 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:24.130 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:24.130 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:24.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:24.178 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 27 2024-09-11T06:26:24.178 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:24.178 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:24.178 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:24.178 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=111, tag_tid=20, entry_tid=14163], [object_number=110, tag_tid=20, entry_tid=14162], [object_number=109, tag_tid=20, entry_tid=14161], [object_number=108, tag_tid=20, entry_tid=14160]]], state=connected] 2024-09-11T06:26:24.181 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:24.183 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:24.184 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:26:24.184 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:24.184 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:24.184 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:24.184 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:26:24.184 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:24.185 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:24.185 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:24.185 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:26:24.185 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:24.185 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:24.185 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:24.185 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:24.186 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:24.186 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:24.232 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 27 2024-09-11T06:26:24.233 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:24.233 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:24.233 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:24.233 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=111, tag_tid=20, entry_tid=14163], [object_number=110, tag_tid=20, entry_tid=14162], [object_number=109, tag_tid=20, entry_tid=14161], [object_number=108, tag_tid=20, entry_tid=14160]]], state=connected] 2024-09-11T06:26:24.236 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:24.238 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=111, tag_tid=20, entry_tid=14163' 2024-09-11T06:26:24.238 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=111, tag_tid=20, entry_tid=14163' 2024-09-11T06:26:24.238 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=111, tag_tid=20, entry_tid=14163' '!=' 'object_number=111, tag_tid=20, entry_tid=14079' 2024-09-11T06:26:24.238 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:26:24.238 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=111, tag_tid=20, entry_tid=14163' = 'object_number=111, tag_tid=20, entry_tid=14079' 2024-09-11T06:26:24.238 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=111, tag_tid=20, entry_tid=14163' 2024-09-11T06:26:24.238 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:24.238 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:24.238 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:24.239 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:26:24.240 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=111, tag_tid=20, entry_tid=14163' 2024-09-11T06:26:24.240 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:24.240 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:24.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:26:24.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:24.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:24.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:24.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:26:24.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=111, tag_tid=20, entry_tid=14163' 2024-09-11T06:26:24.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:24.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:24.244 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=14163 2024-09-11T06:26:24.244 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 14163 2024-09-11T06:26:24.244 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:26:24.244 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:26:24.245 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:26:24.446 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:26:24.446 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:24.446 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:26:24.446 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:26:24.446 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:26:24.446 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:26:24.447 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:26:24.447 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:26:24.447 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:26:24.447 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:24.447 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:24.447 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:26:24.447 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:26:24.447 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:26:24.447 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:26:24.448 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:26:24.448 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:26:24.451 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:26:24.451 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:26:24.452 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:26:24.452 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:24.452 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:26:24.452 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:26:24.452 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:26:24.452 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:26:24.452 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:26:24.452 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:26:24.452 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:26:24.452 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:24.452 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:26:24.452 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:24.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:26:24.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:24.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:26:24.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:26:24.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:26:24.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:26:24.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:26:24.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:26:24.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:26:24.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:26:24.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:24.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:24.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:24.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:26:24.454 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:26:24.454 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:26:24.454 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:26:24.454 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:26:24.478 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:24.478 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:24.478 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:24.478 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:26:24.607 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:26:24.608 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:26:24.608 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:24.608 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:24.608 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:24.609 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:26:24.609 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:24.609 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:24.609 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:24.609 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:26:24.609 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:24.610 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:24.610 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:24.611 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:24.611 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:24.611 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:24.658 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 27 2024-09-11T06:26:24.658 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:24.659 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:24.659 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:24.659 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=110, tag_tid=20, entry_tid=14242], [object_number=109, tag_tid=20, entry_tid=14241], [object_number=108, tag_tid=20, entry_tid=14240], [object_number=111, tag_tid=20, entry_tid=14239]]], state=connected] 2024-09-11T06:26:24.662 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:24.663 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:24.664 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:26:24.664 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:24.664 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:24.665 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:24.665 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:26:24.665 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:24.665 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:24.665 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:24.665 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:26:24.665 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:24.665 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:24.665 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:24.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:24.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:24.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:24.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 27 2024-09-11T06:26:24.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:24.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:24.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:24.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=110, tag_tid=20, entry_tid=14242], [object_number=109, tag_tid=20, entry_tid=14241], [object_number=108, tag_tid=20, entry_tid=14240], [object_number=111, tag_tid=20, entry_tid=14239]]], state=connected] 2024-09-11T06:26:24.716 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:24.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=110, tag_tid=20, entry_tid=14242' 2024-09-11T06:26:24.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=110, tag_tid=20, entry_tid=14242' 2024-09-11T06:26:24.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=110, tag_tid=20, entry_tid=14242' '!=' 'object_number=111, tag_tid=20, entry_tid=14163' 2024-09-11T06:26:24.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:26:24.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=110, tag_tid=20, entry_tid=14242' = 'object_number=111, tag_tid=20, entry_tid=14163' 2024-09-11T06:26:24.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=110, tag_tid=20, entry_tid=14242' 2024-09-11T06:26:24.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:24.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:24.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:24.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:26:24.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=110, tag_tid=20, entry_tid=14242' 2024-09-11T06:26:24.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:24.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:24.721 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:26:24.722 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:24.722 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:24.722 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:24.723 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:26:24.723 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=110, tag_tid=20, entry_tid=14242' 2024-09-11T06:26:24.723 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:24.724 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:24.725 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=14242 2024-09-11T06:26:24.725 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 14242 2024-09-11T06:26:24.725 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:26:24.725 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:26:24.725 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:26:24.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:26:24.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:24.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:26:24.928 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:26:24.928 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:26:24.928 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:26:24.928 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:26:24.928 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:26:24.928 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:26:24.928 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:24.928 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:24.928 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:26:24.928 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:26:24.928 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:26:24.928 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:26:24.929 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:26:24.932 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:26:24.932 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:26:24.932 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:26:24.932 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:26:24.932 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:24.932 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:26:24.932 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:26:24.932 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:26:24.932 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:26:24.933 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:26:24.933 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:26:24.933 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:26:24.933 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:24.933 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:26:24.933 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:24.933 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:26:24.933 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:24.933 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:26:24.933 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:26:24.933 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:26:24.933 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:26:24.933 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:26:24.933 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:26:24.934 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:26:24.934 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:26:24.934 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:24.934 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:24.934 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:24.934 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:26:24.934 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:26:24.934 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:26:24.934 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:26:24.934 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:26:24.956 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:24.957 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:24.957 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:24.957 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:26:25.099 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:26:25.100 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:26:25.100 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:25.100 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:25.100 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:25.100 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:26:25.101 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:25.101 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:25.101 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:25.101 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:26:25.101 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:25.101 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:25.102 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:25.103 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:25.104 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:25.104 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:25.153 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 27 2024-09-11T06:26:25.153 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:25.153 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:25.154 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:25.154 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=110, tag_tid=20, entry_tid=14330], [object_number=109, tag_tid=20, entry_tid=14329], [object_number=108, tag_tid=20, entry_tid=14328], [object_number=111, tag_tid=20, entry_tid=14327]]], state=connected] 2024-09-11T06:26:25.154 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:25.157 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:25.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:26:25.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:25.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:25.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:25.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:26:25.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:25.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:25.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:25.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:26:25.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:25.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:25.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:25.159 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:25.159 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:25.159 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:25.206 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 27 2024-09-11T06:26:25.206 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:25.206 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:25.206 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:25.206 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=110, tag_tid=20, entry_tid=14330], [object_number=109, tag_tid=20, entry_tid=14329], [object_number=108, tag_tid=20, entry_tid=14328], [object_number=111, tag_tid=20, entry_tid=14327]]], state=connected] 2024-09-11T06:26:25.209 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:25.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=110, tag_tid=20, entry_tid=14330' 2024-09-11T06:26:25.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=110, tag_tid=20, entry_tid=14330' 2024-09-11T06:26:25.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=110, tag_tid=20, entry_tid=14330' '!=' 'object_number=110, tag_tid=20, entry_tid=14242' 2024-09-11T06:26:25.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:26:25.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=110, tag_tid=20, entry_tid=14330' = 'object_number=110, tag_tid=20, entry_tid=14242' 2024-09-11T06:26:25.213 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=110, tag_tid=20, entry_tid=14330' 2024-09-11T06:26:25.213 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:25.213 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:25.213 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:25.214 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:26:25.214 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=110, tag_tid=20, entry_tid=14330' 2024-09-11T06:26:25.214 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:25.214 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:25.215 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:26:25.216 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:25.216 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:25.216 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:25.216 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:26:25.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=110, tag_tid=20, entry_tid=14330' 2024-09-11T06:26:25.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:25.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:25.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=14330 2024-09-11T06:26:25.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 14330 2024-09-11T06:26:25.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:26:25.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:26:25.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:26:25.420 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:26:25.421 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:25.421 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:26:25.421 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:26:25.421 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:26:25.421 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:26:25.421 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:26:25.421 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:26:25.421 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:26:25.421 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:25.421 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:25.421 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:26:25.421 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:26:25.421 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:26:25.422 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:26:25.422 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:26:25.422 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:26:25.425 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:26:25.425 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:26:25.425 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:26:25.425 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:25.425 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:26:25.425 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:26:25.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:26:25.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:26:25.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:26:25.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:26:25.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:26:25.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:25.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:26:25.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:25.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:26:25.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:25.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:26:25.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:26:25.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:26:25.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:26:25.427 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:26:25.427 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:26:25.427 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:26:25.427 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:26:25.427 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:25.427 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:25.427 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:25.427 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:26:25.427 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:26:25.427 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:26:25.427 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:26:25.428 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:26:25.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:25.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:25.454 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:25.454 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:26:25.583 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:26:25.584 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:26:25.584 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:25.585 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:25.585 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:25.585 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:26:25.585 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:25.585 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:25.585 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:25.585 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:26:25.585 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:25.586 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:25.586 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:25.587 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:25.587 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:25.587 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:25.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 27 2024-09-11T06:26:25.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:25.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:25.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:25.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=110, tag_tid=20, entry_tid=14434], [object_number=109, tag_tid=20, entry_tid=14433], [object_number=108, tag_tid=20, entry_tid=14432], [object_number=111, tag_tid=20, entry_tid=14431]]], state=connected] 2024-09-11T06:26:25.646 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:25.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:25.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:26:25.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:25.649 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:25.649 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:25.649 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:26:25.649 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:25.649 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:25.649 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:25.649 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:26:25.649 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:25.649 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:25.649 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:25.650 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:25.650 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:25.650 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:25.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 27 2024-09-11T06:26:25.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:25.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:25.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:25.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=110, tag_tid=20, entry_tid=14434], [object_number=109, tag_tid=20, entry_tid=14433], [object_number=108, tag_tid=20, entry_tid=14432], [object_number=111, tag_tid=20, entry_tid=14431]]], state=connected] 2024-09-11T06:26:25.701 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:25.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=110, tag_tid=20, entry_tid=14434' 2024-09-11T06:26:25.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=110, tag_tid=20, entry_tid=14434' 2024-09-11T06:26:25.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=110, tag_tid=20, entry_tid=14434' '!=' 'object_number=110, tag_tid=20, entry_tid=14330' 2024-09-11T06:26:25.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:26:25.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=110, tag_tid=20, entry_tid=14434' = 'object_number=110, tag_tid=20, entry_tid=14330' 2024-09-11T06:26:25.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=110, tag_tid=20, entry_tid=14434' 2024-09-11T06:26:25.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:25.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:25.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:25.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:26:25.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=110, tag_tid=20, entry_tid=14434' 2024-09-11T06:26:25.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:25.706 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:25.706 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:26:25.707 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:25.707 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:25.707 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:25.708 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:26:25.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=110, tag_tid=20, entry_tid=14434' 2024-09-11T06:26:25.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:25.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:25.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=14434 2024-09-11T06:26:25.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 14434 2024-09-11T06:26:25.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:26:25.711 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:26:25.711 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:26:25.912 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:26:25.912 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:25.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:26:25.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:26:25.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:26:25.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:26:25.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:26:25.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:26:25.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:26:25.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:25.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:25.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:26:25.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:26:25.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:26:25.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:26:25.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:26:25.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:26:25.918 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:26:25.918 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:26:25.918 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:26:25.918 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:25.918 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:26:25.918 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:26:25.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:26:25.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:26:25.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:26:25.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:26:25.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:26:25.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:25.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:26:25.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:25.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:26:25.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:25.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:26:25.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:26:25.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:26:25.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:26:25.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:26:25.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:26:25.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:26:25.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:26:25.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:25.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:25.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:25.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:26:25.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:26:25.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:26:25.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:26:25.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:26:25.943 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:25.943 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:25.944 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:25.944 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:26:26.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:26:26.076 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:26:26.077 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:26.077 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:26.077 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:26.077 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:26:26.077 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:26.077 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:26.077 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:26.077 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:26:26.078 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:26.078 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:26.078 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:26.079 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:26.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:26.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:26.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 27 2024-09-11T06:26:26.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:26.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:26.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:26.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=108, tag_tid=20, entry_tid=14528], [object_number=111, tag_tid=20, entry_tid=14527], [object_number=110, tag_tid=20, entry_tid=14526], [object_number=109, tag_tid=20, entry_tid=14525]]], state=connected] 2024-09-11T06:26:26.130 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:26.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:26.133 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:26:26.133 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:26.133 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:26.133 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:26.133 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:26:26.133 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:26.133 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:26.133 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:26.133 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:26:26.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:26.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:26.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:26.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:26.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:26.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:26.181 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 27 2024-09-11T06:26:26.181 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:26.182 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:26.182 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:26.182 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=108, tag_tid=20, entry_tid=14528], [object_number=111, tag_tid=20, entry_tid=14527], [object_number=110, tag_tid=20, entry_tid=14526], [object_number=109, tag_tid=20, entry_tid=14525]]], state=connected] 2024-09-11T06:26:26.185 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:26.187 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=108, tag_tid=20, entry_tid=14528' 2024-09-11T06:26:26.187 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=108, tag_tid=20, entry_tid=14528' 2024-09-11T06:26:26.187 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=108, tag_tid=20, entry_tid=14528' '!=' 'object_number=110, tag_tid=20, entry_tid=14434' 2024-09-11T06:26:26.187 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:26:26.187 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=108, tag_tid=20, entry_tid=14528' = 'object_number=110, tag_tid=20, entry_tid=14434' 2024-09-11T06:26:26.187 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=108, tag_tid=20, entry_tid=14528' 2024-09-11T06:26:26.188 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:26.188 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:26.188 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:26.188 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:26:26.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=108, tag_tid=20, entry_tid=14528' 2024-09-11T06:26:26.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:26.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:26.190 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:26:26.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:26.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:26.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:26.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:26:26.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=108, tag_tid=20, entry_tid=14528' 2024-09-11T06:26:26.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:26.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:26.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=14528 2024-09-11T06:26:26.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 14528 2024-09-11T06:26:26.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:26:26.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:26:26.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:26:26.395 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:26:26.395 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:26.395 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:26:26.396 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:26:26.396 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:26:26.396 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:26:26.396 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:26:26.396 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:26:26.396 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:26:26.396 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:26.396 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:26.396 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:26:26.396 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:26:26.396 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:26:26.396 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:26:26.397 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:26:26.397 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:26:26.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:26:26.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:26:26.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:26:26.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:26.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:26:26.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:26:26.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:26:26.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:26:26.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:26:26.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:26:26.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:26:26.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:26.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:26:26.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:26.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:26:26.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:26.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:26:26.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:26:26.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:26:26.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:26:26.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:26:26.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:26:26.403 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:26:26.403 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:26:26.403 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:26.403 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:26.403 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:26.403 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:26:26.403 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:26:26.403 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:26:26.403 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:26:26.403 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:26:26.429 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:26.429 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:26.429 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:26.429 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:26:26.563 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:26:26.564 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:26:26.564 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:26.564 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:26.565 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:26.565 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:26:26.565 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:26.565 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:26.565 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:26.565 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:26:26.565 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:26.565 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:26.566 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:26.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:26.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:26.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:26.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 27 2024-09-11T06:26:26.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:26.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:26.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:26.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=111, tag_tid=20, entry_tid=14623], [object_number=110, tag_tid=20, entry_tid=14622], [object_number=109, tag_tid=20, entry_tid=14621], [object_number=108, tag_tid=20, entry_tid=14620]]], state=connected] 2024-09-11T06:26:26.623 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:26.626 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:26.626 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:26:26.626 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:26.627 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:26.627 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:26.627 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:26:26.627 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:26.627 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:26.627 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:26.628 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:26:26.628 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:26.628 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:26.628 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:26.629 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:26.629 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:26.629 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:26.675 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 27 2024-09-11T06:26:26.675 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:26.675 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:26.675 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:26.675 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=111, tag_tid=20, entry_tid=14623], [object_number=110, tag_tid=20, entry_tid=14622], [object_number=109, tag_tid=20, entry_tid=14621], [object_number=108, tag_tid=20, entry_tid=14620]]], state=connected] 2024-09-11T06:26:26.678 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:26.680 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=111, tag_tid=20, entry_tid=14623' 2024-09-11T06:26:26.680 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=111, tag_tid=20, entry_tid=14623' 2024-09-11T06:26:26.680 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=111, tag_tid=20, entry_tid=14623' '!=' 'object_number=108, tag_tid=20, entry_tid=14528' 2024-09-11T06:26:26.680 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:26:26.680 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=111, tag_tid=20, entry_tid=14623' = 'object_number=108, tag_tid=20, entry_tid=14528' 2024-09-11T06:26:26.680 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=111, tag_tid=20, entry_tid=14623' 2024-09-11T06:26:26.680 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:26.681 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:26.681 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:26.681 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:26:26.682 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=111, tag_tid=20, entry_tid=14623' 2024-09-11T06:26:26.682 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:26.682 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:26.683 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:26:26.683 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:26.684 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:26.684 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:26.685 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:26:26.685 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=111, tag_tid=20, entry_tid=14623' 2024-09-11T06:26:26.685 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:26.685 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:26.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=14623 2024-09-11T06:26:26.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 14623 2024-09-11T06:26:26.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:26:26.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:26:26.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:26:26.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:26:26.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:26.889 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:26:26.889 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:26:26.889 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:26:26.889 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:26:26.889 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:26:26.889 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:26:26.889 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:26:26.889 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:26.890 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:26.890 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:26:26.890 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:26:26.890 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:26:26.890 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:26:26.893 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:26:26.893 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:26:26.893 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:26:26.893 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:26:26.894 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:26:26.894 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:26.894 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:26:26.894 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:26:26.894 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:26:26.894 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:26:26.895 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:26:26.895 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:26:26.895 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:26:26.895 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:26.895 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:26:26.895 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:26.895 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:26:26.895 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:26.895 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:26:26.895 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:26:26.895 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:26:26.895 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:26:26.895 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:26:26.896 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:26:26.896 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:26:26.896 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:26:26.896 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:26.896 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:26.896 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:26.896 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:26:26.896 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:26:26.896 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:26:26.896 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:26:26.896 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:26:26.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:26.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:26.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:26.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:26:27.050 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:26:27.058 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:26:27.058 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:27.058 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:27.058 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:27.059 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:26:27.059 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:27.059 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:27.059 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:27.059 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:26:27.059 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:27.059 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:27.059 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:27.059 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:27.059 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:27.059 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:27.100 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 27 2024-09-11T06:26:27.100 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:27.101 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:27.101 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:27.101 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=110, tag_tid=20, entry_tid=14714], [object_number=109, tag_tid=20, entry_tid=14713], [object_number=108, tag_tid=20, entry_tid=14712], [object_number=111, tag_tid=20, entry_tid=14711]]], state=connected] 2024-09-11T06:26:27.104 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:27.105 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:27.106 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:26:27.106 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:27.106 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:27.106 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:27.107 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:26:27.107 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:27.107 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:27.107 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:27.107 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:26:27.107 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:27.107 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:27.107 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:27.108 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:27.108 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:27.108 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:27.154 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 27 2024-09-11T06:26:27.154 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:27.155 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:27.155 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:27.155 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=110, tag_tid=20, entry_tid=14714], [object_number=109, tag_tid=20, entry_tid=14713], [object_number=108, tag_tid=20, entry_tid=14712], [object_number=111, tag_tid=20, entry_tid=14711]]], state=connected] 2024-09-11T06:26:27.157 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:27.160 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=110, tag_tid=20, entry_tid=14714' 2024-09-11T06:26:27.160 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=110, tag_tid=20, entry_tid=14714' 2024-09-11T06:26:27.160 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=110, tag_tid=20, entry_tid=14714' '!=' 'object_number=111, tag_tid=20, entry_tid=14623' 2024-09-11T06:26:27.161 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:26:27.161 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=110, tag_tid=20, entry_tid=14714' = 'object_number=111, tag_tid=20, entry_tid=14623' 2024-09-11T06:26:27.161 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=110, tag_tid=20, entry_tid=14714' 2024-09-11T06:26:27.161 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:27.161 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:27.161 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:27.164 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:26:27.164 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=110, tag_tid=20, entry_tid=14714' 2024-09-11T06:26:27.164 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:27.164 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:27.164 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:26:27.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:27.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:27.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:27.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:26:27.167 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=110, tag_tid=20, entry_tid=14714' 2024-09-11T06:26:27.167 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:27.167 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:27.167 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=14714 2024-09-11T06:26:27.167 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 14714 2024-09-11T06:26:27.167 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:26:27.168 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:26:27.168 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:26:27.367 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:26:27.368 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:27.368 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:26:27.368 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:26:27.368 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:26:27.368 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:26:27.368 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:26:27.368 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:26:27.368 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:26:27.368 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:27.368 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:27.368 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:26:27.369 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:26:27.369 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:26:27.369 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:26:27.369 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:26:27.369 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:26:27.373 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:26:27.373 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:26:27.373 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:26:27.373 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:27.373 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:26:27.373 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:26:27.373 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:26:27.374 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:26:27.374 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:26:27.374 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:26:27.374 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:26:27.374 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:27.374 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:26:27.374 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:27.374 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:26:27.374 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:27.374 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:26:27.374 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:26:27.374 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:26:27.374 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:26:27.374 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:26:27.374 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:26:27.375 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:26:27.375 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:26:27.375 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:27.375 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:27.375 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:27.375 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:26:27.375 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:26:27.375 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:26:27.375 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:26:27.375 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:26:27.398 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:27.399 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:27.399 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:27.399 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:26:27.525 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:26:27.526 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:26:27.527 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:27.527 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:27.527 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:27.527 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:26:27.527 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:27.527 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:27.527 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:27.527 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:26:27.527 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:27.528 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:27.528 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:27.528 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:27.529 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:27.529 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:27.583 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 27 2024-09-11T06:26:27.583 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:27.583 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:27.584 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:27.584 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=110, tag_tid=20, entry_tid=14802], [object_number=109, tag_tid=20, entry_tid=14801], [object_number=108, tag_tid=20, entry_tid=14800], [object_number=111, tag_tid=20, entry_tid=14799]]], state=connected] 2024-09-11T06:26:27.587 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:27.588 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:27.590 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:26:27.590 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:27.590 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:27.590 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:27.590 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:26:27.590 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:27.590 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:27.590 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:27.590 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:26:27.590 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:27.590 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:27.590 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:27.591 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:27.591 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:27.591 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:27.858 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 27 2024-09-11T06:26:27.858 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:27.858 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:27.858 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:27.858 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=110, tag_tid=20, entry_tid=14802], [object_number=109, tag_tid=20, entry_tid=14801], [object_number=108, tag_tid=20, entry_tid=14800], [object_number=111, tag_tid=20, entry_tid=14799]]], state=connected] 2024-09-11T06:26:27.862 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:27.864 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=110, tag_tid=20, entry_tid=14802' 2024-09-11T06:26:27.864 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=110, tag_tid=20, entry_tid=14802' 2024-09-11T06:26:27.864 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=110, tag_tid=20, entry_tid=14802' '!=' 'object_number=110, tag_tid=20, entry_tid=14714' 2024-09-11T06:26:27.864 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:26:27.864 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=110, tag_tid=20, entry_tid=14802' = 'object_number=110, tag_tid=20, entry_tid=14714' 2024-09-11T06:26:27.864 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=110, tag_tid=20, entry_tid=14802' 2024-09-11T06:26:27.865 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:27.865 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:27.865 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:27.865 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:26:27.866 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=110, tag_tid=20, entry_tid=14802' 2024-09-11T06:26:27.866 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:27.867 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:27.867 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:26:27.868 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:27.868 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:27.869 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:27.869 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:26:27.870 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=110, tag_tid=20, entry_tid=14802' 2024-09-11T06:26:27.870 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:27.870 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:27.872 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=14802 2024-09-11T06:26:27.872 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 14802 2024-09-11T06:26:27.872 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:26:27.872 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:26:27.872 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:26:28.074 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:26:28.074 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:28.074 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:26:28.074 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:26:28.074 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:26:28.074 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:26:28.074 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:26:28.074 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:26:28.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:26:28.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:28.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:28.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:26:28.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:26:28.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:26:28.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:26:28.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:26:28.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:26:28.078 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:26:28.078 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:26:28.078 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:26:28.078 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:28.078 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:26:28.079 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:26:28.079 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:26:28.079 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:26:28.079 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:26:28.079 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:26:28.079 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:26:28.079 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:28.079 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:26:28.079 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:28.079 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:26:28.079 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:28.079 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:26:28.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:26:28.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:26:28.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:26:28.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:26:28.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:26:28.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:26:28.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:26:28.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:28.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:28.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:28.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:26:28.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:26:28.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:26:28.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:26:28.081 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:26:28.102 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:28.102 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:28.102 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:28.103 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:26:28.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:26:28.254 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:26:28.254 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:28.254 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:28.254 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:28.254 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:26:28.254 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:28.254 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:28.254 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:28.255 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:26:28.255 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:28.255 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:28.255 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:28.257 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:28.257 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:28.257 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:28.315 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 28 2024-09-11T06:26:28.315 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:28.315 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:28.315 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:28.315 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=115, tag_tid=20, entry_tid=14847], [object_number=114, tag_tid=20, entry_tid=14846], [object_number=113, tag_tid=20, entry_tid=14845], [object_number=112, tag_tid=20, entry_tid=14844]]], state=connected] 2024-09-11T06:26:28.319 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:28.320 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:28.321 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:26:28.321 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:28.321 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:28.321 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:28.321 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:26:28.322 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:28.322 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:28.322 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:28.322 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:26:28.322 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:28.322 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:28.322 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:28.323 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:28.323 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:28.323 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:28.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 28 2024-09-11T06:26:28.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:28.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:28.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:28.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=115, tag_tid=20, entry_tid=14847], [object_number=114, tag_tid=20, entry_tid=14846], [object_number=113, tag_tid=20, entry_tid=14845], [object_number=112, tag_tid=20, entry_tid=14844]]], state=connected] 2024-09-11T06:26:28.373 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:28.375 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=115, tag_tid=20, entry_tid=14847' 2024-09-11T06:26:28.375 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=115, tag_tid=20, entry_tid=14847' 2024-09-11T06:26:28.375 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=115, tag_tid=20, entry_tid=14847' '!=' 'object_number=110, tag_tid=20, entry_tid=14802' 2024-09-11T06:26:28.375 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:26:28.376 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=115, tag_tid=20, entry_tid=14847' = 'object_number=110, tag_tid=20, entry_tid=14802' 2024-09-11T06:26:28.376 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=115, tag_tid=20, entry_tid=14847' 2024-09-11T06:26:28.376 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:28.376 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:28.376 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:28.376 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:26:28.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=115, tag_tid=20, entry_tid=14847' 2024-09-11T06:26:28.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:28.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:28.378 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:26:28.379 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:28.379 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:28.379 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:28.380 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:26:28.381 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=115, tag_tid=20, entry_tid=14847' 2024-09-11T06:26:28.381 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:28.381 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:28.382 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=14847 2024-09-11T06:26:28.382 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 14847 2024-09-11T06:26:28.382 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:26:28.382 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:26:28.382 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:26:28.584 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:26:28.584 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:28.584 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:26:28.584 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:26:28.584 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:26:28.584 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:26:28.584 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:26:28.584 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:26:28.584 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:26:28.584 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:28.584 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:28.585 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:26:28.585 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:26:28.585 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:26:28.585 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:26:28.585 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:26:28.585 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:26:28.590 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:26:28.591 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:26:28.591 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:26:28.591 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:28.591 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:26:28.591 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:26:28.591 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:26:28.592 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:26:28.592 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:26:28.592 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:26:28.592 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:26:28.592 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:28.592 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:26:28.593 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:28.593 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:26:28.593 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:28.593 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:26:28.593 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:26:28.594 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:26:28.594 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:26:28.594 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:26:28.594 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:26:28.594 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:26:28.594 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:26:28.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:28.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:28.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:28.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:26:28.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:26:28.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:26:28.596 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:26:28.596 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:26:28.615 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:28.615 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:28.616 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:28.616 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:26:28.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:26:28.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:26:28.745 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:28.745 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:28.745 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:28.745 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:26:28.745 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:28.745 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:28.745 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:28.745 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:26:28.746 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:28.746 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:28.746 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:28.747 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:28.748 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:28.748 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:28.794 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 28 2024-09-11T06:26:28.794 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:28.794 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:28.795 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:28.795 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=113, tag_tid=20, entry_tid=14941], [object_number=112, tag_tid=20, entry_tid=14940], [object_number=115, tag_tid=20, entry_tid=14939], [object_number=114, tag_tid=20, entry_tid=14938]]], state=connected] 2024-09-11T06:26:28.798 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:28.799 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:28.800 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:26:28.800 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:28.800 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:28.801 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:28.801 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:26:28.801 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:28.801 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:28.801 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:28.801 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:26:28.801 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:28.801 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:28.801 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:28.802 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:28.802 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:28.802 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:28.848 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 28 2024-09-11T06:26:28.848 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:28.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:28.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:28.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=113, tag_tid=20, entry_tid=14941], [object_number=112, tag_tid=20, entry_tid=14940], [object_number=115, tag_tid=20, entry_tid=14939], [object_number=114, tag_tid=20, entry_tid=14938]]], state=connected] 2024-09-11T06:26:28.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:28.854 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=113, tag_tid=20, entry_tid=14941' 2024-09-11T06:26:28.854 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=113, tag_tid=20, entry_tid=14941' 2024-09-11T06:26:28.854 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=113, tag_tid=20, entry_tid=14941' '!=' 'object_number=115, tag_tid=20, entry_tid=14847' 2024-09-11T06:26:28.854 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:26:28.854 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=113, tag_tid=20, entry_tid=14941' = 'object_number=115, tag_tid=20, entry_tid=14847' 2024-09-11T06:26:28.854 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=113, tag_tid=20, entry_tid=14941' 2024-09-11T06:26:28.854 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:28.854 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:28.854 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:28.855 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:26:28.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=20, entry_tid=14941' 2024-09-11T06:26:28.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:28.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:28.857 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:26:28.857 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:28.857 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:28.857 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:28.858 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:26:28.859 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=20, entry_tid=14941' 2024-09-11T06:26:28.859 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:28.859 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:28.860 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=14941 2024-09-11T06:26:28.861 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 14941 2024-09-11T06:26:28.861 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:26:28.861 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:26:28.861 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:26:29.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:26:29.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:29.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:26:29.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:26:29.063 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:26:29.063 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:26:29.063 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:26:29.063 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:26:29.063 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:26:29.063 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:29.063 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:29.063 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:26:29.063 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:26:29.063 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:26:29.063 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:26:29.064 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:26:29.064 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:26:29.067 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:26:29.067 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:26:29.067 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:26:29.067 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:29.067 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:26:29.067 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:26:29.067 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:26:29.067 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:26:29.067 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:26:29.067 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:26:29.068 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:26:29.068 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:29.068 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:26:29.068 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:29.068 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:26:29.068 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:29.069 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:26:29.069 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:26:29.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:26:29.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:26:29.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:26:29.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:26:29.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:26:29.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:26:29.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:29.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:29.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:29.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:26:29.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:26:29.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:26:29.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:26:29.071 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:26:29.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:29.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:29.095 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:29.095 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:26:29.225 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:26:29.226 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:26:29.226 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:29.226 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:29.226 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:29.226 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:26:29.226 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:29.227 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:29.227 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:29.227 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:26:29.227 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:29.228 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:29.228 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:29.228 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:29.229 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:29.229 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:29.284 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 28 2024-09-11T06:26:29.284 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:29.284 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:29.285 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:29.285 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=114, tag_tid=20, entry_tid=15026], [object_number=113, tag_tid=20, entry_tid=15025], [object_number=112, tag_tid=20, entry_tid=15024], [object_number=115, tag_tid=20, entry_tid=15023]]], state=connected] 2024-09-11T06:26:29.288 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:29.289 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:29.291 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:26:29.291 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:29.291 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:29.291 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:29.291 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:26:29.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:29.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:29.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:29.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:26:29.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:29.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:29.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:29.293 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:29.293 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:29.293 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:29.340 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 28 2024-09-11T06:26:29.340 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:29.340 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:29.340 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:29.340 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=114, tag_tid=20, entry_tid=15026], [object_number=113, tag_tid=20, entry_tid=15025], [object_number=112, tag_tid=20, entry_tid=15024], [object_number=115, tag_tid=20, entry_tid=15023]]], state=connected] 2024-09-11T06:26:29.343 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:29.345 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=114, tag_tid=20, entry_tid=15026' 2024-09-11T06:26:29.345 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=114, tag_tid=20, entry_tid=15026' 2024-09-11T06:26:29.346 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=114, tag_tid=20, entry_tid=15026' '!=' 'object_number=113, tag_tid=20, entry_tid=14941' 2024-09-11T06:26:29.346 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:26:29.346 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=114, tag_tid=20, entry_tid=15026' = 'object_number=113, tag_tid=20, entry_tid=14941' 2024-09-11T06:26:29.346 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=114, tag_tid=20, entry_tid=15026' 2024-09-11T06:26:29.346 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:29.346 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:29.346 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:29.346 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:26:29.347 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=114, tag_tid=20, entry_tid=15026' 2024-09-11T06:26:29.347 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:29.347 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:29.348 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:26:29.349 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:29.349 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:29.349 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:29.350 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:26:29.350 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=114, tag_tid=20, entry_tid=15026' 2024-09-11T06:26:29.351 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:29.351 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:29.352 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=15026 2024-09-11T06:26:29.352 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 15026 2024-09-11T06:26:29.352 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:26:29.352 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:26:29.352 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:26:29.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:26:29.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:29.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:26:29.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:26:29.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:26:29.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:26:29.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:26:29.555 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:26:29.555 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:26:29.555 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:29.555 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:29.555 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:26:29.555 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:26:29.555 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:26:29.555 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:26:29.556 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:26:29.556 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:26:29.559 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:26:29.559 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:26:29.559 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:26:29.559 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:29.559 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:26:29.559 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:26:29.559 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:26:29.559 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:26:29.560 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:26:29.560 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:26:29.560 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:26:29.560 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:29.560 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:26:29.560 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:29.560 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:26:29.560 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:29.560 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:26:29.560 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:26:29.560 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:26:29.561 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:26:29.561 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:26:29.561 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:26:29.561 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:26:29.561 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:26:29.561 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:29.561 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:29.561 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:29.561 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:26:29.561 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:26:29.561 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:26:29.561 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:26:29.562 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:26:29.586 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:29.586 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:29.586 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:29.586 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:26:29.712 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:26:29.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:26:29.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:29.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:29.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:29.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:26:29.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:29.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:29.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:29.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:26:29.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:29.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:29.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:29.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:29.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:29.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:29.765 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 28 2024-09-11T06:26:29.765 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:29.765 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:29.765 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:29.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=115, tag_tid=20, entry_tid=15111], [object_number=114, tag_tid=20, entry_tid=15110], [object_number=113, tag_tid=20, entry_tid=15109], [object_number=112, tag_tid=20, entry_tid=15108]]], state=connected] 2024-09-11T06:26:29.768 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:29.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:29.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:26:29.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:29.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:29.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:29.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:26:29.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:29.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:29.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:29.772 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:26:29.772 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:29.772 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:29.772 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:29.772 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:29.773 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:29.773 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:29.820 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 28 2024-09-11T06:26:29.820 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:29.820 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:29.820 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:29.820 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=115, tag_tid=20, entry_tid=15111], [object_number=114, tag_tid=20, entry_tid=15110], [object_number=113, tag_tid=20, entry_tid=15109], [object_number=112, tag_tid=20, entry_tid=15108]]], state=connected] 2024-09-11T06:26:29.823 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:29.825 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=115, tag_tid=20, entry_tid=15111' 2024-09-11T06:26:29.825 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=115, tag_tid=20, entry_tid=15111' 2024-09-11T06:26:29.825 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=115, tag_tid=20, entry_tid=15111' '!=' 'object_number=114, tag_tid=20, entry_tid=15026' 2024-09-11T06:26:29.826 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:26:29.826 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=115, tag_tid=20, entry_tid=15111' = 'object_number=114, tag_tid=20, entry_tid=15026' 2024-09-11T06:26:29.826 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=115, tag_tid=20, entry_tid=15111' 2024-09-11T06:26:29.826 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:29.826 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:29.826 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:29.826 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:26:29.827 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=115, tag_tid=20, entry_tid=15111' 2024-09-11T06:26:29.827 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:29.827 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:29.828 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:26:29.829 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:29.829 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:29.829 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:29.830 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:26:29.830 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=115, tag_tid=20, entry_tid=15111' 2024-09-11T06:26:29.830 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:29.830 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:29.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=15111 2024-09-11T06:26:29.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 15111 2024-09-11T06:26:29.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:26:29.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:26:29.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:26:30.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:26:30.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:30.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:26:30.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:26:30.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:26:30.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:26:30.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:26:30.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:26:30.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:26:30.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:30.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:30.035 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:26:30.035 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:26:30.035 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:26:30.035 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:26:30.035 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:26:30.035 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:26:30.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:26:30.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:26:30.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:26:30.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:30.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:26:30.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:26:30.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:26:30.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:26:30.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:26:30.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:26:30.040 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:26:30.040 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:30.040 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:26:30.040 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:30.040 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:26:30.040 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:30.040 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:26:30.040 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:26:30.040 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:26:30.040 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:26:30.040 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:26:30.040 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:26:30.040 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:26:30.040 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:26:30.041 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:30.041 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:30.041 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:30.041 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:26:30.041 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:26:30.041 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:26:30.041 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:26:30.041 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:26:30.065 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:30.065 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:30.065 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:30.065 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:26:30.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:26:30.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:26:30.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:30.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:30.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:30.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:26:30.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:30.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:30.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:30.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:26:30.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:30.196 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:30.196 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:30.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:30.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:30.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:30.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 28 2024-09-11T06:26:30.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:30.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:30.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:30.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=115, tag_tid=20, entry_tid=15195], [object_number=114, tag_tid=20, entry_tid=15194], [object_number=113, tag_tid=20, entry_tid=15193], [object_number=112, tag_tid=20, entry_tid=15192]]], state=connected] 2024-09-11T06:26:30.254 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:30.256 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:30.260 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:26:30.260 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:30.260 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:30.260 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:30.260 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:26:30.261 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:30.261 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:30.261 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:30.261 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:26:30.261 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:30.261 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:30.261 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:30.262 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:30.262 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:30.262 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:30.307 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 28 2024-09-11T06:26:30.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:30.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:30.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:30.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=115, tag_tid=20, entry_tid=15195], [object_number=114, tag_tid=20, entry_tid=15194], [object_number=113, tag_tid=20, entry_tid=15193], [object_number=112, tag_tid=20, entry_tid=15192]]], state=connected] 2024-09-11T06:26:30.311 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:30.313 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=115, tag_tid=20, entry_tid=15195' 2024-09-11T06:26:30.313 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=115, tag_tid=20, entry_tid=15195' 2024-09-11T06:26:30.313 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=115, tag_tid=20, entry_tid=15195' '!=' 'object_number=115, tag_tid=20, entry_tid=15111' 2024-09-11T06:26:30.313 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:26:30.313 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=115, tag_tid=20, entry_tid=15195' = 'object_number=115, tag_tid=20, entry_tid=15111' 2024-09-11T06:26:30.313 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=115, tag_tid=20, entry_tid=15195' 2024-09-11T06:26:30.313 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:30.314 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:30.314 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:30.314 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:26:30.315 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=115, tag_tid=20, entry_tid=15195' 2024-09-11T06:26:30.315 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:30.315 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:30.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:26:30.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:30.317 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:30.317 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:30.317 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:26:30.318 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=115, tag_tid=20, entry_tid=15195' 2024-09-11T06:26:30.318 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:30.318 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:30.320 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=15195 2024-09-11T06:26:30.320 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 15195 2024-09-11T06:26:30.320 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:26:30.320 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:26:30.320 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:26:30.521 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:26:30.521 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:30.522 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:26:30.522 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:26:30.522 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:26:30.522 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:26:30.522 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:26:30.522 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:26:30.522 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:26:30.522 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:30.522 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:30.522 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:26:30.522 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:26:30.522 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:26:30.522 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:26:30.523 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:26:30.523 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:26:30.527 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:26:30.527 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:26:30.527 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:26:30.527 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:30.527 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:26:30.528 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:26:30.528 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:26:30.528 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:26:30.528 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:26:30.528 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:26:30.528 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:26:30.528 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:30.528 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:26:30.528 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:30.528 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:26:30.528 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:30.528 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:26:30.528 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:26:30.529 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:26:30.529 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:26:30.529 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:26:30.529 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:26:30.529 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:26:30.529 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:26:30.529 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:30.529 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:30.529 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:30.529 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:26:30.529 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:26:30.529 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:26:30.529 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:26:30.530 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:26:30.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:30.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:30.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:30.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:26:30.685 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:26:30.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:26:30.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:30.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:30.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:30.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:26:30.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:30.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:30.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:30.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:26:30.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:30.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:30.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:30.690 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:30.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:30.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:30.737 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 28 2024-09-11T06:26:30.737 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:30.737 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:30.737 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:30.737 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=112, tag_tid=20, entry_tid=15244], [object_number=115, tag_tid=20, entry_tid=15243], [object_number=114, tag_tid=20, entry_tid=15242], [object_number=113, tag_tid=20, entry_tid=15241]]], state=connected] 2024-09-11T06:26:30.740 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:30.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:30.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:26:30.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:30.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:30.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:30.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:26:30.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:30.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:30.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:30.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:26:30.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:30.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:30.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:30.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:30.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:30.745 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:30.791 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 28 2024-09-11T06:26:30.791 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:30.792 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:30.792 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:30.792 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=112, tag_tid=20, entry_tid=15244], [object_number=115, tag_tid=20, entry_tid=15243], [object_number=114, tag_tid=20, entry_tid=15242], [object_number=113, tag_tid=20, entry_tid=15241]]], state=connected] 2024-09-11T06:26:30.795 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:30.796 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=112, tag_tid=20, entry_tid=15244' 2024-09-11T06:26:30.796 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=112, tag_tid=20, entry_tid=15244' 2024-09-11T06:26:30.796 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=112, tag_tid=20, entry_tid=15244' '!=' 'object_number=115, tag_tid=20, entry_tid=15195' 2024-09-11T06:26:30.797 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:26:30.797 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=112, tag_tid=20, entry_tid=15244' = 'object_number=115, tag_tid=20, entry_tid=15195' 2024-09-11T06:26:30.797 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=112, tag_tid=20, entry_tid=15244' 2024-09-11T06:26:30.798 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:30.798 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:30.798 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:30.798 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:26:30.798 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=112, tag_tid=20, entry_tid=15244' 2024-09-11T06:26:30.799 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:30.799 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:30.800 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:26:30.800 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:30.800 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:30.800 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:30.801 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:26:30.802 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=112, tag_tid=20, entry_tid=15244' 2024-09-11T06:26:30.802 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:30.802 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:30.803 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=15244 2024-09-11T06:26:30.803 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 15244 2024-09-11T06:26:30.804 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:26:30.804 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:26:30.804 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:26:31.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:26:31.007 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:31.007 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:26:31.007 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:26:31.007 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:26:31.007 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:26:31.007 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:26:31.008 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:26:31.008 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:26:31.008 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:31.008 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:31.008 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:26:31.009 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:26:31.009 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:26:31.009 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:26:31.013 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:26:31.013 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:26:31.013 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:26:31.013 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:26:31.013 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:26:31.013 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:31.013 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:26:31.013 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:26:31.013 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:26:31.013 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:26:31.013 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:26:31.014 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:26:31.014 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:26:31.014 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:31.014 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:26:31.014 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:31.014 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:26:31.014 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:31.014 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:26:31.014 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:26:31.014 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:26:31.014 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:26:31.014 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:26:31.014 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:26:31.015 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:26:31.015 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:26:31.015 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:31.015 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:31.015 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:31.015 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:26:31.015 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:26:31.015 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:26:31.015 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:26:31.015 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:26:31.038 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:31.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:31.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:31.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:26:31.171 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:26:31.173 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:26:31.173 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:31.173 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:31.173 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:31.173 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:26:31.173 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:31.173 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:31.173 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:31.173 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:26:31.173 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:31.174 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:31.174 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:31.175 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:31.175 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:31.176 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:31.227 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 28 2024-09-11T06:26:31.228 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:31.228 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:31.228 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:31.228 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=114, tag_tid=20, entry_tid=15322], [object_number=113, tag_tid=20, entry_tid=15321], [object_number=112, tag_tid=20, entry_tid=15320], [object_number=115, tag_tid=20, entry_tid=15319]]], state=connected] 2024-09-11T06:26:31.231 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:31.232 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:31.233 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:26:31.233 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:31.234 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:31.234 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:31.234 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:26:31.234 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:31.234 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:31.234 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:31.234 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:26:31.234 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:31.234 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:31.234 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:31.235 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:31.235 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:31.235 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:31.282 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 28 2024-09-11T06:26:31.283 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:31.283 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:31.283 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:31.283 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=114, tag_tid=20, entry_tid=15322], [object_number=113, tag_tid=20, entry_tid=15321], [object_number=112, tag_tid=20, entry_tid=15320], [object_number=115, tag_tid=20, entry_tid=15319]]], state=connected] 2024-09-11T06:26:31.290 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:31.290 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=114, tag_tid=20, entry_tid=15322' 2024-09-11T06:26:31.290 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=114, tag_tid=20, entry_tid=15322' 2024-09-11T06:26:31.290 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=114, tag_tid=20, entry_tid=15322' '!=' 'object_number=112, tag_tid=20, entry_tid=15244' 2024-09-11T06:26:31.290 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:26:31.290 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=114, tag_tid=20, entry_tid=15322' = 'object_number=112, tag_tid=20, entry_tid=15244' 2024-09-11T06:26:31.291 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=114, tag_tid=20, entry_tid=15322' 2024-09-11T06:26:31.291 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:31.291 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:31.291 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:31.291 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:26:31.291 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=114, tag_tid=20, entry_tid=15322' 2024-09-11T06:26:31.291 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:31.291 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:31.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:26:31.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:31.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:31.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:31.293 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:26:31.293 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=114, tag_tid=20, entry_tid=15322' 2024-09-11T06:26:31.293 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:31.293 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:31.294 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=15322 2024-09-11T06:26:31.294 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 15322 2024-09-11T06:26:31.295 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:26:31.295 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:26:31.295 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:26:31.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:26:31.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:31.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:26:31.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:26:31.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:26:31.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:26:31.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:26:31.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:26:31.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:26:31.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:31.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:31.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:26:31.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:26:31.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:26:31.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:26:31.498 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:26:31.498 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:26:31.501 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:26:31.502 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:26:31.502 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:26:31.502 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:31.502 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:26:31.502 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:26:31.502 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:26:31.502 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:26:31.502 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:26:31.502 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:26:31.502 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:26:31.502 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:31.502 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:26:31.503 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:31.503 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:26:31.503 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:31.503 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:26:31.503 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:26:31.503 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:26:31.503 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:26:31.503 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:26:31.503 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:26:31.503 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:26:31.503 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:26:31.503 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:31.503 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:31.503 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:31.504 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:26:31.504 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:26:31.504 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:26:31.504 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:26:31.504 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:26:31.527 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:31.527 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:31.527 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:31.527 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:26:31.658 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:26:31.659 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:26:31.659 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:31.659 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:31.659 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:31.659 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:26:31.660 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:31.660 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:31.660 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:31.660 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:26:31.660 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:31.660 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:31.660 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:31.662 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:31.663 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:31.663 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:31.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 28 2024-09-11T06:26:31.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:31.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:31.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:31.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=113, tag_tid=20, entry_tid=15405], [object_number=112, tag_tid=20, entry_tid=15404], [object_number=115, tag_tid=20, entry_tid=15403], [object_number=114, tag_tid=20, entry_tid=15402]]], state=connected] 2024-09-11T06:26:31.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:31.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:31.716 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:26:31.716 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:31.716 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:31.716 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:31.716 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:26:31.716 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:31.716 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:31.716 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:31.716 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:26:31.716 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:31.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:31.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:31.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:31.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:31.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:31.764 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 28 2024-09-11T06:26:31.764 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:31.764 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:31.764 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:31.764 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=113, tag_tid=20, entry_tid=15405], [object_number=112, tag_tid=20, entry_tid=15404], [object_number=115, tag_tid=20, entry_tid=15403], [object_number=114, tag_tid=20, entry_tid=15402]]], state=connected] 2024-09-11T06:26:31.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:31.769 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=113, tag_tid=20, entry_tid=15405' 2024-09-11T06:26:31.769 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=113, tag_tid=20, entry_tid=15405' 2024-09-11T06:26:31.769 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=113, tag_tid=20, entry_tid=15405' '!=' 'object_number=114, tag_tid=20, entry_tid=15322' 2024-09-11T06:26:31.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:26:31.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=113, tag_tid=20, entry_tid=15405' = 'object_number=114, tag_tid=20, entry_tid=15322' 2024-09-11T06:26:31.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=113, tag_tid=20, entry_tid=15405' 2024-09-11T06:26:31.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:31.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:31.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:31.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:26:31.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=20, entry_tid=15405' 2024-09-11T06:26:31.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:31.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:31.772 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:26:31.773 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:31.773 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:31.773 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:31.774 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:26:31.774 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=20, entry_tid=15405' 2024-09-11T06:26:31.775 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:31.775 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:31.776 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=15405 2024-09-11T06:26:31.776 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 15405 2024-09-11T06:26:31.776 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:26:31.776 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:26:31.776 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:26:31.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:26:31.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:31.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:26:31.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:26:31.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:26:31.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:26:31.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:26:31.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:26:31.979 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:26:31.979 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:31.979 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:31.979 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:26:31.979 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:26:31.979 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:26:31.979 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:26:31.979 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:26:31.980 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:26:31.983 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:26:31.983 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:26:31.983 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:26:31.983 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:31.983 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:26:31.983 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:26:31.983 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:26:31.983 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:26:31.983 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:26:31.983 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:26:31.983 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:26:31.983 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:31.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:26:31.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:31.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:26:31.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:31.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:26:31.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:26:31.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:26:31.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:26:31.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:26:31.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:26:31.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:26:31.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:26:31.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:31.985 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:31.985 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:31.985 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:26:31.985 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:26:31.985 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:26:31.985 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:26:31.985 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:26:32.010 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:32.010 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:32.010 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:32.010 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:26:32.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:26:32.139 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:26:32.139 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:32.139 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:32.139 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:32.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:26:32.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:32.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:32.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:32.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:26:32.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:32.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:32.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:32.143 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:32.143 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:32.143 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:32.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 28 2024-09-11T06:26:32.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:32.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:32.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:32.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=114, tag_tid=20, entry_tid=15502], [object_number=113, tag_tid=20, entry_tid=15501], [object_number=112, tag_tid=20, entry_tid=15500], [object_number=115, tag_tid=20, entry_tid=15499]]], state=connected] 2024-09-11T06:26:32.197 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:32.200 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:32.200 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:26:32.200 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:32.200 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:32.200 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:32.200 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:26:32.201 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:32.201 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:32.201 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:32.201 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:26:32.201 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:32.201 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:32.201 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:32.202 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:32.202 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:32.202 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:32.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 28 2024-09-11T06:26:32.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:32.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:32.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:32.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=114, tag_tid=20, entry_tid=15502], [object_number=113, tag_tid=20, entry_tid=15501], [object_number=112, tag_tid=20, entry_tid=15500], [object_number=115, tag_tid=20, entry_tid=15499]]], state=connected] 2024-09-11T06:26:32.252 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:32.254 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=114, tag_tid=20, entry_tid=15502' 2024-09-11T06:26:32.254 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=114, tag_tid=20, entry_tid=15502' 2024-09-11T06:26:32.254 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=114, tag_tid=20, entry_tid=15502' '!=' 'object_number=113, tag_tid=20, entry_tid=15405' 2024-09-11T06:26:32.254 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:26:32.254 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=114, tag_tid=20, entry_tid=15502' = 'object_number=113, tag_tid=20, entry_tid=15405' 2024-09-11T06:26:32.254 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=114, tag_tid=20, entry_tid=15502' 2024-09-11T06:26:32.255 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:32.255 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:32.255 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:32.256 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:26:32.256 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=114, tag_tid=20, entry_tid=15502' 2024-09-11T06:26:32.256 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:32.256 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:32.257 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:26:32.258 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:32.258 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:32.258 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:32.259 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:26:32.259 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=114, tag_tid=20, entry_tid=15502' 2024-09-11T06:26:32.260 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:32.260 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:32.261 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=15502 2024-09-11T06:26:32.261 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 15502 2024-09-11T06:26:32.261 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:26:32.261 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:26:32.261 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:26:32.463 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:26:32.463 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:32.463 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:26:32.463 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:26:32.463 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:26:32.463 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:26:32.463 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:26:32.463 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:26:32.463 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:26:32.463 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:32.463 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:32.464 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:26:32.464 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:26:32.464 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:26:32.464 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:26:32.464 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:26:32.464 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:26:32.468 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:26:32.468 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:26:32.468 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:26:32.468 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:32.469 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:26:32.469 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:26:32.469 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:26:32.469 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:26:32.469 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:26:32.469 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:26:32.469 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:26:32.469 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:32.469 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:26:32.469 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:32.469 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:26:32.470 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:32.470 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:26:32.470 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:26:32.470 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:26:32.470 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:26:32.470 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:26:32.470 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:26:32.470 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:26:32.470 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:26:32.470 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:32.470 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:32.470 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:32.470 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:26:32.471 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:26:32.471 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:26:32.471 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:26:32.471 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:26:32.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:32.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:32.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:32.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:26:32.623 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:26:32.625 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:26:32.625 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:32.625 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:32.625 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:32.625 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:26:32.626 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:32.626 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:32.626 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:32.626 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:26:32.626 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:32.626 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:32.626 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:32.627 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:32.627 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:32.627 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:32.682 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 28 2024-09-11T06:26:32.682 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:32.682 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:32.682 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:32.682 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=115, tag_tid=20, entry_tid=15595], [object_number=114, tag_tid=20, entry_tid=15594], [object_number=113, tag_tid=20, entry_tid=15593], [object_number=112, tag_tid=20, entry_tid=15592]]], state=connected] 2024-09-11T06:26:32.685 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:32.686 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:32.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:26:32.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:32.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:32.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:32.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:26:32.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:32.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:32.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:32.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:26:32.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:32.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:32.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:32.689 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:32.689 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:32.689 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:32.737 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 28 2024-09-11T06:26:32.737 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:32.737 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:32.737 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:32.737 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=115, tag_tid=20, entry_tid=15595], [object_number=114, tag_tid=20, entry_tid=15594], [object_number=113, tag_tid=20, entry_tid=15593], [object_number=112, tag_tid=20, entry_tid=15592]]], state=connected] 2024-09-11T06:26:32.741 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:32.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=115, tag_tid=20, entry_tid=15595' 2024-09-11T06:26:32.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=115, tag_tid=20, entry_tid=15595' 2024-09-11T06:26:32.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=115, tag_tid=20, entry_tid=15595' '!=' 'object_number=114, tag_tid=20, entry_tid=15502' 2024-09-11T06:26:32.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:26:32.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=115, tag_tid=20, entry_tid=15595' = 'object_number=114, tag_tid=20, entry_tid=15502' 2024-09-11T06:26:32.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=115, tag_tid=20, entry_tid=15595' 2024-09-11T06:26:32.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:32.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:32.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:32.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:26:32.745 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=115, tag_tid=20, entry_tid=15595' 2024-09-11T06:26:32.745 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:32.745 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:32.746 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:26:32.748 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:32.749 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:32.749 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:32.749 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:26:32.749 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=115, tag_tid=20, entry_tid=15595' 2024-09-11T06:26:32.749 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:32.749 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:32.750 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=15595 2024-09-11T06:26:32.750 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 15595 2024-09-11T06:26:32.750 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:26:32.750 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:26:32.750 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:26:32.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:26:32.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:32.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:26:32.952 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:26:32.952 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:26:32.952 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:26:32.952 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:26:32.952 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:26:32.952 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:26:32.952 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:32.952 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:32.952 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:26:32.952 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:26:32.952 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:26:32.952 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:26:32.953 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:26:32.953 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:26:32.957 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:26:32.957 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:26:32.957 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:26:32.957 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:32.957 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:26:32.957 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:26:32.957 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:26:32.957 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:26:32.957 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:26:32.957 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:26:32.958 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:26:32.958 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:32.958 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:26:32.958 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:32.958 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:26:32.958 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:32.958 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:26:32.958 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:26:32.958 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:26:32.958 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:26:32.958 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:26:32.959 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:26:32.959 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:26:32.959 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:26:32.959 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:32.959 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:32.959 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:32.959 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:26:32.959 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:26:32.959 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:26:32.959 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:26:32.960 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:26:32.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:32.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:32.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:32.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:26:33.113 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:26:33.114 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:26:33.114 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:33.114 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:33.114 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:33.114 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:26:33.115 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:33.115 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:33.115 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:33.115 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:26:33.115 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:33.115 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:33.116 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:33.118 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:33.118 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:33.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:33.166 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 28 2024-09-11T06:26:33.166 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:33.166 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:33.166 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:33.166 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=115, tag_tid=20, entry_tid=15687], [object_number=114, tag_tid=20, entry_tid=15686], [object_number=113, tag_tid=20, entry_tid=15685], [object_number=112, tag_tid=20, entry_tid=15684]]], state=connected] 2024-09-11T06:26:33.170 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:33.171 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:33.172 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:26:33.172 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:33.172 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:33.172 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:33.173 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:26:33.173 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:33.173 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:33.173 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:33.173 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:26:33.173 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:33.173 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:33.173 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:33.174 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:33.174 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:33.174 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:33.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 28 2024-09-11T06:26:33.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:33.221 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:33.221 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:33.221 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=115, tag_tid=20, entry_tid=15687], [object_number=114, tag_tid=20, entry_tid=15686], [object_number=113, tag_tid=20, entry_tid=15685], [object_number=112, tag_tid=20, entry_tid=15684]]], state=connected] 2024-09-11T06:26:33.223 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:33.225 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=115, tag_tid=20, entry_tid=15687' 2024-09-11T06:26:33.225 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=115, tag_tid=20, entry_tid=15687' 2024-09-11T06:26:33.225 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=115, tag_tid=20, entry_tid=15687' '!=' 'object_number=115, tag_tid=20, entry_tid=15595' 2024-09-11T06:26:33.226 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:26:33.226 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=115, tag_tid=20, entry_tid=15687' = 'object_number=115, tag_tid=20, entry_tid=15595' 2024-09-11T06:26:33.226 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=115, tag_tid=20, entry_tid=15687' 2024-09-11T06:26:33.226 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:33.226 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:33.226 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:33.226 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:26:33.227 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=115, tag_tid=20, entry_tid=15687' 2024-09-11T06:26:33.227 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:33.227 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:33.228 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:26:33.229 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:33.229 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:33.229 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:33.230 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:26:33.230 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=115, tag_tid=20, entry_tid=15687' 2024-09-11T06:26:33.231 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:33.231 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:33.232 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=15687 2024-09-11T06:26:33.232 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 15687 2024-09-11T06:26:33.232 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:26:33.232 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:26:33.232 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:26:33.434 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:26:33.434 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:33.434 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:26:33.434 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:26:33.434 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:26:33.434 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:26:33.434 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:26:33.434 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:26:33.434 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:26:33.434 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:33.434 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:33.434 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:26:33.435 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:26:33.435 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:26:33.435 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:26:33.435 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:26:33.435 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:26:33.439 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:26:33.439 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:26:33.439 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:26:33.439 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:33.439 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:26:33.439 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:26:33.439 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:26:33.439 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:26:33.439 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:26:33.440 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:26:33.440 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:26:33.440 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:33.440 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:26:33.440 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:33.440 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:26:33.440 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:33.440 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:26:33.440 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:26:33.440 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:26:33.440 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:26:33.440 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:26:33.440 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:26:33.441 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:26:33.441 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:26:33.441 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:33.441 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:33.441 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:33.441 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:26:33.441 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:26:33.441 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:26:33.441 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:26:33.441 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:26:33.466 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:33.466 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:33.466 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:33.466 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:26:33.592 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:26:33.593 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:26:33.593 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:33.593 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:33.594 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:33.594 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:26:33.594 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:33.594 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:33.594 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:33.594 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:26:33.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:33.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:33.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:33.597 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:33.598 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:33.598 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:33.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 28 2024-09-11T06:26:33.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:33.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:33.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:33.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=112, tag_tid=20, entry_tid=15776], [object_number=115, tag_tid=20, entry_tid=15775], [object_number=114, tag_tid=20, entry_tid=15774], [object_number=113, tag_tid=20, entry_tid=15773]]], state=connected] 2024-09-11T06:26:33.660 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:33.661 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:33.662 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:26:33.663 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:33.663 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:33.663 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:33.663 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:26:33.663 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:33.663 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:33.663 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:33.663 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:26:33.663 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:33.663 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:33.663 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:33.664 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:33.664 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:33.664 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:33.711 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 28 2024-09-11T06:26:33.712 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:33.712 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:33.712 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:33.712 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=112, tag_tid=20, entry_tid=15776], [object_number=115, tag_tid=20, entry_tid=15775], [object_number=114, tag_tid=20, entry_tid=15774], [object_number=113, tag_tid=20, entry_tid=15773]]], state=connected] 2024-09-11T06:26:33.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:33.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=112, tag_tid=20, entry_tid=15776' 2024-09-11T06:26:33.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=112, tag_tid=20, entry_tid=15776' 2024-09-11T06:26:33.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=112, tag_tid=20, entry_tid=15776' '!=' 'object_number=115, tag_tid=20, entry_tid=15687' 2024-09-11T06:26:33.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:26:33.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=112, tag_tid=20, entry_tid=15776' = 'object_number=115, tag_tid=20, entry_tid=15687' 2024-09-11T06:26:33.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=112, tag_tid=20, entry_tid=15776' 2024-09-11T06:26:33.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:33.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:33.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:33.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:26:33.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=112, tag_tid=20, entry_tid=15776' 2024-09-11T06:26:33.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:33.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:33.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:26:33.721 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:33.721 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:33.721 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:33.722 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:26:33.722 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=112, tag_tid=20, entry_tid=15776' 2024-09-11T06:26:33.722 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:33.723 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:33.724 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=15776 2024-09-11T06:26:33.724 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 15776 2024-09-11T06:26:33.724 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:26:33.724 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:26:33.724 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:26:33.925 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:26:33.926 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:33.926 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:26:33.926 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:26:33.926 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:26:33.926 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:26:33.926 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:26:33.926 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:26:33.926 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:26:33.926 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:33.926 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:33.926 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:26:33.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:26:33.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:26:33.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:26:33.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:26:33.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:26:33.930 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:26:33.930 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:26:33.930 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:26:33.930 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:33.930 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:26:33.930 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:26:33.930 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:26:33.931 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:26:33.931 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:26:33.931 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:26:33.931 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:26:33.931 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:33.931 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:26:33.931 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:33.931 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:26:33.931 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:33.931 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:26:33.931 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:26:33.932 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:26:33.932 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:26:33.932 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:26:33.932 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:26:33.932 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:26:33.932 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:26:33.932 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:33.932 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:33.932 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:33.932 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:26:33.932 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:26:33.932 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:26:33.932 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:26:33.932 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:26:33.955 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:33.956 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:33.956 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:33.956 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:26:34.096 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:26:34.097 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:26:34.098 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:34.098 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:34.098 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:34.098 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:26:34.098 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:34.098 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:34.098 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:34.098 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:26:34.099 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:34.099 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:34.099 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:34.101 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:34.101 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:34.102 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:34.154 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 28 2024-09-11T06:26:34.154 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:34.154 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:34.154 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:34.155 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=112, tag_tid=20, entry_tid=15876], [object_number=115, tag_tid=20, entry_tid=15875], [object_number=114, tag_tid=20, entry_tid=15874], [object_number=113, tag_tid=20, entry_tid=15873]]], state=connected] 2024-09-11T06:26:34.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:34.159 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:34.160 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:26:34.160 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:34.160 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:34.161 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:34.161 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:26:34.161 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:34.161 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:34.161 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:34.161 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:26:34.161 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:34.162 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:34.162 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:34.162 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:34.162 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:34.162 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:34.209 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 28 2024-09-11T06:26:34.209 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:34.209 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:34.210 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:34.210 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=112, tag_tid=20, entry_tid=15876], [object_number=115, tag_tid=20, entry_tid=15875], [object_number=114, tag_tid=20, entry_tid=15874], [object_number=113, tag_tid=20, entry_tid=15873]]], state=connected] 2024-09-11T06:26:34.213 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:34.214 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=112, tag_tid=20, entry_tid=15876' 2024-09-11T06:26:34.214 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=112, tag_tid=20, entry_tid=15876' 2024-09-11T06:26:34.215 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=112, tag_tid=20, entry_tid=15876' '!=' 'object_number=112, tag_tid=20, entry_tid=15776' 2024-09-11T06:26:34.215 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:26:34.215 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=112, tag_tid=20, entry_tid=15876' = 'object_number=112, tag_tid=20, entry_tid=15776' 2024-09-11T06:26:34.215 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=112, tag_tid=20, entry_tid=15876' 2024-09-11T06:26:34.215 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:34.215 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:34.215 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:34.216 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:26:34.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=112, tag_tid=20, entry_tid=15876' 2024-09-11T06:26:34.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:34.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:34.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:26:34.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:34.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:34.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:34.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:26:34.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=112, tag_tid=20, entry_tid=15876' 2024-09-11T06:26:34.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:34.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:34.221 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=15876 2024-09-11T06:26:34.221 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 15876 2024-09-11T06:26:34.221 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:26:34.221 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:26:34.222 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:26:34.423 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:26:34.423 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:34.424 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:26:34.424 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:26:34.424 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:26:34.424 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:26:34.424 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:26:34.424 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:26:34.424 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:26:34.424 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:34.424 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:34.424 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:26:34.424 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:26:34.424 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:26:34.424 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:26:34.425 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:26:34.425 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:26:34.428 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:26:34.428 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:26:34.428 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:26:34.428 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:34.428 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:26:34.429 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:26:34.429 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:26:34.429 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:26:34.429 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:26:34.429 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:26:34.429 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:26:34.429 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:34.429 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:26:34.429 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:34.429 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:26:34.429 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:34.429 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:26:34.429 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:26:34.430 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:26:34.430 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:26:34.430 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:26:34.430 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:26:34.430 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:26:34.430 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:26:34.430 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:34.430 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:34.430 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:34.430 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:26:34.430 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:26:34.430 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:26:34.430 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:26:34.430 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:26:34.454 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:34.455 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:34.455 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:34.455 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:26:34.585 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:26:34.586 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:26:34.586 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:34.586 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:34.586 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:34.586 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:26:34.586 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:34.586 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:34.587 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:34.587 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:26:34.587 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:34.587 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:34.587 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:34.589 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:34.589 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:34.590 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:34.637 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 28 2024-09-11T06:26:34.637 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:34.637 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:34.637 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:34.637 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=113, tag_tid=20, entry_tid=15969], [object_number=112, tag_tid=20, entry_tid=15968], [object_number=115, tag_tid=20, entry_tid=15967], [object_number=114, tag_tid=20, entry_tid=15966]]], state=connected] 2024-09-11T06:26:34.640 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:34.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:34.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:26:34.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:34.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:34.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:34.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:26:34.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:34.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:34.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:34.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:26:34.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:34.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:34.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:34.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:34.645 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:34.645 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:34.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 28 2024-09-11T06:26:34.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:34.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:34.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:34.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=113, tag_tid=20, entry_tid=15969], [object_number=112, tag_tid=20, entry_tid=15968], [object_number=115, tag_tid=20, entry_tid=15967], [object_number=114, tag_tid=20, entry_tid=15966]]], state=connected] 2024-09-11T06:26:34.695 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:34.697 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=113, tag_tid=20, entry_tid=15969' 2024-09-11T06:26:34.697 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=113, tag_tid=20, entry_tid=15969' 2024-09-11T06:26:34.697 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=113, tag_tid=20, entry_tid=15969' '!=' 'object_number=112, tag_tid=20, entry_tid=15876' 2024-09-11T06:26:34.697 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:26:34.697 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=113, tag_tid=20, entry_tid=15969' = 'object_number=112, tag_tid=20, entry_tid=15876' 2024-09-11T06:26:34.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=113, tag_tid=20, entry_tid=15969' 2024-09-11T06:26:34.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:34.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:34.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:34.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:26:34.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=20, entry_tid=15969' 2024-09-11T06:26:34.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:34.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:34.700 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:26:34.701 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:34.701 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:34.701 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:34.701 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:26:34.702 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=20, entry_tid=15969' 2024-09-11T06:26:34.702 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:34.702 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:34.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=15969 2024-09-11T06:26:34.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 15969 2024-09-11T06:26:34.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:26:34.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:26:34.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:26:34.905 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:26:34.905 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:34.906 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:26:34.906 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:26:34.906 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:26:34.906 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:26:34.906 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:26:34.906 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:26:34.906 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:26:34.906 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:34.906 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:34.906 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:26:34.906 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:26:34.906 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:26:34.906 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:26:34.907 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:26:34.907 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:26:34.911 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:26:34.911 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:26:34.911 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:26:34.911 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:34.912 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:26:34.912 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:26:34.912 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:26:34.912 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:26:34.912 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:26:34.912 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:26:34.912 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:26:34.912 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:34.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:26:34.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:34.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:26:34.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:34.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:26:34.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:26:34.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:26:34.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:26:34.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:26:34.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:26:34.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:26:34.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:26:34.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:34.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:34.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:34.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:26:34.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:26:34.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:26:34.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:26:34.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:26:34.936 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:34.937 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:34.937 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:34.937 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:26:35.069 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:26:35.069 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:26:35.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:35.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:35.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:35.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:26:35.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:35.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:35.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:35.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:26:35.071 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:35.071 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:35.071 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:35.073 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:35.073 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:35.073 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:35.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 28 2024-09-11T06:26:35.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:35.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:35.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:35.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=113, tag_tid=20, entry_tid=16061], [object_number=112, tag_tid=20, entry_tid=16060], [object_number=115, tag_tid=20, entry_tid=16059], [object_number=114, tag_tid=20, entry_tid=16058]]], state=connected] 2024-09-11T06:26:35.130 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:35.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:35.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:26:35.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:35.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:35.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:35.133 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:26:35.133 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:35.133 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:35.133 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:35.133 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:26:35.133 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:35.133 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:35.133 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:35.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:35.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:35.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:35.389 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 28 2024-09-11T06:26:35.390 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:35.390 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:35.390 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:35.390 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=113, tag_tid=20, entry_tid=16061], [object_number=112, tag_tid=20, entry_tid=16060], [object_number=115, tag_tid=20, entry_tid=16059], [object_number=114, tag_tid=20, entry_tid=16058]]], state=connected] 2024-09-11T06:26:35.396 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:35.398 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=113, tag_tid=20, entry_tid=16061' 2024-09-11T06:26:35.398 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=113, tag_tid=20, entry_tid=16061' 2024-09-11T06:26:35.398 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=113, tag_tid=20, entry_tid=16061' '!=' 'object_number=113, tag_tid=20, entry_tid=15969' 2024-09-11T06:26:35.398 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:26:35.398 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=113, tag_tid=20, entry_tid=16061' = 'object_number=113, tag_tid=20, entry_tid=15969' 2024-09-11T06:26:35.398 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=113, tag_tid=20, entry_tid=16061' 2024-09-11T06:26:35.399 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:35.399 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:35.399 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:35.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:26:35.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=20, entry_tid=16061' 2024-09-11T06:26:35.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:35.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:35.404 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:26:35.405 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:35.405 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:35.405 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:35.406 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:26:35.407 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=20, entry_tid=16061' 2024-09-11T06:26:35.407 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:35.407 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:35.409 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=16061 2024-09-11T06:26:35.409 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 16061 2024-09-11T06:26:35.409 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:26:35.409 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:26:35.409 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:26:35.611 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:26:35.611 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:35.612 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:26:35.612 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:26:35.612 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:26:35.612 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:26:35.612 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:26:35.612 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:26:35.612 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:26:35.612 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:35.612 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:35.612 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:26:35.612 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:26:35.613 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:26:35.613 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:26:35.613 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:26:35.614 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:26:35.617 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:26:35.617 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:26:35.618 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:26:35.618 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:35.618 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:26:35.618 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:26:35.618 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:26:35.618 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:26:35.618 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:26:35.618 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:26:35.618 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:26:35.618 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:35.618 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:26:35.618 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:35.618 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:26:35.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:35.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:26:35.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:26:35.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:26:35.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:26:35.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:26:35.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:26:35.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:26:35.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:26:35.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:35.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:35.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:35.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:26:35.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:26:35.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:26:35.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:26:35.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:26:35.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:35.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:35.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:35.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:26:35.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:26:35.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:26:35.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:35.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:35.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:35.768 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:26:35.768 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:35.768 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:35.768 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:35.768 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:26:35.768 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:35.769 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:35.769 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:35.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:35.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:35.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:35.818 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 28 2024-09-11T06:26:35.818 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:35.818 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:35.818 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:35.818 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=113, tag_tid=20, entry_tid=16201], [object_number=112, tag_tid=20, entry_tid=16200], [object_number=115, tag_tid=20, entry_tid=16199], [object_number=114, tag_tid=20, entry_tid=16198]]], state=connected] 2024-09-11T06:26:35.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:35.822 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:35.824 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:26:35.824 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:35.824 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:35.824 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:35.824 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:26:35.824 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:35.824 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:35.824 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:35.824 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:26:35.824 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:35.825 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:35.825 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:35.825 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:35.825 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:35.825 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:35.872 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 28 2024-09-11T06:26:35.872 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:35.872 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:35.872 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:35.872 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=113, tag_tid=20, entry_tid=16201], [object_number=112, tag_tid=20, entry_tid=16200], [object_number=115, tag_tid=20, entry_tid=16199], [object_number=114, tag_tid=20, entry_tid=16198]]], state=connected] 2024-09-11T06:26:35.876 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:35.877 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=113, tag_tid=20, entry_tid=16201' 2024-09-11T06:26:35.877 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=113, tag_tid=20, entry_tid=16201' 2024-09-11T06:26:35.877 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=113, tag_tid=20, entry_tid=16201' '!=' 'object_number=113, tag_tid=20, entry_tid=16061' 2024-09-11T06:26:35.877 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:26:35.878 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=113, tag_tid=20, entry_tid=16201' = 'object_number=113, tag_tid=20, entry_tid=16061' 2024-09-11T06:26:35.878 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=113, tag_tid=20, entry_tid=16201' 2024-09-11T06:26:35.878 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:35.878 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:35.878 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:35.879 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:26:35.879 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=20, entry_tid=16201' 2024-09-11T06:26:35.880 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:35.880 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:35.880 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:26:35.881 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:35.881 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:35.881 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:35.882 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:26:35.883 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=20, entry_tid=16201' 2024-09-11T06:26:35.883 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:35.883 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:35.884 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=16201 2024-09-11T06:26:35.884 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 16201 2024-09-11T06:26:35.884 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:26:35.884 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:26:35.885 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:26:36.086 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:26:36.086 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:36.086 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:26:36.086 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:26:36.087 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:26:36.087 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:26:36.087 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:26:36.087 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:26:36.087 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:26:36.087 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:36.087 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:36.087 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:26:36.087 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:26:36.087 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:26:36.087 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:26:36.088 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:26:36.088 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:26:36.091 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:26:36.091 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:26:36.091 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:26:36.091 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:36.092 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:26:36.092 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:26:36.092 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:26:36.092 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:26:36.092 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:26:36.092 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:26:36.092 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:26:36.092 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:36.092 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:26:36.092 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:36.092 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:26:36.092 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:36.092 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:26:36.093 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:26:36.093 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:26:36.093 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:26:36.093 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:26:36.093 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:26:36.093 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:26:36.093 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:26:36.093 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:36.093 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:36.093 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:36.093 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:26:36.093 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:26:36.093 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:26:36.093 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:26:36.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:26:36.117 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:36.117 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:36.118 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:36.118 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:26:36.244 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:26:36.245 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:26:36.246 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:36.246 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:36.246 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:36.246 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:26:36.246 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:36.246 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:36.246 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:36.246 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:26:36.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:36.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:36.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:36.248 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:36.248 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:36.248 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:36.294 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 28 2024-09-11T06:26:36.294 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:36.294 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:36.294 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:36.294 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=114, tag_tid=20, entry_tid=16286], [object_number=113, tag_tid=20, entry_tid=16285], [object_number=112, tag_tid=20, entry_tid=16284], [object_number=115, tag_tid=20, entry_tid=16283]]], state=connected] 2024-09-11T06:26:36.297 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:36.300 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:36.301 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:26:36.301 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:36.301 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:36.301 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:36.301 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:26:36.301 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:36.301 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:36.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:36.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:26:36.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:36.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:36.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:36.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:36.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:36.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:36.348 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 28 2024-09-11T06:26:36.348 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:36.348 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:36.348 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:36.348 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=114, tag_tid=20, entry_tid=16286], [object_number=113, tag_tid=20, entry_tid=16285], [object_number=112, tag_tid=20, entry_tid=16284], [object_number=115, tag_tid=20, entry_tid=16283]]], state=connected] 2024-09-11T06:26:36.351 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:36.353 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=114, tag_tid=20, entry_tid=16286' 2024-09-11T06:26:36.354 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=114, tag_tid=20, entry_tid=16286' 2024-09-11T06:26:36.354 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=114, tag_tid=20, entry_tid=16286' '!=' 'object_number=113, tag_tid=20, entry_tid=16201' 2024-09-11T06:26:36.354 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:26:36.354 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=114, tag_tid=20, entry_tid=16286' = 'object_number=113, tag_tid=20, entry_tid=16201' 2024-09-11T06:26:36.354 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=114, tag_tid=20, entry_tid=16286' 2024-09-11T06:26:36.354 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:36.354 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:36.354 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:36.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:26:36.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=114, tag_tid=20, entry_tid=16286' 2024-09-11T06:26:36.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:36.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:36.356 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:26:36.357 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:36.357 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:36.357 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:36.358 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:26:36.358 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=114, tag_tid=20, entry_tid=16286' 2024-09-11T06:26:36.359 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:36.359 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:36.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=16286 2024-09-11T06:26:36.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 16286 2024-09-11T06:26:36.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:26:36.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:26:36.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:26:36.562 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:26:36.562 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:36.562 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:26:36.562 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:26:36.562 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:26:36.562 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:26:36.562 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:26:36.562 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:26:36.563 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:26:36.563 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:36.563 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:36.563 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:26:36.563 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:26:36.563 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:26:36.563 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:26:36.563 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:26:36.564 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:26:36.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:26:36.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:26:36.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:26:36.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:36.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:26:36.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:26:36.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:26:36.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:26:36.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:26:36.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:26:36.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:26:36.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:36.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:26:36.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:36.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:26:36.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:36.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:26:36.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:26:36.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:26:36.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:26:36.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:26:36.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:26:36.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:26:36.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:26:36.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:36.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:36.569 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:36.569 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:26:36.569 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:26:36.569 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:26:36.569 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:26:36.569 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:26:36.591 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:36.592 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:36.592 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:36.592 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:26:36.726 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:26:36.727 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:26:36.727 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:36.727 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:36.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:36.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:26:36.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:36.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:36.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:36.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:26:36.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:36.729 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:36.729 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:36.730 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:36.730 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:36.730 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:36.783 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 28 2024-09-11T06:26:36.783 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:36.783 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:36.783 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:36.783 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=112, tag_tid=20, entry_tid=16372], [object_number=115, tag_tid=20, entry_tid=16371], [object_number=114, tag_tid=20, entry_tid=16370], [object_number=113, tag_tid=20, entry_tid=16369]]], state=connected] 2024-09-11T06:26:36.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:36.788 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:36.789 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:26:36.789 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:36.789 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:36.789 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:36.790 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:26:36.790 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:36.790 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:36.790 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:36.790 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:26:36.790 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:36.790 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:36.790 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:36.791 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:36.791 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:36.791 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:36.838 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 28 2024-09-11T06:26:36.838 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:36.838 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:36.839 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:36.839 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=112, tag_tid=20, entry_tid=16372], [object_number=115, tag_tid=20, entry_tid=16371], [object_number=114, tag_tid=20, entry_tid=16370], [object_number=113, tag_tid=20, entry_tid=16369]]], state=connected] 2024-09-11T06:26:36.841 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:36.843 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=112, tag_tid=20, entry_tid=16372' 2024-09-11T06:26:36.844 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=112, tag_tid=20, entry_tid=16372' 2024-09-11T06:26:36.844 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=112, tag_tid=20, entry_tid=16372' '!=' 'object_number=114, tag_tid=20, entry_tid=16286' 2024-09-11T06:26:36.844 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:26:36.844 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=112, tag_tid=20, entry_tid=16372' = 'object_number=114, tag_tid=20, entry_tid=16286' 2024-09-11T06:26:36.844 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=112, tag_tid=20, entry_tid=16372' 2024-09-11T06:26:36.844 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:36.844 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:36.844 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:36.845 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:26:36.845 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=112, tag_tid=20, entry_tid=16372' 2024-09-11T06:26:36.845 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:36.845 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:36.846 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:26:36.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:36.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:36.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:36.848 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:26:36.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=112, tag_tid=20, entry_tid=16372' 2024-09-11T06:26:36.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:36.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:36.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=16372 2024-09-11T06:26:36.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 16372 2024-09-11T06:26:36.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:26:36.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:26:36.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:26:37.053 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:26:37.053 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:37.053 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:26:37.054 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:26:37.054 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:26:37.054 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:26:37.054 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:26:37.054 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:26:37.054 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:26:37.054 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:37.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:37.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:26:37.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:26:37.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:26:37.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:26:37.058 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:26:37.059 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:26:37.059 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:26:37.059 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:26:37.059 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:26:37.059 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:37.060 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:26:37.060 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:26:37.060 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:26:37.060 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:26:37.060 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:26:37.060 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:26:37.060 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:26:37.060 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:37.060 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:26:37.060 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:37.060 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:26:37.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:37.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:26:37.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:26:37.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:26:37.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:26:37.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:26:37.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:26:37.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:26:37.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:26:37.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:37.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:37.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:37.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:26:37.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:26:37.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:26:37.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:26:37.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:26:37.085 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:37.085 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:37.085 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:37.085 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:26:37.214 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:26:37.215 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:26:37.215 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:37.216 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:37.216 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:37.216 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:26:37.216 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:37.216 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:37.216 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:37.216 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:26:37.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:37.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:37.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:37.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:37.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:37.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:37.272 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 28 2024-09-11T06:26:37.272 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:37.272 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:37.272 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:37.273 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=115, tag_tid=20, entry_tid=16467], [object_number=114, tag_tid=20, entry_tid=16466], [object_number=113, tag_tid=20, entry_tid=16465], [object_number=112, tag_tid=20, entry_tid=16464]]], state=connected] 2024-09-11T06:26:37.277 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:37.278 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:37.278 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:26:37.278 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:37.278 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:37.279 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:37.279 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:26:37.279 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:37.279 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:37.279 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:37.279 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:26:37.279 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:37.279 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:37.279 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:37.280 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:37.280 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:37.280 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:37.327 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 28 2024-09-11T06:26:37.327 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:37.327 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:37.327 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:37.327 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=115, tag_tid=20, entry_tid=16467], [object_number=114, tag_tid=20, entry_tid=16466], [object_number=113, tag_tid=20, entry_tid=16465], [object_number=112, tag_tid=20, entry_tid=16464]]], state=connected] 2024-09-11T06:26:37.330 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:37.332 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=115, tag_tid=20, entry_tid=16467' 2024-09-11T06:26:37.332 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=115, tag_tid=20, entry_tid=16467' 2024-09-11T06:26:37.332 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=115, tag_tid=20, entry_tid=16467' '!=' 'object_number=112, tag_tid=20, entry_tid=16372' 2024-09-11T06:26:37.332 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:26:37.332 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=115, tag_tid=20, entry_tid=16467' = 'object_number=112, tag_tid=20, entry_tid=16372' 2024-09-11T06:26:37.332 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=115, tag_tid=20, entry_tid=16467' 2024-09-11T06:26:37.333 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:37.333 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:37.333 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:37.334 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:26:37.334 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=115, tag_tid=20, entry_tid=16467' 2024-09-11T06:26:37.334 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:37.334 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:37.335 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:26:37.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:37.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:37.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:37.337 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:26:37.338 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=115, tag_tid=20, entry_tid=16467' 2024-09-11T06:26:37.338 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:37.338 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:37.339 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=16467 2024-09-11T06:26:37.339 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 16467 2024-09-11T06:26:37.339 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:26:37.339 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:26:37.339 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:26:37.541 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:26:37.541 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:37.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:26:37.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:26:37.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:26:37.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:26:37.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:26:37.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:26:37.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:26:37.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:37.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:37.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:26:37.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:26:37.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:26:37.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:26:37.543 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:26:37.543 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:26:37.546 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:26:37.546 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:26:37.546 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:26:37.546 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:37.547 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:26:37.547 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:26:37.547 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:26:37.547 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:26:37.547 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:26:37.547 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:26:37.547 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:26:37.547 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:37.547 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:26:37.547 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:37.547 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:26:37.547 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:37.547 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:26:37.548 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:26:37.548 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:26:37.548 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:26:37.548 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:26:37.548 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:26:37.548 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:26:37.548 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:26:37.548 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:37.548 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:37.548 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:37.548 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:26:37.548 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:26:37.548 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:26:37.548 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:26:37.549 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:26:37.572 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:37.572 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:37.572 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:37.572 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:26:37.697 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:26:37.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:26:37.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:37.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:37.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:37.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:26:37.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:37.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:37.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:37.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:26:37.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:37.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:37.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:37.700 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:37.701 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:37.701 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:37.754 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 28 2024-09-11T06:26:37.754 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:37.754 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:37.754 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:37.754 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=115, tag_tid=20, entry_tid=16555], [object_number=114, tag_tid=20, entry_tid=16554], [object_number=113, tag_tid=20, entry_tid=16553], [object_number=112, tag_tid=20, entry_tid=16552]]], state=connected] 2024-09-11T06:26:37.757 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:37.758 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:37.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:26:37.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:37.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:37.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:37.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:26:37.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:37.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:37.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:37.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:26:37.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:37.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:37.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:37.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:37.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:37.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:37.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 28 2024-09-11T06:26:37.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:37.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:37.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:37.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=115, tag_tid=20, entry_tid=16555], [object_number=114, tag_tid=20, entry_tid=16554], [object_number=113, tag_tid=20, entry_tid=16553], [object_number=112, tag_tid=20, entry_tid=16552]]], state=connected] 2024-09-11T06:26:37.813 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:37.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=115, tag_tid=20, entry_tid=16555' 2024-09-11T06:26:37.815 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=115, tag_tid=20, entry_tid=16555' 2024-09-11T06:26:37.815 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=115, tag_tid=20, entry_tid=16555' '!=' 'object_number=115, tag_tid=20, entry_tid=16467' 2024-09-11T06:26:37.815 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:26:37.815 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=115, tag_tid=20, entry_tid=16555' = 'object_number=115, tag_tid=20, entry_tid=16467' 2024-09-11T06:26:37.815 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=115, tag_tid=20, entry_tid=16555' 2024-09-11T06:26:37.815 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:37.815 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:37.816 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:37.816 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:26:37.817 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=115, tag_tid=20, entry_tid=16555' 2024-09-11T06:26:37.817 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:37.817 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:37.818 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:26:37.818 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:37.818 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:37.818 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:37.819 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:26:37.820 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=115, tag_tid=20, entry_tid=16555' 2024-09-11T06:26:37.820 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:37.820 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:37.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=16555 2024-09-11T06:26:37.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 16555 2024-09-11T06:26:37.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:26:37.822 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:26:37.822 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:26:38.024 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:26:38.024 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:38.024 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:26:38.024 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:26:38.024 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:26:38.024 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:26:38.024 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:26:38.024 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:26:38.024 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:26:38.024 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:38.024 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:38.025 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:26:38.025 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:26:38.025 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:26:38.025 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:26:38.025 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:26:38.025 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:26:38.028 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:26:38.028 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:26:38.028 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:26:38.028 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:38.028 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:26:38.028 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:26:38.029 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:26:38.029 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:26:38.029 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:26:38.029 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:26:38.029 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:26:38.029 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:38.029 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:26:38.029 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:38.029 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:26:38.029 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:38.029 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:26:38.029 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:26:38.029 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:26:38.030 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:26:38.030 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:26:38.030 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:26:38.030 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:26:38.030 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:26:38.030 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:38.030 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:38.030 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:38.030 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:26:38.030 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:26:38.030 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:26:38.030 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:26:38.030 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:26:38.054 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:38.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:38.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:38.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:26:38.184 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:26:38.185 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:26:38.185 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:38.185 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:38.185 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:38.185 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:26:38.186 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:38.186 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:38.186 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:38.186 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:26:38.186 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:38.187 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:38.187 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:38.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:38.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:38.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:38.245 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 28 2024-09-11T06:26:38.245 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:38.245 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:38.245 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:38.245 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=112, tag_tid=20, entry_tid=16648], [object_number=115, tag_tid=20, entry_tid=16647], [object_number=114, tag_tid=20, entry_tid=16646], [object_number=113, tag_tid=20, entry_tid=16645]]], state=connected] 2024-09-11T06:26:38.246 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:38.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:38.248 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:26:38.248 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:38.248 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:38.248 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:38.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:26:38.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:38.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:38.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:38.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:26:38.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:38.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:38.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:38.250 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:38.250 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:38.250 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:38.297 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 28 2024-09-11T06:26:38.297 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:38.297 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:38.297 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:38.297 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=112, tag_tid=20, entry_tid=16648], [object_number=115, tag_tid=20, entry_tid=16647], [object_number=114, tag_tid=20, entry_tid=16646], [object_number=113, tag_tid=20, entry_tid=16645]]], state=connected] 2024-09-11T06:26:38.300 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:38.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=112, tag_tid=20, entry_tid=16648' 2024-09-11T06:26:38.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=112, tag_tid=20, entry_tid=16648' 2024-09-11T06:26:38.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=112, tag_tid=20, entry_tid=16648' '!=' 'object_number=115, tag_tid=20, entry_tid=16555' 2024-09-11T06:26:38.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:26:38.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=112, tag_tid=20, entry_tid=16648' = 'object_number=115, tag_tid=20, entry_tid=16555' 2024-09-11T06:26:38.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=112, tag_tid=20, entry_tid=16648' 2024-09-11T06:26:38.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:38.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:38.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:38.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:26:38.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=112, tag_tid=20, entry_tid=16648' 2024-09-11T06:26:38.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:38.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:38.305 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:26:38.306 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:38.306 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:38.306 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:38.307 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:26:38.307 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=112, tag_tid=20, entry_tid=16648' 2024-09-11T06:26:38.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:38.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:38.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=16648 2024-09-11T06:26:38.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 16648 2024-09-11T06:26:38.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:26:38.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:26:38.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:26:38.511 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:26:38.511 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:38.511 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:26:38.511 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:26:38.511 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:26:38.511 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:26:38.511 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:26:38.511 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:26:38.511 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:26:38.511 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:38.511 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:38.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:26:38.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:26:38.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:26:38.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:26:38.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:26:38.513 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:26:38.517 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:26:38.517 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:26:38.517 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:26:38.517 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:38.517 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:26:38.518 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:26:38.518 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:26:38.518 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:26:38.518 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:26:38.518 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:26:38.518 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:26:38.518 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:38.518 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:26:38.518 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:38.518 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:26:38.518 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:38.518 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:26:38.518 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:26:38.518 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:26:38.519 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:26:38.519 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:26:38.519 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:26:38.519 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:26:38.519 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:26:38.519 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:38.519 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:38.519 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:38.519 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:26:38.519 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:26:38.519 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:26:38.519 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:26:38.519 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:26:38.544 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:38.544 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:38.544 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:38.544 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:26:38.679 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:26:38.681 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:26:38.681 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:38.681 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:38.681 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:38.681 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:26:38.681 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:38.681 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:38.682 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:38.682 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:26:38.682 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:38.682 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:38.682 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:38.683 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:38.683 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:38.683 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:38.737 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 28 2024-09-11T06:26:38.737 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:38.737 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:38.737 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:38.737 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=114, tag_tid=20, entry_tid=16742], [object_number=113, tag_tid=20, entry_tid=16741], [object_number=112, tag_tid=20, entry_tid=16740], [object_number=115, tag_tid=20, entry_tid=16739]]], state=connected] 2024-09-11T06:26:38.741 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:38.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:38.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:26:38.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:38.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:38.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:38.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:26:38.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:38.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:38.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:38.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:26:38.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:38.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:38.745 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:38.745 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:38.745 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:38.746 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:38.793 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 28 2024-09-11T06:26:38.794 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:38.794 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:38.794 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:38.794 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=114, tag_tid=20, entry_tid=16742], [object_number=113, tag_tid=20, entry_tid=16741], [object_number=112, tag_tid=20, entry_tid=16740], [object_number=115, tag_tid=20, entry_tid=16739]]], state=connected] 2024-09-11T06:26:38.797 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:38.799 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=114, tag_tid=20, entry_tid=16742' 2024-09-11T06:26:38.799 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=114, tag_tid=20, entry_tid=16742' 2024-09-11T06:26:38.799 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=114, tag_tid=20, entry_tid=16742' '!=' 'object_number=112, tag_tid=20, entry_tid=16648' 2024-09-11T06:26:38.799 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:26:38.799 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=114, tag_tid=20, entry_tid=16742' = 'object_number=112, tag_tid=20, entry_tid=16648' 2024-09-11T06:26:38.799 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=114, tag_tid=20, entry_tid=16742' 2024-09-11T06:26:38.800 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:38.800 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:38.800 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:38.801 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:26:38.801 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=114, tag_tid=20, entry_tid=16742' 2024-09-11T06:26:38.801 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:38.801 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:38.802 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:26:38.802 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:38.803 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:38.803 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:38.803 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:26:38.804 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=114, tag_tid=20, entry_tid=16742' 2024-09-11T06:26:38.804 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:38.804 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:38.806 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=16742 2024-09-11T06:26:38.806 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 16742 2024-09-11T06:26:38.806 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:26:38.806 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:26:38.806 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:26:39.007 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:26:39.007 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:39.007 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:26:39.007 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:26:39.008 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:26:39.008 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:26:39.008 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:26:39.008 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:26:39.008 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:26:39.008 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:39.008 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:39.008 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:26:39.008 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:26:39.008 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:26:39.008 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:26:39.009 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:26:39.009 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:26:39.013 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:26:39.013 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:26:39.013 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:26:39.013 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:39.013 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:26:39.013 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:26:39.013 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:26:39.013 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:26:39.014 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:26:39.014 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:26:39.014 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:26:39.014 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:39.014 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:26:39.014 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:39.014 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:26:39.014 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:39.014 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:26:39.014 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:26:39.014 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:26:39.014 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:26:39.014 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:26:39.015 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:26:39.015 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:26:39.015 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:26:39.015 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:39.015 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:39.015 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:39.015 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:26:39.015 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:26:39.015 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:26:39.015 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:26:39.015 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:26:39.038 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:39.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:39.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:39.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:26:39.167 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:26:39.168 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:26:39.168 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:39.168 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:39.168 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:39.168 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:26:39.168 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:39.168 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:39.168 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:39.168 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:26:39.169 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:39.169 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:39.169 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:39.171 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:39.171 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:39.171 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:39.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 28 2024-09-11T06:26:39.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:39.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:39.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:39.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=115, tag_tid=20, entry_tid=16827], [object_number=114, tag_tid=20, entry_tid=16826], [object_number=113, tag_tid=20, entry_tid=16825], [object_number=112, tag_tid=20, entry_tid=16824]]], state=connected] 2024-09-11T06:26:39.223 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:39.224 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:39.225 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:26:39.225 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:39.225 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:39.225 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:39.225 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:26:39.226 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:39.226 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:39.226 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:39.226 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:26:39.226 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:39.226 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:39.226 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:39.227 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:39.227 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:39.227 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:39.274 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 28 2024-09-11T06:26:39.274 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:39.274 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:39.274 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:39.274 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=115, tag_tid=20, entry_tid=16827], [object_number=114, tag_tid=20, entry_tid=16826], [object_number=113, tag_tid=20, entry_tid=16825], [object_number=112, tag_tid=20, entry_tid=16824]]], state=connected] 2024-09-11T06:26:39.284 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:39.286 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=115, tag_tid=20, entry_tid=16827' 2024-09-11T06:26:39.286 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=115, tag_tid=20, entry_tid=16827' 2024-09-11T06:26:39.286 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=115, tag_tid=20, entry_tid=16827' '!=' 'object_number=114, tag_tid=20, entry_tid=16742' 2024-09-11T06:26:39.286 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:26:39.286 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=115, tag_tid=20, entry_tid=16827' = 'object_number=114, tag_tid=20, entry_tid=16742' 2024-09-11T06:26:39.287 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=115, tag_tid=20, entry_tid=16827' 2024-09-11T06:26:39.287 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:39.287 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:39.287 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:39.288 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:26:39.288 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=115, tag_tid=20, entry_tid=16827' 2024-09-11T06:26:39.288 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:39.288 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:39.289 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:26:39.290 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:39.290 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:39.290 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:39.291 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:26:39.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=115, tag_tid=20, entry_tid=16827' 2024-09-11T06:26:39.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:39.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:39.293 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=16827 2024-09-11T06:26:39.293 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 16827 2024-09-11T06:26:39.293 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:26:39.293 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:26:39.293 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:26:39.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:26:39.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:39.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:26:39.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:26:39.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:26:39.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:26:39.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:26:39.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:26:39.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:26:39.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:39.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:39.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:26:39.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:26:39.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:26:39.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:26:39.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:26:39.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:26:39.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:26:39.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:26:39.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:26:39.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:39.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:26:39.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:26:39.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:26:39.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:26:39.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:26:39.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:26:39.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:26:39.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:39.501 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:26:39.501 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:39.501 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:26:39.501 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:39.501 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:26:39.501 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:26:39.501 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:26:39.501 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:26:39.501 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:26:39.501 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:26:39.501 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:26:39.501 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:26:39.501 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:39.502 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:39.502 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:39.502 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:26:39.502 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:26:39.502 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:26:39.502 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:26:39.502 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:26:39.527 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:39.527 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:39.527 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:39.527 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:26:39.658 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:26:39.659 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:26:39.659 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:39.659 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:39.659 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:39.659 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:26:39.659 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:39.659 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:39.659 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:39.659 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:26:39.660 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:39.660 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:39.660 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:39.661 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:39.662 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:39.662 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:39.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 28 2024-09-11T06:26:39.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:39.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:39.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:39.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=114, tag_tid=20, entry_tid=16910], [object_number=113, tag_tid=20, entry_tid=16909], [object_number=112, tag_tid=20, entry_tid=16908], [object_number=115, tag_tid=20, entry_tid=16907]]], state=connected] 2024-09-11T06:26:39.723 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:39.724 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:39.725 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:26:39.725 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:39.726 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:39.726 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:39.726 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:26:39.726 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:39.726 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:39.726 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:39.726 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:26:39.726 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:39.726 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:39.726 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:39.727 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:39.727 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:39.727 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:39.774 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 28 2024-09-11T06:26:39.775 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:39.775 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:39.775 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:39.775 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=114, tag_tid=20, entry_tid=16910], [object_number=113, tag_tid=20, entry_tid=16909], [object_number=112, tag_tid=20, entry_tid=16908], [object_number=115, tag_tid=20, entry_tid=16907]]], state=connected] 2024-09-11T06:26:39.778 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:39.779 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=114, tag_tid=20, entry_tid=16910' 2024-09-11T06:26:39.779 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=114, tag_tid=20, entry_tid=16910' 2024-09-11T06:26:39.779 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=114, tag_tid=20, entry_tid=16910' '!=' 'object_number=115, tag_tid=20, entry_tid=16827' 2024-09-11T06:26:39.780 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:26:39.780 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=114, tag_tid=20, entry_tid=16910' = 'object_number=115, tag_tid=20, entry_tid=16827' 2024-09-11T06:26:39.780 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=114, tag_tid=20, entry_tid=16910' 2024-09-11T06:26:39.780 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:39.780 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:39.780 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:39.781 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:26:39.781 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=114, tag_tid=20, entry_tid=16910' 2024-09-11T06:26:39.782 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:39.782 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:39.782 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:26:39.783 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:39.783 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:39.783 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:39.785 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:26:39.785 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=114, tag_tid=20, entry_tid=16910' 2024-09-11T06:26:39.785 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:39.785 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:39.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=16910 2024-09-11T06:26:39.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 16910 2024-09-11T06:26:39.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:26:39.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:26:39.787 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:26:39.988 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:26:39.989 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:39.989 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:26:39.989 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:26:39.989 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:26:39.989 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:26:39.989 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:26:39.989 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:26:39.989 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:26:39.989 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:39.989 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:39.989 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:26:39.990 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:26:39.990 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:26:39.990 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:26:39.990 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:26:39.990 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:26:39.992 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:26:39.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:26:39.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:26:39.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:39.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:26:39.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:26:39.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:26:39.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:26:39.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:26:39.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:26:39.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:26:39.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:39.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:26:39.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:39.994 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:26:39.994 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:39.995 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:26:39.995 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:26:39.995 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:26:39.996 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:26:39.996 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:26:39.996 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:26:39.996 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:26:39.996 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:26:39.996 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:39.996 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:39.996 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:39.996 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:26:39.996 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:26:39.996 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:26:39.996 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:26:39.997 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:26:40.019 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:40.020 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:40.020 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:40.020 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:26:40.154 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:26:40.155 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:26:40.155 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:40.156 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:40.156 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:40.156 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:26:40.156 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:40.156 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:40.156 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:40.156 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:26:40.157 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:40.157 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:40.157 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:40.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:40.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:40.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:40.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 28 2024-09-11T06:26:40.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:40.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:40.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:40.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=113, tag_tid=20, entry_tid=16993], [object_number=112, tag_tid=20, entry_tid=16992], [object_number=115, tag_tid=20, entry_tid=16991], [object_number=114, tag_tid=20, entry_tid=16990]]], state=connected] 2024-09-11T06:26:40.215 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:40.216 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:40.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:26:40.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:40.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:40.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:40.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:26:40.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:40.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:40.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:40.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:26:40.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:40.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:40.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:40.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:40.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:40.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:40.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 28 2024-09-11T06:26:40.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:40.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:40.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:40.269 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=113, tag_tid=20, entry_tid=16993], [object_number=112, tag_tid=20, entry_tid=16992], [object_number=115, tag_tid=20, entry_tid=16991], [object_number=114, tag_tid=20, entry_tid=16990]]], state=connected] 2024-09-11T06:26:40.272 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:40.274 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=113, tag_tid=20, entry_tid=16993' 2024-09-11T06:26:40.274 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=113, tag_tid=20, entry_tid=16993' 2024-09-11T06:26:40.274 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=113, tag_tid=20, entry_tid=16993' '!=' 'object_number=114, tag_tid=20, entry_tid=16910' 2024-09-11T06:26:40.274 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:26:40.274 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=113, tag_tid=20, entry_tid=16993' = 'object_number=114, tag_tid=20, entry_tid=16910' 2024-09-11T06:26:40.274 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=113, tag_tid=20, entry_tid=16993' 2024-09-11T06:26:40.274 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:40.275 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:40.275 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:40.275 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:26:40.276 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=20, entry_tid=16993' 2024-09-11T06:26:40.276 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:40.276 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:40.277 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:26:40.278 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:40.278 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:40.278 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:40.279 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:26:40.279 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=20, entry_tid=16993' 2024-09-11T06:26:40.279 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:40.279 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:40.281 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=16993 2024-09-11T06:26:40.281 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 16993 2024-09-11T06:26:40.281 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:26:40.281 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:26:40.281 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:26:40.483 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:26:40.483 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:40.483 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:26:40.483 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:26:40.483 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:26:40.483 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:26:40.483 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:26:40.483 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:26:40.483 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:26:40.483 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:40.483 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:40.484 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:26:40.484 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:26:40.484 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:26:40.484 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:26:40.484 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:26:40.484 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:26:40.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:26:40.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:26:40.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:26:40.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:40.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:26:40.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:26:40.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:26:40.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:26:40.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:26:40.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:26:40.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:26:40.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:40.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:26:40.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:40.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:26:40.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:40.492 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:26:40.492 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:26:40.492 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:26:40.492 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:26:40.492 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:26:40.492 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:26:40.492 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:26:40.493 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:26:40.493 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:40.493 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:40.493 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:40.493 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:26:40.493 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:26:40.493 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:26:40.493 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:26:40.493 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:26:40.514 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:40.515 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:40.515 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:40.515 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:26:40.649 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:26:40.650 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:26:40.650 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:40.650 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:40.650 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:40.651 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:26:40.651 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:40.651 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:40.651 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:40.651 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:26:40.651 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:40.652 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:40.652 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:40.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:40.654 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:40.654 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:40.701 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 28 2024-09-11T06:26:40.701 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:40.701 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:40.701 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:40.701 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=114, tag_tid=20, entry_tid=17078], [object_number=113, tag_tid=20, entry_tid=17077], [object_number=112, tag_tid=20, entry_tid=17076], [object_number=115, tag_tid=20, entry_tid=17075]]], state=connected] 2024-09-11T06:26:40.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:40.706 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:40.707 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:26:40.707 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:40.707 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:40.707 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:40.707 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:26:40.707 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:40.707 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:40.707 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:40.707 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:26:40.707 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:40.707 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:40.707 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:40.708 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:40.708 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:40.708 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:40.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 28 2024-09-11T06:26:40.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:40.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:40.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:40.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=114, tag_tid=20, entry_tid=17078], [object_number=113, tag_tid=20, entry_tid=17077], [object_number=112, tag_tid=20, entry_tid=17076], [object_number=115, tag_tid=20, entry_tid=17075]]], state=connected] 2024-09-11T06:26:40.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:40.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=114, tag_tid=20, entry_tid=17078' 2024-09-11T06:26:40.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=114, tag_tid=20, entry_tid=17078' 2024-09-11T06:26:40.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=114, tag_tid=20, entry_tid=17078' '!=' 'object_number=113, tag_tid=20, entry_tid=16993' 2024-09-11T06:26:40.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:26:40.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=114, tag_tid=20, entry_tid=17078' = 'object_number=113, tag_tid=20, entry_tid=16993' 2024-09-11T06:26:40.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=114, tag_tid=20, entry_tid=17078' 2024-09-11T06:26:40.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:40.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:40.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:40.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:26:40.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=114, tag_tid=20, entry_tid=17078' 2024-09-11T06:26:40.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:40.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:40.764 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:26:40.764 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:40.765 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:40.765 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:40.765 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:26:40.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=114, tag_tid=20, entry_tid=17078' 2024-09-11T06:26:40.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:40.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:40.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=17078 2024-09-11T06:26:40.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 17078 2024-09-11T06:26:40.768 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:26:40.768 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:26:40.768 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:26:40.969 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:26:40.969 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:40.970 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:26:40.970 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:26:40.970 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:26:40.970 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:26:40.970 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:26:40.970 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:26:40.970 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:26:40.970 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:40.970 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:40.970 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:26:40.970 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:26:40.970 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:26:40.970 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:26:40.971 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:26:40.971 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:26:40.975 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:26:40.975 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:26:40.975 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:26:40.975 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:40.975 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:26:40.975 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:26:40.976 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:26:40.976 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:26:40.976 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:26:40.976 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:26:40.976 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:26:40.976 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:40.976 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:26:40.976 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:40.976 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:26:40.976 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:40.976 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:26:40.976 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:26:40.976 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:26:40.976 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:26:40.977 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:26:40.977 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:26:40.977 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:26:40.977 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:26:40.977 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:40.977 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:40.977 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:40.977 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:26:40.977 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:26:40.977 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:26:40.977 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:26:40.977 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:26:41.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:41.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:41.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:41.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:26:41.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:26:41.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:26:41.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:41.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:41.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:41.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:26:41.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:41.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:41.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:41.133 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:26:41.133 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:41.133 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:41.133 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:41.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:41.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:41.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:41.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 28 2024-09-11T06:26:41.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:41.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:41.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:41.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=113, tag_tid=20, entry_tid=17157], [object_number=112, tag_tid=20, entry_tid=17156], [object_number=115, tag_tid=20, entry_tid=17155], [object_number=114, tag_tid=20, entry_tid=17154]]], state=connected] 2024-09-11T06:26:41.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:41.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:41.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:26:41.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:41.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:41.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:41.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:26:41.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:41.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:41.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:41.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:26:41.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:41.196 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:41.196 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:41.196 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:41.197 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:41.197 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:41.245 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 28 2024-09-11T06:26:41.245 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:41.245 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:41.245 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:41.245 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=113, tag_tid=20, entry_tid=17157], [object_number=112, tag_tid=20, entry_tid=17156], [object_number=115, tag_tid=20, entry_tid=17155], [object_number=114, tag_tid=20, entry_tid=17154]]], state=connected] 2024-09-11T06:26:41.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:41.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=113, tag_tid=20, entry_tid=17157' 2024-09-11T06:26:41.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=113, tag_tid=20, entry_tid=17157' 2024-09-11T06:26:41.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=113, tag_tid=20, entry_tid=17157' '!=' 'object_number=114, tag_tid=20, entry_tid=17078' 2024-09-11T06:26:41.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:26:41.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=113, tag_tid=20, entry_tid=17157' = 'object_number=114, tag_tid=20, entry_tid=17078' 2024-09-11T06:26:41.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=113, tag_tid=20, entry_tid=17157' 2024-09-11T06:26:41.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:41.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:41.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:41.252 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:26:41.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=20, entry_tid=17157' 2024-09-11T06:26:41.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:41.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:41.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:26:41.254 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:41.254 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:41.254 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:41.255 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:26:41.256 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=20, entry_tid=17157' 2024-09-11T06:26:41.256 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:41.256 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:41.257 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=17157 2024-09-11T06:26:41.257 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 17157 2024-09-11T06:26:41.257 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:26:41.258 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:26:41.258 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:26:41.459 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:26:41.459 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:41.459 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:26:41.459 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:26:41.459 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:26:41.459 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:26:41.459 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:26:41.459 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:26:41.460 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:26:41.460 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:41.460 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:41.460 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:26:41.460 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:26:41.460 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:26:41.460 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:26:41.460 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:26:41.461 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:26:41.464 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:26:41.464 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:26:41.464 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:26:41.465 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:41.465 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:26:41.465 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:26:41.465 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:26:41.465 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:26:41.465 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:26:41.465 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:26:41.465 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:26:41.465 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:41.465 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:26:41.465 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:41.465 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:26:41.466 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:41.466 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:26:41.466 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:26:41.466 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:26:41.466 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:26:41.466 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:26:41.466 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:26:41.466 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:26:41.466 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:26:41.466 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:41.466 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:41.466 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:41.466 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:26:41.467 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:26:41.467 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:26:41.467 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:26:41.467 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:26:41.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:41.490 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:41.490 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:41.490 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:26:41.618 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:26:41.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:26:41.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:41.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:41.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:41.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:26:41.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:41.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:41.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:41.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:26:41.621 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:41.621 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:41.621 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:41.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:41.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:41.623 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:41.674 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 28 2024-09-11T06:26:41.674 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:41.674 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:41.674 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:41.674 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=113, tag_tid=20, entry_tid=17237], [object_number=112, tag_tid=20, entry_tid=17236], [object_number=115, tag_tid=20, entry_tid=17235], [object_number=114, tag_tid=20, entry_tid=17234]]], state=connected] 2024-09-11T06:26:41.677 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:41.679 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:41.680 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:26:41.680 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:41.680 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:41.680 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:41.680 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:26:41.680 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:41.680 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:41.681 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:41.681 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:26:41.681 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:41.681 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:41.681 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:41.681 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:41.682 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:41.682 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:41.936 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 28 2024-09-11T06:26:41.937 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:41.937 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:41.937 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:41.937 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=113, tag_tid=20, entry_tid=17237], [object_number=112, tag_tid=20, entry_tid=17236], [object_number=115, tag_tid=20, entry_tid=17235], [object_number=114, tag_tid=20, entry_tid=17234]]], state=connected] 2024-09-11T06:26:41.943 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:41.946 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=113, tag_tid=20, entry_tid=17237' 2024-09-11T06:26:41.946 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=113, tag_tid=20, entry_tid=17237' 2024-09-11T06:26:41.946 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=113, tag_tid=20, entry_tid=17237' '!=' 'object_number=113, tag_tid=20, entry_tid=17157' 2024-09-11T06:26:41.946 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:26:41.946 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=113, tag_tid=20, entry_tid=17237' = 'object_number=113, tag_tid=20, entry_tid=17157' 2024-09-11T06:26:41.946 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=113, tag_tid=20, entry_tid=17237' 2024-09-11T06:26:41.947 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:41.947 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:41.947 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:41.949 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:26:41.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=20, entry_tid=17237' 2024-09-11T06:26:41.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:41.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:41.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:26:41.952 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:41.952 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:41.952 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:41.954 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:26:41.954 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=20, entry_tid=17237' 2024-09-11T06:26:41.955 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:41.955 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:41.956 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=17237 2024-09-11T06:26:41.957 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 17237 2024-09-11T06:26:41.957 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:26:41.957 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:26:41.957 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:26:42.159 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:26:42.159 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:42.159 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:26:42.159 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:26:42.159 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:26:42.159 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:26:42.159 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:26:42.159 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:26:42.159 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:26:42.159 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:42.160 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:42.160 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:26:42.160 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:26:42.160 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:26:42.160 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:26:42.160 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:26:42.161 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:26:42.164 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:26:42.164 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:26:42.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:26:42.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:42.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:26:42.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:26:42.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:26:42.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:26:42.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:26:42.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:26:42.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:26:42.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:42.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:26:42.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:42.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:26:42.166 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:42.166 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:26:42.166 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:26:42.166 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:26:42.166 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:26:42.166 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:26:42.166 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:26:42.166 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:26:42.166 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:26:42.166 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:42.166 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:42.166 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:42.166 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:26:42.166 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:26:42.166 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:26:42.167 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:26:42.167 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:26:42.190 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:42.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:42.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:42.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:26:42.321 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:26:42.322 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:26:42.322 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:42.322 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:42.322 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:42.322 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:26:42.323 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:42.323 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:42.323 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:42.323 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:26:42.323 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:42.323 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:42.323 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:42.325 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:42.325 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:42.325 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:42.378 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 28 2024-09-11T06:26:42.413 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:42.413 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:42.413 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:42.413 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=115, tag_tid=20, entry_tid=17351], [object_number=114, tag_tid=20, entry_tid=17350], [object_number=113, tag_tid=20, entry_tid=17349], [object_number=112, tag_tid=20, entry_tid=17348]]], state=connected] 2024-09-11T06:26:42.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:42.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:42.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:26:42.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:42.416 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:42.416 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:42.416 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:26:42.416 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:42.416 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:42.416 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:42.416 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:26:42.416 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:42.416 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:42.416 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:42.416 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:42.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:42.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:42.435 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 28 2024-09-11T06:26:42.435 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:42.435 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:42.435 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:42.435 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=115, tag_tid=20, entry_tid=17351], [object_number=114, tag_tid=20, entry_tid=17350], [object_number=113, tag_tid=20, entry_tid=17349], [object_number=112, tag_tid=20, entry_tid=17348]]], state=connected] 2024-09-11T06:26:42.438 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:42.440 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=115, tag_tid=20, entry_tid=17351' 2024-09-11T06:26:42.440 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=115, tag_tid=20, entry_tid=17351' 2024-09-11T06:26:42.440 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=115, tag_tid=20, entry_tid=17351' '!=' 'object_number=113, tag_tid=20, entry_tid=17237' 2024-09-11T06:26:42.440 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:26:42.440 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=115, tag_tid=20, entry_tid=17351' = 'object_number=113, tag_tid=20, entry_tid=17237' 2024-09-11T06:26:42.441 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=115, tag_tid=20, entry_tid=17351' 2024-09-11T06:26:42.441 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:42.441 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:42.441 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:42.442 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:26:42.442 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=115, tag_tid=20, entry_tid=17351' 2024-09-11T06:26:42.442 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:42.442 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:42.444 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:26:42.444 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:42.444 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:42.444 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:42.445 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:26:42.446 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=115, tag_tid=20, entry_tid=17351' 2024-09-11T06:26:42.446 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:42.446 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:42.447 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=17351 2024-09-11T06:26:42.447 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 17351 2024-09-11T06:26:42.447 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:26:42.447 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:26:42.447 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:26:42.649 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:26:42.649 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:42.649 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:26:42.649 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:26:42.650 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:26:42.650 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:26:42.650 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:26:42.650 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:26:42.650 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:26:42.650 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:42.650 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:42.650 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:26:42.650 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:26:42.650 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:26:42.650 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:26:42.651 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:26:42.651 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:26:42.654 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:26:42.655 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:26:42.655 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:26:42.655 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:42.655 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:26:42.655 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:26:42.655 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:26:42.655 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:26:42.655 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:26:42.655 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:26:42.655 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:26:42.655 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:42.655 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:26:42.656 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:42.656 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:26:42.656 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:42.656 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:26:42.656 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:26:42.656 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:26:42.656 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:26:42.656 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:26:42.656 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:26:42.656 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:26:42.656 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:26:42.656 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:42.656 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:42.656 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:42.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:26:42.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:26:42.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:26:42.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:26:42.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:26:42.680 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:42.680 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:42.681 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:42.681 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:26:42.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:26:42.812 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:26:42.812 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:42.813 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:42.813 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:42.813 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:26:42.813 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:42.813 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:42.813 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:42.813 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:26:42.813 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:42.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:42.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:42.815 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:42.815 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:42.815 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:42.865 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 28 2024-09-11T06:26:42.865 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:42.866 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:42.866 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:42.866 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=112, tag_tid=20, entry_tid=17432], [object_number=115, tag_tid=20, entry_tid=17431], [object_number=114, tag_tid=20, entry_tid=17430], [object_number=113, tag_tid=20, entry_tid=17429]]], state=connected] 2024-09-11T06:26:42.868 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:42.869 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:42.870 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:26:42.870 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:42.871 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:42.871 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:42.871 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:26:42.871 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:42.871 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:42.871 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:42.871 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:26:42.871 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:42.871 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:42.871 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:42.872 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:42.872 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:42.872 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:42.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 28 2024-09-11T06:26:42.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:42.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:42.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:42.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=112, tag_tid=20, entry_tid=17432], [object_number=115, tag_tid=20, entry_tid=17431], [object_number=114, tag_tid=20, entry_tid=17430], [object_number=113, tag_tid=20, entry_tid=17429]]], state=connected] 2024-09-11T06:26:42.922 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:42.925 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=112, tag_tid=20, entry_tid=17432' 2024-09-11T06:26:42.925 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=112, tag_tid=20, entry_tid=17432' 2024-09-11T06:26:42.925 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=112, tag_tid=20, entry_tid=17432' '!=' 'object_number=115, tag_tid=20, entry_tid=17351' 2024-09-11T06:26:42.925 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:26:42.925 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=112, tag_tid=20, entry_tid=17432' = 'object_number=115, tag_tid=20, entry_tid=17351' 2024-09-11T06:26:42.925 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=112, tag_tid=20, entry_tid=17432' 2024-09-11T06:26:42.925 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:42.925 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:42.925 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:42.926 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:26:42.926 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=112, tag_tid=20, entry_tid=17432' 2024-09-11T06:26:42.926 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:42.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:42.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:26:42.928 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:42.928 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:42.928 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:42.929 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:26:42.930 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=112, tag_tid=20, entry_tid=17432' 2024-09-11T06:26:42.930 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:42.930 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:42.931 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=17432 2024-09-11T06:26:42.931 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 17432 2024-09-11T06:26:42.931 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:26:42.931 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:26:42.931 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:26:43.133 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:26:43.133 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:43.133 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:26:43.133 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:26:43.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:26:43.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:26:43.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:26:43.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:26:43.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:26:43.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:43.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:43.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:26:43.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:26:43.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:26:43.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:26:43.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:26:43.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:26:43.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:26:43.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:26:43.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:26:43.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:43.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:26:43.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:26:43.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:26:43.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:26:43.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:26:43.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:26:43.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:26:43.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:43.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:26:43.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:43.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:26:43.139 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:43.139 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:26:43.139 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:26:43.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:26:43.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:26:43.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:26:43.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:26:43.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:26:43.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:26:43.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:43.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:43.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:43.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:26:43.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:26:43.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:26:43.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:26:43.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:26:43.166 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:43.166 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:43.166 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:43.166 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:26:43.297 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:26:43.298 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:26:43.298 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:43.298 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:43.298 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:43.298 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:26:43.298 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:43.298 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:43.298 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:43.298 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:26:43.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:43.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:43.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:43.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:43.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:43.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:43.349 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 28 2024-09-11T06:26:43.349 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:43.349 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:43.349 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:43.349 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=113, tag_tid=20, entry_tid=17513], [object_number=112, tag_tid=20, entry_tid=17512], [object_number=115, tag_tid=20, entry_tid=17511], [object_number=114, tag_tid=20, entry_tid=17510]]], state=connected] 2024-09-11T06:26:43.352 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:43.353 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:43.354 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:26:43.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:43.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:43.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:43.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:26:43.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:43.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:43.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:43.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:26:43.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:43.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:43.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:43.356 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:43.356 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:43.356 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:43.405 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 28 2024-09-11T06:26:43.405 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:43.405 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:43.405 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:43.405 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=113, tag_tid=20, entry_tid=17513], [object_number=112, tag_tid=20, entry_tid=17512], [object_number=115, tag_tid=20, entry_tid=17511], [object_number=114, tag_tid=20, entry_tid=17510]]], state=connected] 2024-09-11T06:26:43.408 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:43.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=113, tag_tid=20, entry_tid=17513' 2024-09-11T06:26:43.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=113, tag_tid=20, entry_tid=17513' 2024-09-11T06:26:43.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=113, tag_tid=20, entry_tid=17513' '!=' 'object_number=112, tag_tid=20, entry_tid=17432' 2024-09-11T06:26:43.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:26:43.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=113, tag_tid=20, entry_tid=17513' = 'object_number=112, tag_tid=20, entry_tid=17432' 2024-09-11T06:26:43.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=113, tag_tid=20, entry_tid=17513' 2024-09-11T06:26:43.411 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:43.411 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:43.411 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:43.411 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:26:43.412 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=20, entry_tid=17513' 2024-09-11T06:26:43.412 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:43.412 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:43.413 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:26:43.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:43.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:43.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:43.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:26:43.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=20, entry_tid=17513' 2024-09-11T06:26:43.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:43.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:43.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=17513 2024-09-11T06:26:43.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 17513 2024-09-11T06:26:43.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:26:43.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:26:43.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:26:43.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:26:43.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:43.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:26:43.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:26:43.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:26:43.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:26:43.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:26:43.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:26:43.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:26:43.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:43.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:43.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:26:43.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:26:43.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:26:43.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:26:43.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:26:43.621 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:26:43.624 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:26:43.625 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:26:43.625 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:26:43.625 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:43.625 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:26:43.625 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:26:43.625 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:26:43.625 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:26:43.625 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:26:43.625 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:26:43.625 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:26:43.625 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:43.626 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:26:43.626 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:43.626 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:26:43.626 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:43.626 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:26:43.626 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:26:43.626 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:26:43.626 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:26:43.626 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:26:43.626 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:26:43.627 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:26:43.627 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:26:43.627 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:43.627 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:43.627 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:43.627 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:26:43.627 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:26:43.627 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:26:43.627 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:26:43.627 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:26:43.650 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:43.651 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:43.651 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:43.651 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:26:43.785 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:26:43.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:26:43.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:43.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:43.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:43.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:26:43.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:43.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:43.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:43.787 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:26:43.787 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:43.787 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:43.787 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:43.790 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:43.790 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:43.790 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:43.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 28 2024-09-11T06:26:43.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:43.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:43.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:43.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=114, tag_tid=20, entry_tid=17586], [object_number=113, tag_tid=20, entry_tid=17585], [object_number=112, tag_tid=20, entry_tid=17584], [object_number=115, tag_tid=20, entry_tid=17583]]], state=connected] 2024-09-11T06:26:43.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:43.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:43.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:26:43.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:43.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:43.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:43.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:26:43.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:43.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:43.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:43.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:26:43.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:43.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:43.854 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:43.854 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:43.854 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:43.854 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:43.902 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 28 2024-09-11T06:26:43.902 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:43.902 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:43.902 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:43.902 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=114, tag_tid=20, entry_tid=17586], [object_number=113, tag_tid=20, entry_tid=17585], [object_number=112, tag_tid=20, entry_tid=17584], [object_number=115, tag_tid=20, entry_tid=17583]]], state=connected] 2024-09-11T06:26:43.905 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:43.907 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=114, tag_tid=20, entry_tid=17586' 2024-09-11T06:26:43.908 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=114, tag_tid=20, entry_tid=17586' 2024-09-11T06:26:43.908 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=114, tag_tid=20, entry_tid=17586' '!=' 'object_number=113, tag_tid=20, entry_tid=17513' 2024-09-11T06:26:43.908 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:26:43.908 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=114, tag_tid=20, entry_tid=17586' = 'object_number=113, tag_tid=20, entry_tid=17513' 2024-09-11T06:26:43.908 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=114, tag_tid=20, entry_tid=17586' 2024-09-11T06:26:43.908 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:43.909 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:43.909 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:43.909 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:26:43.909 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=114, tag_tid=20, entry_tid=17586' 2024-09-11T06:26:43.909 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:43.910 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:43.910 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:26:43.911 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:43.911 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:43.911 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:43.912 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:26:43.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=114, tag_tid=20, entry_tid=17586' 2024-09-11T06:26:43.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:43.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:43.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=17586 2024-09-11T06:26:43.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 17586 2024-09-11T06:26:43.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:26:43.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:26:43.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:26:44.116 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:26:44.116 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:44.116 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:26:44.116 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:26:44.116 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:26:44.116 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:26:44.116 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:26:44.116 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:26:44.116 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:26:44.116 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:44.116 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:44.117 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:26:44.117 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:26:44.117 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:26:44.117 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:26:44.117 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:26:44.118 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:26:44.120 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:26:44.120 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:26:44.120 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:26:44.121 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:44.121 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:26:44.121 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:26:44.121 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:26:44.121 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:26:44.121 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:26:44.121 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:26:44.121 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:26:44.121 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:44.121 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:26:44.121 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:44.121 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:26:44.121 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:44.122 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:26:44.122 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:26:44.123 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:26:44.123 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:26:44.123 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:26:44.123 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:26:44.123 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:26:44.123 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:26:44.123 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:44.123 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:44.123 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:44.123 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:26:44.123 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:26:44.123 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:26:44.123 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:26:44.124 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:26:44.147 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:44.147 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:44.148 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:44.148 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:26:44.278 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:26:44.279 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:26:44.279 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:44.279 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:44.279 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:44.279 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:26:44.279 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:44.280 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:44.280 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:44.280 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:26:44.281 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:44.281 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:44.281 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:44.282 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:44.283 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:44.283 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:44.338 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 28 2024-09-11T06:26:44.338 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:44.338 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:44.338 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:44.339 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=115, tag_tid=20, entry_tid=17671], [object_number=114, tag_tid=20, entry_tid=17670], [object_number=113, tag_tid=20, entry_tid=17669], [object_number=112, tag_tid=20, entry_tid=17668]]], state=connected] 2024-09-11T06:26:44.341 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:44.343 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:44.344 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:26:44.344 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:44.345 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:44.345 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:44.345 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:26:44.345 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:44.345 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:44.345 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:44.345 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:26:44.345 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:44.345 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:44.345 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:44.346 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:44.346 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:44.346 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:44.393 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 28 2024-09-11T06:26:44.393 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:44.393 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:44.393 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:44.393 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=115, tag_tid=20, entry_tid=17671], [object_number=114, tag_tid=20, entry_tid=17670], [object_number=113, tag_tid=20, entry_tid=17669], [object_number=112, tag_tid=20, entry_tid=17668]]], state=connected] 2024-09-11T06:26:44.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:44.404 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=115, tag_tid=20, entry_tid=17671' 2024-09-11T06:26:44.404 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=115, tag_tid=20, entry_tid=17671' 2024-09-11T06:26:44.404 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=115, tag_tid=20, entry_tid=17671' '!=' 'object_number=114, tag_tid=20, entry_tid=17586' 2024-09-11T06:26:44.404 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:26:44.404 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=115, tag_tid=20, entry_tid=17671' = 'object_number=114, tag_tid=20, entry_tid=17586' 2024-09-11T06:26:44.404 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=115, tag_tid=20, entry_tid=17671' 2024-09-11T06:26:44.405 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:44.405 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:44.405 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:44.405 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:26:44.406 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=115, tag_tid=20, entry_tid=17671' 2024-09-11T06:26:44.406 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:44.406 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:44.407 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:26:44.408 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:44.408 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:44.408 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:44.409 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:26:44.409 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=115, tag_tid=20, entry_tid=17671' 2024-09-11T06:26:44.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:44.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:44.411 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=17671 2024-09-11T06:26:44.411 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 17671 2024-09-11T06:26:44.411 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:26:44.411 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:26:44.411 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:26:44.613 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:26:44.613 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:44.614 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:26:44.614 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:26:44.614 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:26:44.614 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:26:44.614 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:26:44.614 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:26:44.614 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:26:44.614 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:44.614 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:44.614 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:26:44.614 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:26:44.615 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:26:44.615 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:26:44.616 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:26:44.616 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:26:44.617 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:26:44.617 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:26:44.617 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:26:44.617 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:44.617 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:26:44.617 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:26:44.617 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:26:44.617 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:26:44.617 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:26:44.617 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:26:44.617 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:26:44.617 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:44.617 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:26:44.618 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:44.618 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:26:44.618 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:44.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:26:44.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:26:44.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:26:44.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:26:44.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:26:44.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:26:44.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:26:44.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:26:44.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:44.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:44.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:44.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:26:44.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:26:44.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:26:44.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:26:44.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:26:44.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:44.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:44.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:44.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:26:44.785 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:26:44.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:26:44.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:44.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:44.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:44.787 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:26:44.787 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:44.787 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:44.787 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:44.787 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:26:44.787 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:44.788 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:44.788 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:44.789 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:44.789 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:44.790 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:44.838 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 28 2024-09-11T06:26:44.838 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:44.838 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:44.838 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:44.838 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=112, tag_tid=20, entry_tid=17760], [object_number=115, tag_tid=20, entry_tid=17759], [object_number=114, tag_tid=20, entry_tid=17758], [object_number=113, tag_tid=20, entry_tid=17757]]], state=connected] 2024-09-11T06:26:44.841 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:44.842 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:44.843 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:26:44.843 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:44.843 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:44.843 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:44.844 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:26:44.844 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:44.844 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:44.844 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:44.844 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:26:44.844 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:44.844 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:44.844 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:44.845 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:44.845 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:44.845 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:44.891 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 28 2024-09-11T06:26:44.892 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:44.892 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:44.892 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:44.892 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=112, tag_tid=20, entry_tid=17760], [object_number=115, tag_tid=20, entry_tid=17759], [object_number=114, tag_tid=20, entry_tid=17758], [object_number=113, tag_tid=20, entry_tid=17757]]], state=connected] 2024-09-11T06:26:44.895 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:44.897 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=112, tag_tid=20, entry_tid=17760' 2024-09-11T06:26:44.897 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=112, tag_tid=20, entry_tid=17760' 2024-09-11T06:26:44.898 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=112, tag_tid=20, entry_tid=17760' '!=' 'object_number=115, tag_tid=20, entry_tid=17671' 2024-09-11T06:26:44.898 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:26:44.898 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=112, tag_tid=20, entry_tid=17760' = 'object_number=115, tag_tid=20, entry_tid=17671' 2024-09-11T06:26:44.898 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=112, tag_tid=20, entry_tid=17760' 2024-09-11T06:26:44.898 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:44.898 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:44.898 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:44.899 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:26:44.899 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=112, tag_tid=20, entry_tid=17760' 2024-09-11T06:26:44.899 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:44.899 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:44.900 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:26:44.901 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:44.901 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:44.901 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:44.902 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:26:44.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=112, tag_tid=20, entry_tid=17760' 2024-09-11T06:26:44.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:44.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:44.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=17760 2024-09-11T06:26:44.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 17760 2024-09-11T06:26:44.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:26:44.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:26:44.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:26:45.106 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:26:45.106 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:45.106 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:26:45.106 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:26:45.106 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:26:45.106 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:26:45.106 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:26:45.106 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:26:45.107 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:26:45.107 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:45.107 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:45.107 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:26:45.107 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:26:45.107 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:26:45.107 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:26:45.107 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:26:45.108 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:26:45.110 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:26:45.110 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:26:45.110 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:26:45.110 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:45.110 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:26:45.110 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:26:45.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:26:45.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:26:45.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:26:45.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:26:45.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:26:45.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:45.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:26:45.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:45.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:26:45.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:45.112 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:26:45.112 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:26:45.112 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:26:45.112 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:26:45.113 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:26:45.113 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:26:45.113 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:26:45.113 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:26:45.113 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:45.113 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:45.113 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:45.113 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:26:45.113 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:26:45.113 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:26:45.114 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:26:45.114 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:26:45.139 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:45.139 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:45.139 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:45.139 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:26:45.275 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:26:45.276 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:26:45.276 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:45.276 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:45.276 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:45.276 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:26:45.276 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:45.276 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:45.277 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:45.277 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:26:45.277 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:45.277 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:45.277 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:45.279 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:45.279 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:45.279 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:45.332 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 28 2024-09-11T06:26:45.332 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:45.332 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:45.332 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:45.332 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=112, tag_tid=20, entry_tid=17848], [object_number=115, tag_tid=20, entry_tid=17847], [object_number=114, tag_tid=20, entry_tid=17846], [object_number=113, tag_tid=20, entry_tid=17845]]], state=connected] 2024-09-11T06:26:45.335 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:45.337 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:45.338 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:26:45.338 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:45.338 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:45.339 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:45.339 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:26:45.339 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:45.339 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:45.339 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:45.339 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:26:45.339 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:45.339 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:45.339 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:45.340 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:45.340 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:45.340 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:45.386 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 28 2024-09-11T06:26:45.387 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:45.387 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:45.387 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:45.387 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=112, tag_tid=20, entry_tid=17848], [object_number=115, tag_tid=20, entry_tid=17847], [object_number=114, tag_tid=20, entry_tid=17846], [object_number=113, tag_tid=20, entry_tid=17845]]], state=connected] 2024-09-11T06:26:45.390 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:45.392 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=112, tag_tid=20, entry_tid=17848' 2024-09-11T06:26:45.392 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=112, tag_tid=20, entry_tid=17848' 2024-09-11T06:26:45.392 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=112, tag_tid=20, entry_tid=17848' '!=' 'object_number=112, tag_tid=20, entry_tid=17760' 2024-09-11T06:26:45.392 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:26:45.392 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=112, tag_tid=20, entry_tid=17848' = 'object_number=112, tag_tid=20, entry_tid=17760' 2024-09-11T06:26:45.392 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=112, tag_tid=20, entry_tid=17848' 2024-09-11T06:26:45.393 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:45.393 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:45.393 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:45.394 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:26:45.394 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=112, tag_tid=20, entry_tid=17848' 2024-09-11T06:26:45.394 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:45.394 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:45.395 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:26:45.396 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:45.396 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:45.396 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:45.397 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:26:45.397 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=112, tag_tid=20, entry_tid=17848' 2024-09-11T06:26:45.398 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:45.398 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:45.399 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=17848 2024-09-11T06:26:45.399 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 17848 2024-09-11T06:26:45.399 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:26:45.399 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:26:45.399 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:26:45.602 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:26:45.602 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:45.602 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:26:45.602 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:26:45.602 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:26:45.602 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:26:45.602 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:26:45.602 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:26:45.602 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:26:45.603 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:45.603 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:45.603 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:26:45.603 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:26:45.603 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:26:45.603 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:26:45.603 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:26:45.606 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:26:45.606 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:26:45.607 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:26:45.607 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:26:45.607 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:45.607 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:26:45.607 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:26:45.607 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:26:45.607 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:26:45.607 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:26:45.607 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:26:45.607 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:26:45.607 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:45.607 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:26:45.607 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:45.608 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:26:45.608 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:45.608 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:26:45.608 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:26:45.608 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:26:45.608 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:26:45.608 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:26:45.608 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:26:45.608 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:26:45.608 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:26:45.608 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:45.608 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:45.608 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:45.609 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:26:45.609 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:26:45.609 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:26:45.609 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:26:45.609 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:26:45.631 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:45.631 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:45.632 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:45.632 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:26:45.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:26:45.764 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:26:45.764 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:45.765 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:45.765 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:45.765 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:26:45.765 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:45.765 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:45.765 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:45.765 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:26:45.765 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:45.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:45.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:45.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:45.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:45.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:45.819 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 28 2024-09-11T06:26:45.819 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:45.819 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:45.819 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:45.819 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=113, tag_tid=20, entry_tid=17941], [object_number=112, tag_tid=20, entry_tid=17940], [object_number=115, tag_tid=20, entry_tid=17939], [object_number=114, tag_tid=20, entry_tid=17938]]], state=connected] 2024-09-11T06:26:45.822 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:45.823 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:45.825 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:26:45.825 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:45.825 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:45.825 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:45.825 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:26:45.825 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:45.825 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:45.825 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:45.825 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:26:45.825 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:45.825 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:45.826 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:45.826 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:45.826 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:45.826 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:45.876 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 28 2024-09-11T06:26:45.876 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:45.876 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:45.876 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:45.876 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=113, tag_tid=20, entry_tid=17941], [object_number=112, tag_tid=20, entry_tid=17940], [object_number=115, tag_tid=20, entry_tid=17939], [object_number=114, tag_tid=20, entry_tid=17938]]], state=connected] 2024-09-11T06:26:45.879 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:45.881 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=113, tag_tid=20, entry_tid=17941' 2024-09-11T06:26:45.881 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=113, tag_tid=20, entry_tid=17941' 2024-09-11T06:26:45.881 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=113, tag_tid=20, entry_tid=17941' '!=' 'object_number=112, tag_tid=20, entry_tid=17848' 2024-09-11T06:26:45.881 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:26:45.881 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=113, tag_tid=20, entry_tid=17941' = 'object_number=112, tag_tid=20, entry_tid=17848' 2024-09-11T06:26:45.881 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=113, tag_tid=20, entry_tid=17941' 2024-09-11T06:26:45.881 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:45.881 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:45.882 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:45.882 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:26:45.883 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=20, entry_tid=17941' 2024-09-11T06:26:45.883 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:45.883 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:45.884 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:26:45.884 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:45.885 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:45.885 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:45.885 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:26:45.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=20, entry_tid=17941' 2024-09-11T06:26:45.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:45.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:45.887 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=17941 2024-09-11T06:26:45.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 17941 2024-09-11T06:26:45.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:26:45.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:26:45.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:26:46.090 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:26:46.090 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:46.090 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:26:46.090 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:26:46.090 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:26:46.090 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:26:46.090 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:26:46.090 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:26:46.090 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:26:46.090 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:46.091 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:46.091 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:26:46.091 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:26:46.091 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:26:46.091 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:26:46.091 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:26:46.091 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:26:46.095 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:26:46.095 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:26:46.095 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:26:46.095 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:46.095 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:26:46.096 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:26:46.096 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:26:46.096 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:26:46.096 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:26:46.096 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:26:46.096 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:26:46.096 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:46.096 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:26:46.096 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:46.096 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:26:46.097 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:46.097 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:26:46.097 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:26:46.097 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:26:46.097 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:26:46.097 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:26:46.097 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:26:46.097 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:26:46.097 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:26:46.097 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:46.098 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:46.098 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:46.098 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:26:46.098 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:26:46.098 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:26:46.098 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:26:46.098 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:26:46.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:46.120 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:46.120 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:46.120 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:26:46.252 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:26:46.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:26:46.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:46.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:46.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:46.254 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:26:46.254 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:46.254 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:46.254 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:46.254 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:26:46.254 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:46.255 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:46.255 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:46.257 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:46.257 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:46.257 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:46.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 28 2024-09-11T06:26:46.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:46.310 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:46.310 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:46.310 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=115, tag_tid=20, entry_tid=18031], [object_number=114, tag_tid=20, entry_tid=18030], [object_number=113, tag_tid=20, entry_tid=18029], [object_number=112, tag_tid=20, entry_tid=18028]]], state=connected] 2024-09-11T06:26:46.313 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:46.314 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:46.315 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:26:46.315 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:46.315 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:46.315 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:46.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:26:46.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:46.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:46.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:46.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:26:46.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:46.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:46.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:46.317 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:46.317 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:46.317 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:46.364 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 28 2024-09-11T06:26:46.364 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:46.364 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:46.364 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:46.364 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=115, tag_tid=20, entry_tid=18031], [object_number=114, tag_tid=20, entry_tid=18030], [object_number=113, tag_tid=20, entry_tid=18029], [object_number=112, tag_tid=20, entry_tid=18028]]], state=connected] 2024-09-11T06:26:46.368 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:46.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=115, tag_tid=20, entry_tid=18031' 2024-09-11T06:26:46.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=115, tag_tid=20, entry_tid=18031' 2024-09-11T06:26:46.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=115, tag_tid=20, entry_tid=18031' '!=' 'object_number=113, tag_tid=20, entry_tid=17941' 2024-09-11T06:26:46.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:26:46.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=115, tag_tid=20, entry_tid=18031' = 'object_number=113, tag_tid=20, entry_tid=17941' 2024-09-11T06:26:46.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=115, tag_tid=20, entry_tid=18031' 2024-09-11T06:26:46.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:46.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:46.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:46.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:26:46.372 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=115, tag_tid=20, entry_tid=18031' 2024-09-11T06:26:46.372 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:46.372 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:46.373 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:26:46.374 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:46.374 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:46.374 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:46.375 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:26:46.375 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=115, tag_tid=20, entry_tid=18031' 2024-09-11T06:26:46.375 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:46.376 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:46.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=18031 2024-09-11T06:26:46.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 18031 2024-09-11T06:26:46.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:26:46.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:26:46.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:26:46.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:26:46.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:46.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:26:46.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:26:46.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:26:46.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:26:46.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:26:46.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:26:46.580 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:26:46.580 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:46.580 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:46.580 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:26:46.580 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:26:46.580 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:26:46.580 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:26:46.581 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:26:46.581 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:26:46.584 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:26:46.584 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:26:46.584 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:26:46.584 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:46.584 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:26:46.584 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:26:46.584 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:26:46.584 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:26:46.584 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:26:46.584 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:26:46.584 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:26:46.584 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:46.584 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:26:46.585 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:46.585 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:26:46.585 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:46.585 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:26:46.585 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:26:46.585 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:26:46.585 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:26:46.585 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:26:46.585 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:26:46.585 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:26:46.585 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:26:46.586 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:46.586 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:46.586 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:46.586 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:26:46.586 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:26:46.586 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:26:46.586 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:26:46.586 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:26:46.610 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:46.611 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:46.611 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:46.611 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:26:46.740 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:26:46.741 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:26:46.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:46.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:46.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:46.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:26:46.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:46.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:46.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:46.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:26:46.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:46.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:46.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:46.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:46.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:46.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:46.797 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 28 2024-09-11T06:26:46.797 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:46.797 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:46.797 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:46.797 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=115, tag_tid=20, entry_tid=18123], [object_number=114, tag_tid=20, entry_tid=18122], [object_number=113, tag_tid=20, entry_tid=18121], [object_number=112, tag_tid=20, entry_tid=18120]]], state=connected] 2024-09-11T06:26:46.800 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:46.801 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:46.803 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:26:46.803 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:46.803 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:46.803 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:46.803 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:26:46.803 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:46.803 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:46.803 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:46.803 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:26:46.804 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:46.804 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:46.804 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:46.804 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:46.804 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:46.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:46.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 28 2024-09-11T06:26:46.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:46.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:46.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:46.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=115, tag_tid=20, entry_tid=18123], [object_number=114, tag_tid=20, entry_tid=18122], [object_number=113, tag_tid=20, entry_tid=18121], [object_number=112, tag_tid=20, entry_tid=18120]]], state=connected] 2024-09-11T06:26:46.855 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:46.857 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=115, tag_tid=20, entry_tid=18123' 2024-09-11T06:26:46.857 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=115, tag_tid=20, entry_tid=18123' 2024-09-11T06:26:46.857 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=115, tag_tid=20, entry_tid=18123' '!=' 'object_number=115, tag_tid=20, entry_tid=18031' 2024-09-11T06:26:46.857 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:26:46.857 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=115, tag_tid=20, entry_tid=18123' = 'object_number=115, tag_tid=20, entry_tid=18031' 2024-09-11T06:26:46.857 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=115, tag_tid=20, entry_tid=18123' 2024-09-11T06:26:46.857 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:46.858 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:46.858 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:46.858 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:26:46.859 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=115, tag_tid=20, entry_tid=18123' 2024-09-11T06:26:46.859 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:46.859 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:46.860 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:26:46.860 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:46.861 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:46.861 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:46.861 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:26:46.862 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=115, tag_tid=20, entry_tid=18123' 2024-09-11T06:26:46.862 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:46.862 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:46.863 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=18123 2024-09-11T06:26:46.864 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 18123 2024-09-11T06:26:46.864 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:26:46.864 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:26:46.864 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:26:47.065 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:26:47.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:47.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:26:47.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:26:47.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:26:47.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:26:47.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:26:47.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:26:47.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:26:47.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:47.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:47.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:26:47.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:26:47.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:26:47.067 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:26:47.067 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:26:47.067 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:26:47.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:26:47.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:26:47.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:26:47.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:47.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:26:47.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:26:47.071 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:26:47.071 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:26:47.071 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:26:47.071 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:26:47.071 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:26:47.071 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:47.071 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:26:47.071 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:47.071 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:26:47.071 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:47.071 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:26:47.071 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:26:47.071 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:26:47.072 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:26:47.072 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:26:47.072 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:26:47.072 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:26:47.072 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:26:47.072 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:47.072 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:47.072 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:47.072 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:26:47.072 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:26:47.072 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:26:47.072 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:26:47.072 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:26:47.096 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:47.096 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:47.096 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:47.096 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:26:47.221 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:26:47.222 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:26:47.222 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:47.222 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:47.223 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:47.223 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:26:47.223 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:47.223 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:47.223 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:47.223 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:26:47.224 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:47.224 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:47.224 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:47.225 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:47.226 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:47.226 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:47.280 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 28 2024-09-11T06:26:47.280 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:47.280 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:47.280 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:47.280 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=113, tag_tid=20, entry_tid=18193], [object_number=112, tag_tid=20, entry_tid=18192], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:47.284 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:47.285 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:47.286 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:26:47.286 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:47.286 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:47.286 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:47.286 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:26:47.286 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:47.287 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:47.287 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:47.287 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:26:47.287 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:47.287 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:47.287 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:47.287 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:47.288 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:47.288 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:47.334 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 28 2024-09-11T06:26:47.334 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:47.334 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:47.334 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:47.335 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=113, tag_tid=20, entry_tid=18193], [object_number=112, tag_tid=20, entry_tid=18192], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:47.338 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:47.340 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=113, tag_tid=20, entry_tid=18193' 2024-09-11T06:26:47.340 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=113, tag_tid=20, entry_tid=18193' 2024-09-11T06:26:47.340 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=113, tag_tid=20, entry_tid=18193' '!=' 'object_number=115, tag_tid=20, entry_tid=18123' 2024-09-11T06:26:47.340 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:26:47.340 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=113, tag_tid=20, entry_tid=18193' = 'object_number=115, tag_tid=20, entry_tid=18123' 2024-09-11T06:26:47.340 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=113, tag_tid=20, entry_tid=18193' 2024-09-11T06:26:47.341 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:47.341 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:47.341 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:47.341 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=21 2024-09-11T06:26:47.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=20, entry_tid=18193' 2024-09-11T06:26:47.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:26:47.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:47.343 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=20 2024-09-11T06:26:47.344 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:47.344 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:47.344 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:47.345 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:26:47.345 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=113, tag_tid=20, entry_tid=18193' 2024-09-11T06:26:47.345 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:26:47.345 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:26:47.347 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=18193 2024-09-11T06:26:47.347 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 21 = 20 -a 1 -le 18193 2024-09-11T06:26:47.347 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:26:47.347 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:26:47.347 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:26:47.548 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:26:47.548 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:47.549 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:26:47.549 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:26:47.549 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:26:47.549 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:26:47.549 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:26:47.549 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:26:47.549 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:26:47.549 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:47.549 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:47.549 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:26:47.549 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:26:47.550 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:26:47.550 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:26:47.550 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:26:47.550 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:26:47.553 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:26:47.553 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:26:47.553 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:26:47.553 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:47.553 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:26:47.553 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:26:47.553 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:26:47.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:26:47.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:26:47.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:26:47.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:26:47.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:47.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:26:47.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:47.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:26:47.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:47.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:26:47.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:26:47.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:26:47.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:26:47.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:26:47.555 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:26:47.555 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:26:47.555 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:26:47.555 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:47.555 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:47.555 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:47.555 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:26:47.555 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:26:47.555 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:26:47.555 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:26:47.556 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:26:47.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:47.580 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:47.580 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:47.580 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:26:47.700 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:26:47.702 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:26:47.702 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:47.702 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:47.702 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:47.702 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:26:47.702 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:47.702 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:47.702 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:47.702 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:26:47.702 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:47.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:47.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:47.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:47.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:47.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:47.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 28 2024-09-11T06:26:47.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:47.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:47.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:47.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=113, tag_tid=20, entry_tid=18193], [object_number=112, tag_tid=20, entry_tid=18192], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:47.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:47.769 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:47.769 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:26:47.769 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:47.769 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:47.769 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:47.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:26:47.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:47.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:47.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:47.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:26:47.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:47.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:47.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:47.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:47.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:47.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:47.820 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 28 2024-09-11T06:26:47.820 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:47.820 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:47.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:47.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=113, tag_tid=20, entry_tid=18193], [object_number=112, tag_tid=20, entry_tid=18192], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:47.823 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:47.825 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=113, tag_tid=20, entry_tid=18193' 2024-09-11T06:26:47.825 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=113, tag_tid=20, entry_tid=18193' 2024-09-11T06:26:47.826 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=113, tag_tid=20, entry_tid=18193' '!=' 'object_number=113, tag_tid=20, entry_tid=18193' 2024-09-11T06:26:47.826 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:26:47.826 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.4 2024-09-11T06:26:48.227 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:26:48.227 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:48.228 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:26:48.228 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:26:48.228 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:26:48.228 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:26:48.228 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:26:48.228 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:26:48.228 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:26:48.228 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:48.228 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:48.228 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:26:48.228 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:26:48.228 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:26:48.228 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:26:48.229 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:26:48.229 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:26:48.232 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:26:48.232 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:26:48.232 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:26:48.232 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:48.232 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:26:48.232 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:26:48.232 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:26:48.232 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:26:48.233 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:26:48.233 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:26:48.233 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:26:48.233 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:48.233 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:26:48.233 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:48.233 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:26:48.234 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:48.234 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:26:48.235 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:26:48.235 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:26:48.235 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:26:48.235 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:26:48.235 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:26:48.235 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:26:48.235 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:26:48.235 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:48.235 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:48.235 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:48.235 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:26:48.235 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:26:48.235 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:26:48.235 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:26:48.236 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:26:48.258 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:48.259 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:48.259 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:48.259 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:26:48.379 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:26:48.380 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:26:48.380 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:48.380 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:48.380 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:48.380 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:26:48.380 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:48.380 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:48.381 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:48.381 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:26:48.381 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:48.381 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:48.381 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:48.383 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:48.383 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:48.383 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:48.431 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 28 2024-09-11T06:26:48.431 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:48.431 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:48.431 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:48.431 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=113, tag_tid=20, entry_tid=18193], [object_number=112, tag_tid=20, entry_tid=18192], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:48.434 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:48.436 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:48.437 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:26:48.437 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:48.437 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:48.437 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:48.437 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:26:48.437 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:48.437 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:48.437 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:48.437 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:26:48.437 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:48.438 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:48.438 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:48.438 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:48.438 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:48.438 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:48.484 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 28 2024-09-11T06:26:48.484 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:48.484 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:48.484 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:48.485 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=113, tag_tid=20, entry_tid=18193], [object_number=112, tag_tid=20, entry_tid=18192], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:48.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:48.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=113, tag_tid=20, entry_tid=18193' 2024-09-11T06:26:48.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=113, tag_tid=20, entry_tid=18193' 2024-09-11T06:26:48.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=113, tag_tid=20, entry_tid=18193' '!=' 'object_number=113, tag_tid=20, entry_tid=18193' 2024-09-11T06:26:48.490 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:26:48.490 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.8 2024-09-11T06:26:49.294 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:26:49.294 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:49.294 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:26:49.295 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:26:49.295 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:26:49.295 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:26:49.295 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:26:49.295 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:26:49.295 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:26:49.295 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:49.295 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:49.295 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:26:49.295 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:26:49.295 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:26:49.295 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:26:49.295 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:26:49.295 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:26:49.298 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:26:49.298 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:26:49.298 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:26:49.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:26:49.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:26:49.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:26:49.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:26:49.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:26:49.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:26:49.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:26:49.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:26:49.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:49.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:26:49.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:49.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:26:49.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:49.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:26:49.300 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:26:49.300 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:26:49.300 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:26:49.300 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:26:49.300 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:26:49.300 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:26:49.300 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:26:49.300 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:26:49.300 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:26:49.300 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:26:49.300 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:26:49.300 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:26:49.301 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:26:49.301 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:26:49.301 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:26:49.323 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:49.323 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:49.323 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:26:49.323 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:26:49.451 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:26:49.452 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:26:49.452 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:49.452 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:49.452 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:49.452 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:26:49.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:49.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:49.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:49.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:26:49.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:49.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:49.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:49.455 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:49.455 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:49.455 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:49.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 28 2024-09-11T06:26:49.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:49.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:49.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:49.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:49.509 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:49.510 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:49.511 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:26:49.511 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:49.511 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:49.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:49.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:26:49.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:26:49.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:26:49.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:26:49.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:26:49.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:26:49.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:26:49.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:26:49.513 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:49.513 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:26:49.513 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:49.561 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 28 2024-09-11T06:26:49.561 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 28 2024-09-11T06:26:49.561 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:26:49.561 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:49.561 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=113, tag_tid=21, entry_tid=1], [object_number=112, tag_tid=21, entry_tid=0], [object_number=115, tag_tid=20, entry_tid=18191], [object_number=114, tag_tid=20, entry_tid=18190]]], state=connected] 2024-09-11T06:26:49.564 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:26:49.566 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:49.566 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=113, tag_tid=21, entry_tid=1' -a 'object_number=113, tag_tid=21, entry_tid=1' = 'object_number=113, tag_tid=21, entry_tid=1' 2024-09-11T06:26:49.566 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:26:49.566 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_snap_present cluster1 mirror test snap7 2024-09-11T06:26:49.566 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:26:49.566 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:26:49.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:26:49.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap7 2024-09-11T06:26:49.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:26:49.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T06:26:49.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:26:50.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 info mirror/test@snap7 2024-09-11T06:26:50.638 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:rbd image 'test': 2024-09-11T06:26:50.638 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: size 512 MiB in 128 objects 2024-09-11T06:26:50.638 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: order 22 (4 MiB objects) 2024-09-11T06:26:50.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: snapshot_count: 13 2024-09-11T06:26:50.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: id: 10a618cfe337 2024-09-11T06:26:50.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: block_name_prefix: rbd_data.10a618cfe337 2024-09-11T06:26:50.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: format: 2 2024-09-11T06:26:50.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: features: layering, exclusive-lock, journaling 2024-09-11T06:26:50.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: op_features: 2024-09-11T06:26:50.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: flags: 2024-09-11T06:26:50.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: create_timestamp: Wed Sep 11 06:16:46 2024 2024-09-11T06:26:50.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: access_timestamp: Wed Sep 11 06:24:53 2024 2024-09-11T06:26:50.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: modify_timestamp: Wed Sep 11 06:26:00 2024 2024-09-11T06:26:50.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: protected: False 2024-09-11T06:26:50.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: journal: 10a618cfe337 2024-09-11T06:26:50.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring state: enabled 2024-09-11T06:26:50.640 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring mode: journal 2024-09-11T06:26:50.640 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring global id: a0f7d515-874e-4c67-a26b-4958c7ab3083 2024-09-11T06:26:50.640 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring primary: false 2024-09-11T06:26:50.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:26:50.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n snap6-clean ']' 2024-09-11T06:26:50.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ compare_image_snaps mirror test snap6-clean 2024-09-11T06:26:50.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:26:50.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:26:50.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap6-clean 2024-09-11T06:26:50.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local ret=0 2024-09-11T06:26:50.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local rmt_export=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export 2024-09-11T06:26:50.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loc_export=/tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:26:50.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:26:50.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6-clean /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export 2024-09-11T06:26:51.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T06:26:51.368 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6-clean /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:26:52.972 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T06:26:52.996 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmp /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:26:53.274 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:26:53.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:26:53.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ compare_image_snaps mirror test snap7 2024-09-11T06:26:53.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:26:53.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:26:53.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap7 2024-09-11T06:26:53.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local ret=0 2024-09-11T06:26:53.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local rmt_export=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export 2024-09-11T06:26:53.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loc_export=/tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:26:53.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:26:53.411 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7 /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export 2024-09-11T06:26:54.090 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T06:26:54.103 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7 /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:26:55.792 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T06:26:55.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmp /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:26:56.098 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:26:56.234 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:26:56.234 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ clean_snap_name=snap7-clean 2024-09-11T06:26:56.234 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_snap cluster2 mirror test snap7-clean 2024-09-11T06:26:56.234 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:26:56.234 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:26:56.235 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:26:56.235 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap7-clean 2024-09-11T06:26:56.235 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap7-clean --debug-rbd=20 --debug-journaler=20 2024-09-11T06:26:57.466 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 10` 2024-09-11T06:26:57.467 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ stress_write_image cluster2 mirror test 2024-09-11T06:26:57.467 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:26:57.467 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:26:57.467 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:26:57.467 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T06:26:57.469 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local duration=6 2024-09-11T06:26:57.469 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set +e 2024-09-11T06:26:57.469 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-11T06:26:59.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:100 IOs, 33937408 bytes 2024-09-11T06:27:00.955 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:200 IOs, 45604864 bytes 2024-09-11T06:27:01.843 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:300 IOs, 53929984 bytes 2024-09-11T06:27:02.887 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:400 IOs, 79935488 bytes 2024-09-11T06:27:03.478 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ error_code=124 2024-09-11T06:27:03.478 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set -e 2024-09-11T06:27:03.478 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T06:27:03.478 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:27:03.478 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-11T06:27:03.478 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:27:03.478 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:27:03.478 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:27:03.478 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state_pattern=up+replaying 2024-09-11T06:27:03.478 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local description_pattern= 2024-09-11T06:27:03.478 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local service_pattern= 2024-09-11T06:27:03.479 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:27:03.479 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:27:04.479 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-11T06:27:04.479 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:27:04.479 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:27:04.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:27:04.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state_pattern=up+replaying 2024-09-11T06:27:04.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local description_pattern= 2024-09-11T06:27:04.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local service_pattern= 2024-09-11T06:27:04.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local status 2024-09-11T06:27:04.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ CEPH_ARGS= 2024-09-11T06:27:04.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-11T06:27:04.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ status='test: 2024-09-11T06:27:04.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: global_id: a0f7d515-874e-4c67-a26b-4958c7ab3083 2024-09-11T06:27:04.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: state: up+replaying 2024-09-11T06:27:04.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: description: replaying, {"bytes_per_second":2863272.13,"entries_behind_primary":523,"entries_per_second":177.23,"non_primary_position":{"entry_tid":17671,"object_number":115,"tag_tid":20},"primary_position":{"entry_tid":1,"object_number":113,"tag_tid":21},"seconds_until_synced":2} 2024-09-11T06:27:04.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: service: mirror.0 on smithi112 2024-09-11T06:27:04.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: last_update: 2024-09-11 06:26:44 2024-09-11T06:27:04.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: peer_sites: 2024-09-11T06:27:04.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: name: cluster2 2024-09-11T06:27:04.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: state: up+stopped 2024-09-11T06:27:04.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: description: local image is primary 2024-09-11T06:27:04.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: last_update: 2024-09-11 06:26:59' 2024-09-11T06:27:04.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ grep '^ state: .*up+replaying' 2024-09-11T06:27:04.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: state: up+replaying 2024-09-11T06:27:04.569 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ grep '^ description: .*' 2024-09-11T06:27:04.570 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: description: replaying, {"bytes_per_second":2863272.13,"entries_behind_primary":523,"entries_per_second":177.23,"non_primary_position":{"entry_tid":17671,"object_number":115,"tag_tid":20},"primary_position":{"entry_tid":1,"object_number":113,"tag_tid":21},"seconds_until_synced":2} 2024-09-11T06:27:04.570 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n '' ']' 2024-09-11T06:27:04.570 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ echo up+replaying 2024-09-11T06:27:04.570 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ grep '^up+' 2024-09-11T06:27:04.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:up+replaying 2024-09-11T06:27:04.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ grep 'service: *mirror..* on ' 2024-09-11T06:27:04.572 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: service: mirror.0 on smithi112 2024-09-11T06:27:04.572 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local last_update 2024-09-11T06:27:04.572 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-11T06:27:04.574 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_update='2024-09-11 06:26:44' 2024-09-11T06:27:04.574 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-11 06:26:44' 2024-09-11T06:27:04.574 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:27:04.574 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:27:04.575 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:27:04.575 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state_pattern=up+replaying 2024-09-11T06:27:04.575 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'prev_last_update=2024-09-11 06:26:44' 2024-09-11T06:27:04.575 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local status 2024-09-11T06:27:04.575 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ CEPH_ARGS= 2024-09-11T06:27:04.575 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-11T06:27:04.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ status='OKOKOK142554262mirror.0smithi11219.1.1-298-gb53b6d3etrueOK42594265mirror.1smithi11219.1.1-298-gb53b6d3efalseOKtesta0f7d515-874e-4c67-a26b-4958c7ab3083up+replayingreplaying, {"bytes_per_second":2863272.13,"entries_behind_primary":523,"entries_per_second":177.23,"non_primary_position":{"entry_tid":17671,"object_number":115,"tag_tid":20},"primary_position":{"entry_tid":1,"object_number":113,"tag_tid":21},"seconds_until_synced":2}42554262mirror.0smithi1122024-09-11 06:26:44cluster20bfc6eda-f3c5-424a-8ff7-5c4d3d9fd488up+stoppedlocal image is primary2024-09-11 06:26:59' 2024-09-11T06:27:04.668 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local last_update state 2024-09-11T06:27:04.668 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-11T06:27:04.672 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_update='2024-09-11 06:26:44' 2024-09-11T06:27:04.672 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-11T06:27:04.675 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ state=up+replaying 2024-09-11T06:27:04.676 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ echo up+replaying 2024-09-11T06:27:04.676 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ grep up+replaying 2024-09-11T06:27:04.677 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:up+replaying 2024-09-11T06:27:04.678 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:27:04.678 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:27:04.678 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ snap_name=snap8 2024-09-11T06:27:04.678 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_snap cluster2 mirror test snap8 2024-09-11T06:27:04.678 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:27:04.678 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:27:04.678 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:27:04.678 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap8 2024-09-11T06:27:04.678 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap8 --debug-rbd=20 --debug-journaler=20 2024-09-11T06:27:35.557 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-11T06:27:35.557 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:27:35.557 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:27:35.557 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:27:35.557 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-11T06:27:35.558 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:27:35.558 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:27:35.558 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:27:35.558 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=started 2024-09-11T06:27:35.558 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:27:35.558 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:27:35.558 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:27:36.558 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-11T06:27:36.558 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:27:36.559 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:27:36.559 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:27:36.559 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=started 2024-09-11T06:27:36.559 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local status_result 2024-09-11T06:27:36.559 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=stopped 2024-09-11T06:27:36.560 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-11T06:27:36.560 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_instance=cluster1 2024-09-11T06:27:36.560 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:27:36.560 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:27:36.560 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance=cluster1 2024-09-11T06:27:36.560 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local loop_instance 2024-09-11T06:27:36.560 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:27:36.560 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sleep 0 2024-09-11T06:27:36.560 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -i state 2024-09-11T06:27:36.561 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:27:36.561 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ seq 0 1 2024-09-11T06:27:36.564 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:27:36.564 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-11T06:27:36.564 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1:0 2024-09-11T06:27:36.564 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:27:36.564 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:27:36.564 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:27:36.564 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:27:36.564 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=cluster 2024-09-11T06:27:36.564 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:27:36.564 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:27:36.565 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:27:36.565 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:27:36.565 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval cluster=cluster1 2024-09-11T06:27:36.565 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:27:36.565 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:27:36.565 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:27:36.565 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:27:36.565 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local local_cluster=cluster1:0 2024-09-11T06:27:36.565 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster=cluster1 2024-09-11T06:27:36.566 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance 2024-09-11T06:27:36.566 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:27:36.566 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local val=cluster1:0 2024-09-11T06:27:36.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T06:27:36.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance_var_name=instance 2024-09-11T06:27:36.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:27:36.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:27:36.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T06:27:36.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval local_cluster=cluster1 2024-09-11T06:27:36.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ local_cluster=cluster1 2024-09-11T06:27:36.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval instance=0 2024-09-11T06:27:36.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ instance=0 2024-09-11T06:27:36.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:27:36.586 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:27:36.586 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:27:36.586 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:27:36.586 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-11T06:27:36.707 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ return 0 2024-09-11T06:27:36.708 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T06:27:36.708 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ echo ' "state": "Replaying"' 2024-09-11T06:27:36.708 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ grep -i Replaying 2024-09-11T06:27:36.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: "state": "Replaying" 2024-09-11T06:27:36.712 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_state=started 2024-09-11T06:27:36.712 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test started = started 2024-09-11T06:27:36.712 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:27:36.712 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-11T06:27:36.712 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:27:36.712 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:27:36.712 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:27:36.712 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:27:36.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' journal = journal ']' 2024-09-11T06:27:36.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-09-11T06:27:36.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:27:36.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:27:36.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:27:36.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:27:36.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-11T06:27:36.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-11T06:27:36.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:27:36.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:27:36.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:27:36.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:27:36.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:27:36.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:27:36.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:27:36.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:27:36.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:27:36.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:27:36.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:27:36.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:27:36.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:27:36.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:27:36.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:27:36.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:27:36.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:27:36.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:27:36.915 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:27:36.915 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:27:36.917 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:27:36.917 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:27:36.917 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:27:36.917 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:27:36.918 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:27:36.918 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:27:36.918 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:27:36.918 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:27:36.918 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:27:36.918 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:27:36.918 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:27:36.918 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:27:36.918 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:27:36.918 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:27:36.918 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:27:36.918 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:27:36.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:27:36.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:27:36.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:27:36.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:27:36.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:27:36.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:27:36.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:27:36.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:27:36.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:27:36.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:27:36.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:27:36.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:27:36.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:27:36.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:27:36.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:27:36.921 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:27:36.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:27:36.946 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:27:36.946 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:27:36.946 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:27:37.078 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:27:37.079 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:27:37.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:27:37.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:27:37.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:27:37.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:27:37.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:27:37.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:27:37.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:27:37.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:27:37.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:27:37.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:27:37.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:27:37.082 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:27:37.082 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:27:37.082 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:27:37.129 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 30 2024-09-11T06:27:37.129 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 30 2024-09-11T06:27:37.129 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:27:37.129 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=24, entry_tid=1], [object_number=120, tag_tid=24, entry_tid=0], [object_number=122, tag_tid=23, entry_tid=5970], [object_number=123, tag_tid=23, entry_tid=5967]]], state=connected] 2024-09-11T06:27:37.129 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=123, tag_tid=23, entry_tid=5867], [object_number=122, tag_tid=23, entry_tid=5866], [object_number=121, tag_tid=23, entry_tid=5865], [object_number=120, tag_tid=23, entry_tid=5864]]], state=connected] 2024-09-11T06:27:37.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:27:37.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=121, tag_tid=24, entry_tid=1' 2024-09-11T06:27:37.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:27:37.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:27:37.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:27:37.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:27:37.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:27:37.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:27:37.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:27:37.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:27:37.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:27:37.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:27:37.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:27:37.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:27:37.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:27:37.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:27:37.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:27:37.183 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 30 2024-09-11T06:27:37.184 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 30 2024-09-11T06:27:37.184 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:27:37.184 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=24, entry_tid=1], [object_number=120, tag_tid=24, entry_tid=0], [object_number=122, tag_tid=23, entry_tid=5970], [object_number=123, tag_tid=23, entry_tid=5967]]], state=connected] 2024-09-11T06:27:37.184 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=123, tag_tid=23, entry_tid=5867], [object_number=122, tag_tid=23, entry_tid=5866], [object_number=121, tag_tid=23, entry_tid=5865], [object_number=120, tag_tid=23, entry_tid=5864]]], state=connected] 2024-09-11T06:27:37.186 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:27:37.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=123, tag_tid=23, entry_tid=5867' 2024-09-11T06:27:37.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=121, tag_tid=24, entry_tid=1' -a 'object_number=121, tag_tid=24, entry_tid=1' = 'object_number=123, tag_tid=23, entry_tid=5867' 2024-09-11T06:27:37.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=123, tag_tid=23, entry_tid=5867' '!=' '' 2024-09-11T06:27:37.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:27:37.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=123, tag_tid=23, entry_tid=5867' = '' 2024-09-11T06:27:37.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=123, tag_tid=23, entry_tid=5867' 2024-09-11T06:27:37.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=121, tag_tid=24, entry_tid=1' 2024-09-11T06:27:37.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:27:37.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:27:37.190 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=24 2024-09-11T06:27:37.190 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=123, tag_tid=23, entry_tid=5867' 2024-09-11T06:27:37.190 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:27:37.190 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:27:37.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=23 2024-09-11T06:27:37.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=121, tag_tid=24, entry_tid=1' 2024-09-11T06:27:37.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:27:37.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:27:37.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:27:37.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=123, tag_tid=23, entry_tid=5867' 2024-09-11T06:27:37.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:27:37.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:27:37.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=5867 2024-09-11T06:27:37.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 24 = 23 -a 1 -le 5867 2024-09-11T06:27:37.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:27:37.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:27:37.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:27:37.397 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:27:37.397 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:27:37.398 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:27:37.398 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:27:37.398 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:27:37.398 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:27:37.398 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:27:37.398 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:27:37.398 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:27:37.398 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:27:37.398 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:27:37.398 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:27:37.398 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:27:37.398 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:27:37.399 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:27:37.399 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:27:37.399 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:27:37.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:27:37.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:27:37.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:27:37.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:27:37.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:27:37.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:27:37.403 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:27:37.403 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:27:37.403 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:27:37.403 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:27:37.403 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:27:37.403 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:27:37.403 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:27:37.403 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:27:37.403 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:27:37.403 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:27:37.403 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:27:37.403 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:27:37.404 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:27:37.404 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:27:37.404 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:27:37.404 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:27:37.404 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:27:37.404 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:27:37.404 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:27:37.404 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:27:37.404 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:27:37.404 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:27:37.404 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:27:37.404 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:27:37.405 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:27:37.405 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:27:37.429 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:27:37.429 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:27:37.429 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:27:37.430 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:27:37.570 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:27:37.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:27:37.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:27:37.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:27:37.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:27:37.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:27:37.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:27:37.572 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:27:37.572 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:27:37.572 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:27:37.572 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:27:37.572 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:27:37.573 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:27:37.575 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:27:37.575 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:27:37.575 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:27:37.624 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 30 2024-09-11T06:27:37.625 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 30 2024-09-11T06:27:37.625 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:27:37.625 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=24, entry_tid=1], [object_number=120, tag_tid=24, entry_tid=0], [object_number=122, tag_tid=23, entry_tid=5970], [object_number=123, tag_tid=23, entry_tid=5967]]], state=connected] 2024-09-11T06:27:37.625 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=121, tag_tid=23, entry_tid=5949], [object_number=120, tag_tid=23, entry_tid=5948], [object_number=123, tag_tid=23, entry_tid=5947], [object_number=122, tag_tid=23, entry_tid=5946]]], state=connected] 2024-09-11T06:27:37.628 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:27:37.629 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=121, tag_tid=24, entry_tid=1' 2024-09-11T06:27:37.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:27:37.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:27:37.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:27:37.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:27:37.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:27:37.631 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:27:37.631 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:27:37.631 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:27:37.631 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:27:37.631 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:27:37.631 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:27:37.631 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:27:37.632 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:27:37.632 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:27:37.632 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:27:37.679 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 30 2024-09-11T06:27:37.680 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 30 2024-09-11T06:27:37.680 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:27:37.680 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=24, entry_tid=1], [object_number=120, tag_tid=24, entry_tid=0], [object_number=122, tag_tid=23, entry_tid=5970], [object_number=123, tag_tid=23, entry_tid=5967]]], state=connected] 2024-09-11T06:27:37.680 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=121, tag_tid=23, entry_tid=5949], [object_number=120, tag_tid=23, entry_tid=5948], [object_number=123, tag_tid=23, entry_tid=5947], [object_number=122, tag_tid=23, entry_tid=5946]]], state=connected] 2024-09-11T06:27:37.683 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:27:37.685 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=121, tag_tid=23, entry_tid=5949' 2024-09-11T06:27:37.685 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=121, tag_tid=24, entry_tid=1' -a 'object_number=121, tag_tid=24, entry_tid=1' = 'object_number=121, tag_tid=23, entry_tid=5949' 2024-09-11T06:27:37.685 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=121, tag_tid=23, entry_tid=5949' '!=' 'object_number=123, tag_tid=23, entry_tid=5867' 2024-09-11T06:27:37.685 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:27:37.686 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=121, tag_tid=23, entry_tid=5949' = 'object_number=123, tag_tid=23, entry_tid=5867' 2024-09-11T06:27:37.686 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=121, tag_tid=23, entry_tid=5949' 2024-09-11T06:27:37.686 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=121, tag_tid=24, entry_tid=1' 2024-09-11T06:27:37.686 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:27:37.686 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:27:37.686 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=24 2024-09-11T06:27:37.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=121, tag_tid=23, entry_tid=5949' 2024-09-11T06:27:37.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:27:37.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:27:37.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=23 2024-09-11T06:27:37.689 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=121, tag_tid=24, entry_tid=1' 2024-09-11T06:27:37.689 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:27:37.689 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:27:37.690 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:27:37.690 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=121, tag_tid=23, entry_tid=5949' 2024-09-11T06:27:37.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:27:37.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:27:37.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=5949 2024-09-11T06:27:37.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 24 = 23 -a 1 -le 5949 2024-09-11T06:27:37.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:27:37.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:27:37.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:27:37.893 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:27:37.894 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:27:37.894 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:27:37.894 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:27:37.894 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:27:37.894 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:27:37.894 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:27:37.894 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:27:37.894 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:27:37.894 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:27:37.894 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:27:37.894 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:27:37.894 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:27:37.894 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:27:37.894 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:27:37.895 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:27:37.895 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:27:37.899 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:27:37.899 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:27:37.899 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:27:37.899 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:27:37.899 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:27:37.899 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:27:37.899 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:27:37.899 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:27:37.899 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:27:37.899 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:27:37.899 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:27:37.900 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:27:37.900 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:27:37.900 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:27:37.900 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:27:37.900 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:27:37.900 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:27:37.900 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:27:37.900 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:27:37.900 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:27:37.900 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:27:37.900 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:27:37.900 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:27:37.901 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:27:37.901 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:27:37.901 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:27:37.901 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:27:37.901 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:27:37.901 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:27:37.901 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:27:37.901 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:27:37.901 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:27:37.925 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:27:37.925 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:27:37.926 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:27:37.926 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:27:38.060 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:27:38.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:27:38.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:27:38.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:27:38.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:27:38.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:27:38.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:27:38.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:27:38.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:27:38.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:27:38.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:27:38.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:27:38.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:27:38.065 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:27:38.065 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:27:38.065 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:27:38.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 30 2024-09-11T06:27:38.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 30 2024-09-11T06:27:38.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:27:38.120 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=24, entry_tid=1], [object_number=120, tag_tid=24, entry_tid=0], [object_number=122, tag_tid=23, entry_tid=5970], [object_number=123, tag_tid=23, entry_tid=5967]]], state=connected] 2024-09-11T06:27:38.120 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=122, tag_tid=23, entry_tid=5970], [object_number=121, tag_tid=23, entry_tid=5969], [object_number=120, tag_tid=23, entry_tid=5968], [object_number=123, tag_tid=23, entry_tid=5967]]], state=connected] 2024-09-11T06:27:38.123 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:27:38.124 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=121, tag_tid=24, entry_tid=1' 2024-09-11T06:27:38.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:27:38.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:27:38.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:27:38.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:27:38.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:27:38.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:27:38.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:27:38.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:27:38.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:27:38.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:27:38.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:27:38.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:27:38.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:27:38.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:27:38.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:27:38.175 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 30 2024-09-11T06:27:38.175 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 30 2024-09-11T06:27:38.175 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:27:38.175 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=24, entry_tid=1], [object_number=120, tag_tid=24, entry_tid=0], [object_number=122, tag_tid=23, entry_tid=5970], [object_number=123, tag_tid=23, entry_tid=5967]]], state=connected] 2024-09-11T06:27:38.175 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=122, tag_tid=23, entry_tid=5970], [object_number=121, tag_tid=23, entry_tid=5969], [object_number=120, tag_tid=23, entry_tid=5968], [object_number=123, tag_tid=23, entry_tid=5967]]], state=connected] 2024-09-11T06:27:38.178 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:27:38.181 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=122, tag_tid=23, entry_tid=5970' 2024-09-11T06:27:38.181 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=121, tag_tid=24, entry_tid=1' -a 'object_number=121, tag_tid=24, entry_tid=1' = 'object_number=122, tag_tid=23, entry_tid=5970' 2024-09-11T06:27:38.181 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=122, tag_tid=23, entry_tid=5970' '!=' 'object_number=121, tag_tid=23, entry_tid=5949' 2024-09-11T06:27:38.181 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:27:38.181 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=122, tag_tid=23, entry_tid=5970' = 'object_number=121, tag_tid=23, entry_tid=5949' 2024-09-11T06:27:38.181 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=122, tag_tid=23, entry_tid=5970' 2024-09-11T06:27:38.181 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=121, tag_tid=24, entry_tid=1' 2024-09-11T06:27:38.181 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:27:38.181 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:27:38.182 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=24 2024-09-11T06:27:38.182 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=122, tag_tid=23, entry_tid=5970' 2024-09-11T06:27:38.182 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:27:38.183 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:27:38.183 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=23 2024-09-11T06:27:38.184 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=121, tag_tid=24, entry_tid=1' 2024-09-11T06:27:38.184 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:27:38.185 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:27:38.185 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:27:38.186 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=122, tag_tid=23, entry_tid=5970' 2024-09-11T06:27:38.186 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:27:38.186 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:27:38.187 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=5970 2024-09-11T06:27:38.187 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 24 = 23 -a 1 -le 5970 2024-09-11T06:27:38.187 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:27:38.187 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:27:38.187 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:27:38.389 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:27:38.390 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:27:38.390 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:27:38.390 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:27:38.390 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:27:38.390 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:27:38.390 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:27:38.390 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:27:38.390 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:27:38.390 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:27:38.390 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:27:38.390 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:27:38.390 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:27:38.391 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:27:38.391 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:27:38.391 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:27:38.391 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:27:38.394 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:27:38.395 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:27:38.395 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:27:38.395 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:27:38.395 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:27:38.395 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:27:38.395 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:27:38.395 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:27:38.395 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:27:38.395 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:27:38.395 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:27:38.395 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:27:38.395 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:27:38.395 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:27:38.395 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:27:38.396 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:27:38.396 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:27:38.396 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:27:38.396 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:27:38.396 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:27:38.396 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:27:38.396 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:27:38.396 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:27:38.396 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:27:38.396 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:27:38.396 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:27:38.396 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:27:38.396 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:27:38.396 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:27:38.397 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:27:38.397 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:27:38.397 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:27:38.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:27:38.420 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:27:38.420 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:27:38.420 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:27:38.541 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:27:38.543 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:27:38.543 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:27:38.543 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:27:38.543 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:27:38.543 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:27:38.543 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:27:38.543 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:27:38.543 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:27:38.544 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:27:38.544 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:27:38.544 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:27:38.544 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:27:38.545 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:27:38.546 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:27:38.546 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:27:38.598 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 30 2024-09-11T06:27:38.598 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 30 2024-09-11T06:27:38.598 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:27:38.598 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=24, entry_tid=1], [object_number=120, tag_tid=24, entry_tid=0], [object_number=122, tag_tid=23, entry_tid=5970], [object_number=123, tag_tid=23, entry_tid=5967]]], state=connected] 2024-09-11T06:27:38.599 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=122, tag_tid=23, entry_tid=5970], [object_number=121, tag_tid=23, entry_tid=5969], [object_number=120, tag_tid=23, entry_tid=5968], [object_number=123, tag_tid=23, entry_tid=5967]]], state=connected] 2024-09-11T06:27:38.602 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:27:38.603 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=121, tag_tid=24, entry_tid=1' 2024-09-11T06:27:38.605 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:27:38.605 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:27:38.605 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:27:38.605 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:27:38.605 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:27:38.605 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:27:38.605 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:27:38.605 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:27:38.605 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:27:38.605 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:27:38.606 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:27:38.606 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:27:38.606 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:27:38.606 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:27:38.606 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:27:38.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 30 2024-09-11T06:27:38.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 30 2024-09-11T06:27:38.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:27:38.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=24, entry_tid=1], [object_number=120, tag_tid=24, entry_tid=0], [object_number=122, tag_tid=23, entry_tid=5970], [object_number=123, tag_tid=23, entry_tid=5967]]], state=connected] 2024-09-11T06:27:38.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=122, tag_tid=23, entry_tid=5970], [object_number=121, tag_tid=23, entry_tid=5969], [object_number=120, tag_tid=23, entry_tid=5968], [object_number=123, tag_tid=23, entry_tid=5967]]], state=connected] 2024-09-11T06:27:38.656 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:27:38.658 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=122, tag_tid=23, entry_tid=5970' 2024-09-11T06:27:38.658 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=121, tag_tid=24, entry_tid=1' -a 'object_number=121, tag_tid=24, entry_tid=1' = 'object_number=122, tag_tid=23, entry_tid=5970' 2024-09-11T06:27:38.658 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=122, tag_tid=23, entry_tid=5970' '!=' 'object_number=122, tag_tid=23, entry_tid=5970' 2024-09-11T06:27:38.658 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:27:38.658 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.4 2024-09-11T06:27:39.060 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:27:39.060 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:27:39.060 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:27:39.060 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:27:39.060 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:27:39.060 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:27:39.060 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:27:39.060 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:27:39.060 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:27:39.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:27:39.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:27:39.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:27:39.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:27:39.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:27:39.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:27:39.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:27:39.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:27:39.063 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:27:39.064 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:27:39.064 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:27:39.064 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:27:39.064 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:27:39.064 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:27:39.064 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:27:39.064 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:27:39.064 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:27:39.064 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:27:39.064 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:27:39.064 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:27:39.064 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:27:39.065 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:27:39.065 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:27:39.065 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:27:39.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:27:39.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:27:39.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:27:39.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:27:39.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:27:39.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:27:39.067 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:27:39.067 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:27:39.067 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:27:39.067 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:27:39.067 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:27:39.067 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:27:39.067 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:27:39.067 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:27:39.067 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:27:39.067 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:27:39.090 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:27:39.091 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:27:39.091 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:27:39.091 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:27:39.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:27:39.213 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:27:39.213 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:27:39.213 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:27:39.214 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:27:39.214 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:27:39.214 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:27:39.214 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:27:39.214 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:27:39.214 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:27:39.214 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:27:39.215 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:27:39.215 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:27:39.215 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:27:39.216 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:27:39.216 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:27:39.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 30 2024-09-11T06:27:39.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 30 2024-09-11T06:27:39.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:27:39.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=24, entry_tid=1], [object_number=120, tag_tid=24, entry_tid=0], [object_number=122, tag_tid=23, entry_tid=5970], [object_number=123, tag_tid=23, entry_tid=5967]]], state=connected] 2024-09-11T06:27:39.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=122, tag_tid=23, entry_tid=5970], [object_number=121, tag_tid=23, entry_tid=5969], [object_number=120, tag_tid=23, entry_tid=5968], [object_number=123, tag_tid=23, entry_tid=5967]]], state=connected] 2024-09-11T06:27:39.271 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:27:39.273 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=121, tag_tid=24, entry_tid=1' 2024-09-11T06:27:39.274 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:27:39.274 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:27:39.274 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:27:39.274 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:27:39.274 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:27:39.274 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:27:39.274 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:27:39.274 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:27:39.274 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:27:39.274 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:27:39.275 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:27:39.275 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:27:39.275 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:27:39.275 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:27:39.275 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:27:39.323 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 30 2024-09-11T06:27:39.323 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 30 2024-09-11T06:27:39.323 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:27:39.323 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=24, entry_tid=1], [object_number=120, tag_tid=24, entry_tid=0], [object_number=122, tag_tid=23, entry_tid=5970], [object_number=123, tag_tid=23, entry_tid=5967]]], state=connected] 2024-09-11T06:27:39.323 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=122, tag_tid=23, entry_tid=5970], [object_number=121, tag_tid=23, entry_tid=5969], [object_number=120, tag_tid=23, entry_tid=5968], [object_number=123, tag_tid=23, entry_tid=5967]]], state=connected] 2024-09-11T06:27:39.326 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:27:39.328 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=122, tag_tid=23, entry_tid=5970' 2024-09-11T06:27:39.328 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=121, tag_tid=24, entry_tid=1' -a 'object_number=121, tag_tid=24, entry_tid=1' = 'object_number=122, tag_tid=23, entry_tid=5970' 2024-09-11T06:27:39.328 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=122, tag_tid=23, entry_tid=5970' '!=' 'object_number=122, tag_tid=23, entry_tid=5970' 2024-09-11T06:27:39.328 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:27:39.328 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.8 2024-09-11T06:27:40.130 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:27:40.130 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:27:40.130 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:27:40.130 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:27:40.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:27:40.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:27:40.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:27:40.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:27:40.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:27:40.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:27:40.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:27:40.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:27:40.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:27:40.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:27:40.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:27:40.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:27:40.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:27:40.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:27:40.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:27:40.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:27:40.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:27:40.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:27:40.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:27:40.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:27:40.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:27:40.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:27:40.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:27:40.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:27:40.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:27:40.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:27:40.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:27:40.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:27:40.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:27:40.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:27:40.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:27:40.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:27:40.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:27:40.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:27:40.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:27:40.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:27:40.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:27:40.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:27:40.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:27:40.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:27:40.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:27:40.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:27:40.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:27:40.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:27:40.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:27:40.160 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:27:40.160 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:27:40.160 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:27:40.160 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:27:40.295 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:27:40.296 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:27:40.296 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:27:40.296 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:27:40.296 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:27:40.296 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:27:40.296 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:27:40.297 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:27:40.297 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:27:40.297 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:27:40.297 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:27:40.297 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:27:40.297 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:27:40.298 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:27:40.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:27:40.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:27:40.354 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 30 2024-09-11T06:27:40.354 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 30 2024-09-11T06:27:40.354 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:27:40.354 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=24, entry_tid=1], [object_number=120, tag_tid=24, entry_tid=0], [object_number=122, tag_tid=23, entry_tid=5970], [object_number=123, tag_tid=23, entry_tid=5967]]], state=connected] 2024-09-11T06:27:40.354 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=121, tag_tid=24, entry_tid=1], [object_number=120, tag_tid=24, entry_tid=0], [object_number=122, tag_tid=23, entry_tid=5970], [object_number=123, tag_tid=23, entry_tid=5967]]], state=connected] 2024-09-11T06:27:40.357 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:27:40.359 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=121, tag_tid=24, entry_tid=1' 2024-09-11T06:27:40.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:27:40.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:27:40.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:27:40.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:27:40.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:27:40.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:27:40.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:27:40.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:27:40.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:27:40.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:27:40.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:27:40.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:27:40.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:27:40.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:27:40.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:27:40.411 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 30 2024-09-11T06:27:40.411 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 30 2024-09-11T06:27:40.412 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:27:40.412 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=121, tag_tid=24, entry_tid=1], [object_number=120, tag_tid=24, entry_tid=0], [object_number=122, tag_tid=23, entry_tid=5970], [object_number=123, tag_tid=23, entry_tid=5967]]], state=connected] 2024-09-11T06:27:40.412 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=121, tag_tid=24, entry_tid=1], [object_number=120, tag_tid=24, entry_tid=0], [object_number=122, tag_tid=23, entry_tid=5970], [object_number=123, tag_tid=23, entry_tid=5967]]], state=connected] 2024-09-11T06:27:40.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:27:40.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=121, tag_tid=24, entry_tid=1' 2024-09-11T06:27:40.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=121, tag_tid=24, entry_tid=1' -a 'object_number=121, tag_tid=24, entry_tid=1' = 'object_number=121, tag_tid=24, entry_tid=1' 2024-09-11T06:27:40.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:27:40.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_snap_present cluster1 mirror test snap8 2024-09-11T06:27:40.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:27:40.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:27:40.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:27:40.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap8 2024-09-11T06:27:40.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:27:40.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T06:27:40.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:27:41.418 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 info mirror/test@snap8 2024-09-11T06:27:41.493 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:rbd image 'test': 2024-09-11T06:27:41.493 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: size 512 MiB in 128 objects 2024-09-11T06:27:41.493 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: order 22 (4 MiB objects) 2024-09-11T06:27:41.493 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: snapshot_count: 15 2024-09-11T06:27:41.493 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: id: 10a618cfe337 2024-09-11T06:27:41.493 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: block_name_prefix: rbd_data.10a618cfe337 2024-09-11T06:27:41.493 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: format: 2 2024-09-11T06:27:41.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: features: layering, exclusive-lock, journaling 2024-09-11T06:27:41.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: op_features: 2024-09-11T06:27:41.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: flags: 2024-09-11T06:27:41.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: create_timestamp: Wed Sep 11 06:16:46 2024 2024-09-11T06:27:41.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: access_timestamp: Wed Sep 11 06:26:51 2024 2024-09-11T06:27:41.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: modify_timestamp: Wed Sep 11 06:27:01 2024 2024-09-11T06:27:41.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: protected: False 2024-09-11T06:27:41.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: journal: 10a618cfe337 2024-09-11T06:27:41.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring state: enabled 2024-09-11T06:27:41.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring mode: journal 2024-09-11T06:27:41.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring global id: a0f7d515-874e-4c67-a26b-4958c7ab3083 2024-09-11T06:27:41.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring primary: false 2024-09-11T06:27:41.499 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:27:41.499 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n snap7-clean ']' 2024-09-11T06:27:41.499 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ compare_image_snaps mirror test snap7-clean 2024-09-11T06:27:41.499 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:27:41.499 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:27:41.499 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap7-clean 2024-09-11T06:27:41.499 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local ret=0 2024-09-11T06:27:41.499 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local rmt_export=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export 2024-09-11T06:27:41.499 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loc_export=/tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:27:41.499 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:27:41.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7-clean /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export 2024-09-11T06:27:42.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T06:27:42.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7-clean /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:27:43.818 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T06:27:43.843 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmp /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:27:44.122 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:27:44.260 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:27:44.260 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ compare_image_snaps mirror test snap8 2024-09-11T06:27:44.260 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:27:44.260 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:27:44.260 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap8 2024-09-11T06:27:44.261 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local ret=0 2024-09-11T06:27:44.261 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local rmt_export=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export 2024-09-11T06:27:44.261 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loc_export=/tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:27:44.261 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:27:44.261 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8 /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export 2024-09-11T06:27:44.949 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T06:27:44.960 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8 /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:27:46.607 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T06:27:46.628 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmp /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:27:46.908 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:27:47.045 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:27:47.046 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ clean_snap_name=snap8-clean 2024-09-11T06:27:47.046 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_snap cluster2 mirror test snap8-clean 2024-09-11T06:27:47.046 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:27:47.046 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:27:47.046 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:27:47.046 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap8-clean 2024-09-11T06:27:47.046 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap8-clean --debug-rbd=20 --debug-journaler=20 2024-09-11T06:27:47.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 10` 2024-09-11T06:27:47.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ stress_write_image cluster2 mirror test 2024-09-11T06:27:47.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:27:47.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:27:47.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:27:47.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T06:27:47.570 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local duration=5 2024-09-11T06:27:47.570 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set +e 2024-09-11T06:27:47.570 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-11T06:27:49.073 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:100 IOs, 32324608 bytes 2024-09-11T06:27:50.130 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:200 IOs, 53040128 bytes 2024-09-11T06:27:50.756 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:300 IOs, 57066496 bytes 2024-09-11T06:27:52.381 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:400 IOs, 96841728 bytes 2024-09-11T06:27:52.581 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ error_code=124 2024-09-11T06:27:52.581 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set -e 2024-09-11T06:27:52.581 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T06:27:52.581 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:27:52.581 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-11T06:27:52.581 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:27:52.581 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:27:52.581 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:27:52.582 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state_pattern=up+replaying 2024-09-11T06:27:52.582 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local description_pattern= 2024-09-11T06:27:52.582 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local service_pattern= 2024-09-11T06:27:52.582 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:27:52.582 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:27:53.585 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-11T06:27:53.585 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:27:53.585 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:27:53.585 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:27:53.585 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state_pattern=up+replaying 2024-09-11T06:27:53.585 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local description_pattern= 2024-09-11T06:27:53.585 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local service_pattern= 2024-09-11T06:27:53.585 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local status 2024-09-11T06:27:53.586 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ CEPH_ARGS= 2024-09-11T06:27:53.586 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-11T06:27:53.679 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ status='test: 2024-09-11T06:27:53.679 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: global_id: a0f7d515-874e-4c67-a26b-4958c7ab3083 2024-09-11T06:27:53.680 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: state: up+replaying 2024-09-11T06:27:53.680 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: description: replaying, {"bytes_per_second":2136976.37,"entries_behind_primary":0,"entries_per_second":139.03,"non_primary_position":{"entry_tid":1,"object_number":121,"tag_tid":24},"primary_position":{"entry_tid":1,"object_number":121,"tag_tid":24}} 2024-09-11T06:27:53.680 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: service: mirror.0 on smithi112 2024-09-11T06:27:53.680 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: last_update: 2024-09-11 06:27:44 2024-09-11T06:27:53.680 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: peer_sites: 2024-09-11T06:27:53.680 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: name: cluster2 2024-09-11T06:27:53.680 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: state: up+stopped 2024-09-11T06:27:53.680 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: description: local image is primary 2024-09-11T06:27:53.680 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: last_update: 2024-09-11 06:27:49' 2024-09-11T06:27:53.680 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ grep '^ state: .*up+replaying' 2024-09-11T06:27:53.681 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: state: up+replaying 2024-09-11T06:27:53.682 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ grep '^ description: .*' 2024-09-11T06:27:53.683 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: description: replaying, {"bytes_per_second":2136976.37,"entries_behind_primary":0,"entries_per_second":139.03,"non_primary_position":{"entry_tid":1,"object_number":121,"tag_tid":24},"primary_position":{"entry_tid":1,"object_number":121,"tag_tid":24}} 2024-09-11T06:27:53.683 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n '' ']' 2024-09-11T06:27:53.683 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ echo up+replaying 2024-09-11T06:27:53.683 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ grep '^up+' 2024-09-11T06:27:53.684 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:up+replaying 2024-09-11T06:27:53.684 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ grep 'service: *mirror..* on ' 2024-09-11T06:27:53.685 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: service: mirror.0 on smithi112 2024-09-11T06:27:53.686 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local last_update 2024-09-11T06:27:53.686 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-11T06:27:53.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_update='2024-09-11 06:27:44' 2024-09-11T06:27:53.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-11 06:27:44' 2024-09-11T06:27:53.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:27:53.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:27:53.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:27:53.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state_pattern=up+replaying 2024-09-11T06:27:53.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'prev_last_update=2024-09-11 06:27:44' 2024-09-11T06:27:53.689 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local status 2024-09-11T06:27:53.689 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ CEPH_ARGS= 2024-09-11T06:27:53.689 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-11T06:27:53.787 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ status='OKOKOK142554262mirror.0smithi11219.1.1-298-gb53b6d3etrueOK42594265mirror.1smithi11219.1.1-298-gb53b6d3efalseOKtesta0f7d515-874e-4c67-a26b-4958c7ab3083up+replayingreplaying, {"bytes_per_second":2136976.37,"entries_behind_primary":0,"entries_per_second":139.03,"non_primary_position":{"entry_tid":1,"object_number":121,"tag_tid":24},"primary_position":{"entry_tid":1,"object_number":121,"tag_tid":24}}42554262mirror.0smithi1122024-09-11 06:27:44cluster20bfc6eda-f3c5-424a-8ff7-5c4d3d9fd488up+stoppedlocal image is primary2024-09-11 06:27:49' 2024-09-11T06:27:53.787 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local last_update state 2024-09-11T06:27:53.787 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-11T06:27:53.791 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_update='2024-09-11 06:27:44' 2024-09-11T06:27:53.792 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-11T06:27:53.795 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ state=up+replaying 2024-09-11T06:27:53.796 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ echo up+replaying 2024-09-11T06:27:53.796 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ grep up+replaying 2024-09-11T06:27:53.797 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:up+replaying 2024-09-11T06:27:53.798 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:27:53.798 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:27:53.798 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ snap_name=snap9 2024-09-11T06:27:53.798 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_snap cluster2 mirror test snap9 2024-09-11T06:27:53.798 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:27:53.798 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:27:53.798 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:27:53.799 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap9 2024-09-11T06:27:53.799 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap9 --debug-rbd=20 --debug-journaler=20 2024-09-11T06:28:03.578 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:28:03.576+0000 7fb2ca654640 -1 reset not still connected to 0x55ed0d8cd5f0 2024-09-11T06:28:03.578 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:28:03.576+0000 7fb2ca654640 -1 reset not still connected to 0x55ed0e550dd0 2024-09-11T06:28:06.201 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:28:06.200+0000 7fb45318f640 -1 reset not still connected to 0x561315884d00 2024-09-11T06:28:06.204 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:28:06.200+0000 7fb45318f640 -1 reset not still connected to 0x561315ba4270 2024-09-11T06:28:06.205 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:28:06.200+0000 7fb45318f640 -1 reset not still connected to 0x561315ba5520 2024-09-11T06:28:10.519 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:28:10.516+0000 7fc41084f640 -1 reset not still connected to 0x55eb15565d40 2024-09-11T06:28:10.519 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:28:10.516+0000 7fc41084f640 -1 reset not still connected to 0x55eb15806dd0 2024-09-11T06:28:10.520 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:28:10.516+0000 7fc41084f640 -1 reset not still connected to 0x55eb16561450 2024-09-11T06:28:25.390 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-11T06:28:25.390 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:28:25.390 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:28:25.390 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:28:25.390 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-11T06:28:25.390 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:28:25.390 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:28:25.390 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:28:25.391 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=started 2024-09-11T06:28:25.391 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:28:25.391 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:28:25.391 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:28:26.391 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-11T06:28:26.391 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:28:26.391 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:28:26.391 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:28:26.391 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=started 2024-09-11T06:28:26.391 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local status_result 2024-09-11T06:28:26.391 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=stopped 2024-09-11T06:28:26.392 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-11T06:28:26.392 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -i state 2024-09-11T06:28:26.392 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_instance=cluster1 2024-09-11T06:28:26.393 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:28:26.393 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:28:26.393 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance=cluster1 2024-09-11T06:28:26.393 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local loop_instance 2024-09-11T06:28:26.393 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:28:26.393 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sleep 0 2024-09-11T06:28:26.394 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:28:26.394 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ seq 0 1 2024-09-11T06:28:26.396 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:28:26.397 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-11T06:28:26.397 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1:0 2024-09-11T06:28:26.397 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:28:26.397 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:28:26.397 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:28:26.397 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:28:26.397 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=cluster 2024-09-11T06:28:26.397 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:28:26.397 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:28:26.397 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:28:26.397 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:28:26.398 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval cluster=cluster1 2024-09-11T06:28:26.398 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:28:26.398 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:28:26.398 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:28:26.398 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:28:26.398 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local local_cluster=cluster1:0 2024-09-11T06:28:26.398 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster=cluster1 2024-09-11T06:28:26.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance 2024-09-11T06:28:26.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:28:26.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local val=cluster1:0 2024-09-11T06:28:26.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T06:28:26.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance_var_name=instance 2024-09-11T06:28:26.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:28:26.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:28:26.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T06:28:26.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval local_cluster=cluster1 2024-09-11T06:28:26.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ local_cluster=cluster1 2024-09-11T06:28:26.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval instance=0 2024-09-11T06:28:26.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ instance=0 2024-09-11T06:28:26.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:28:26.424 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:28:26.424 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:28:26.424 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:28:26.425 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-11T06:28:26.544 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ return 0 2024-09-11T06:28:26.545 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T06:28:26.545 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ echo ' "state": "Replaying"' 2024-09-11T06:28:26.546 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ grep -i Replaying 2024-09-11T06:28:26.547 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: "state": "Replaying" 2024-09-11T06:28:26.548 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_state=started 2024-09-11T06:28:26.548 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test started = started 2024-09-11T06:28:26.549 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:28:26.549 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-11T06:28:26.549 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:28:26.549 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:28:26.549 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:28:26.549 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:28:26.549 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' journal = journal ']' 2024-09-11T06:28:26.549 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-09-11T06:28:26.549 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:28:26.549 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:28:26.549 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:28:26.549 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:28:26.550 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-11T06:28:26.550 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-11T06:28:26.550 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:28:26.550 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:28:26.550 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:28:26.750 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:28:26.751 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:28:26.751 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:28:26.751 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:28:26.751 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:28:26.751 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:28:26.751 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:28:26.751 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:28:26.751 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:28:26.751 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:28:26.751 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:28:26.751 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:28:26.751 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:28:26.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:28:26.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:28:26.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:28:26.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:28:26.754 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:28:26.754 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:28:26.754 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:28:26.754 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:28:26.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:28:26.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:28:26.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:28:26.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:28:26.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:28:26.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:28:26.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:28:26.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:28:26.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:28:26.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:28:26.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:28:26.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:28:26.756 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:28:26.756 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:28:26.756 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:28:26.757 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:28:26.757 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:28:26.757 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:28:26.757 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:28:26.757 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:28:26.757 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:28:26.757 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:28:26.757 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:28:26.757 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:28:26.757 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:28:26.757 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:28:26.757 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:28:26.757 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:28:26.782 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:28:26.782 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:28:26.783 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:28:26.783 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:28:26.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:28:26.915 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:28:26.916 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:28:26.916 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:28:26.916 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:28:26.916 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:28:26.916 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:28:26.916 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:28:26.916 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:28:26.916 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:28:26.917 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:28:26.917 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:28:26.917 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:28:26.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:28:26.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:28:26.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:28:26.973 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 32 2024-09-11T06:28:26.973 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 32 2024-09-11T06:28:26.973 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:28:26.973 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=129, tag_tid=27, entry_tid=1], [object_number=128, tag_tid=27, entry_tid=0], [object_number=131, tag_tid=26, entry_tid=6295], [object_number=130, tag_tid=26, entry_tid=6294]]], state=connected] 2024-09-11T06:28:26.973 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=128, tag_tid=26, entry_tid=5628], [object_number=131, tag_tid=26, entry_tid=5627], [object_number=130, tag_tid=26, entry_tid=5626], [object_number=129, tag_tid=26, entry_tid=5625]]], state=connected] 2024-09-11T06:28:26.976 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:28:26.979 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=129, tag_tid=27, entry_tid=1' 2024-09-11T06:28:26.979 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:28:26.979 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:28:26.979 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:28:26.979 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:28:26.979 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:28:26.979 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:28:26.980 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:28:26.980 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:28:26.980 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:28:26.980 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:28:26.980 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:28:26.980 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:28:26.980 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:28:26.981 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:28:26.981 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:28:27.029 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 32 2024-09-11T06:28:27.030 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 32 2024-09-11T06:28:27.030 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:28:27.030 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=129, tag_tid=27, entry_tid=1], [object_number=128, tag_tid=27, entry_tid=0], [object_number=131, tag_tid=26, entry_tid=6295], [object_number=130, tag_tid=26, entry_tid=6294]]], state=connected] 2024-09-11T06:28:27.030 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=128, tag_tid=26, entry_tid=5628], [object_number=131, tag_tid=26, entry_tid=5627], [object_number=130, tag_tid=26, entry_tid=5626], [object_number=129, tag_tid=26, entry_tid=5625]]], state=connected] 2024-09-11T06:28:27.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:28:27.035 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=128, tag_tid=26, entry_tid=5628' 2024-09-11T06:28:27.035 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=129, tag_tid=27, entry_tid=1' -a 'object_number=129, tag_tid=27, entry_tid=1' = 'object_number=128, tag_tid=26, entry_tid=5628' 2024-09-11T06:28:27.035 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=128, tag_tid=26, entry_tid=5628' '!=' '' 2024-09-11T06:28:27.035 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:28:27.035 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=128, tag_tid=26, entry_tid=5628' = '' 2024-09-11T06:28:27.036 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=128, tag_tid=26, entry_tid=5628' 2024-09-11T06:28:27.036 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=129, tag_tid=27, entry_tid=1' 2024-09-11T06:28:27.036 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:28:27.036 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:28:27.036 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=27 2024-09-11T06:28:27.037 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=128, tag_tid=26, entry_tid=5628' 2024-09-11T06:28:27.037 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:28:27.037 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:28:27.038 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=26 2024-09-11T06:28:27.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=129, tag_tid=27, entry_tid=1' 2024-09-11T06:28:27.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:28:27.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:28:27.040 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:28:27.040 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=128, tag_tid=26, entry_tid=5628' 2024-09-11T06:28:27.040 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:28:27.040 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:28:27.041 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=5628 2024-09-11T06:28:27.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 27 = 26 -a 1 -le 5628 2024-09-11T06:28:27.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:28:27.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:28:27.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:28:27.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:28:27.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:28:27.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:28:27.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:28:27.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:28:27.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:28:27.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:28:27.244 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:28:27.244 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:28:27.244 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:28:27.244 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:28:27.244 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:28:27.244 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:28:27.244 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:28:27.244 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:28:27.245 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:28:27.245 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:28:27.248 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:28:27.248 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:28:27.248 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:28:27.248 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:28:27.248 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:28:27.248 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:28:27.248 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:28:27.248 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:28:27.248 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:28:27.248 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:28:27.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:28:27.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:28:27.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:28:27.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:28:27.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:28:27.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:28:27.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:28:27.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:28:27.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:28:27.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:28:27.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:28:27.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:28:27.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:28:27.250 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:28:27.250 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:28:27.250 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:28:27.250 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:28:27.250 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:28:27.250 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:28:27.250 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:28:27.250 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:28:27.250 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:28:27.272 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:28:27.273 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:28:27.273 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:28:27.273 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:28:27.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:28:27.404 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:28:27.404 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:28:27.404 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:28:27.404 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:28:27.404 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:28:27.404 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:28:27.404 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:28:27.404 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:28:27.405 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:28:27.405 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:28:27.405 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:28:27.405 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:28:27.406 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:28:27.407 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:28:27.407 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:28:27.460 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 32 2024-09-11T06:28:27.460 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 32 2024-09-11T06:28:27.460 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:28:27.460 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=129, tag_tid=27, entry_tid=1], [object_number=128, tag_tid=27, entry_tid=0], [object_number=131, tag_tid=26, entry_tid=6295], [object_number=130, tag_tid=26, entry_tid=6294]]], state=connected] 2024-09-11T06:28:27.460 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=131, tag_tid=26, entry_tid=5703], [object_number=130, tag_tid=26, entry_tid=5702], [object_number=129, tag_tid=26, entry_tid=5701], [object_number=128, tag_tid=26, entry_tid=5700]]], state=connected] 2024-09-11T06:28:27.463 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:28:27.465 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=129, tag_tid=27, entry_tid=1' 2024-09-11T06:28:27.466 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:28:27.466 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:28:27.466 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:28:27.466 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:28:27.466 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:28:27.466 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:28:27.466 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:28:27.466 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:28:27.467 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:28:27.467 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:28:27.467 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:28:27.467 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:28:27.467 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:28:27.468 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:28:27.468 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:28:27.518 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 32 2024-09-11T06:28:27.519 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 32 2024-09-11T06:28:27.519 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:28:27.519 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=129, tag_tid=27, entry_tid=1], [object_number=128, tag_tid=27, entry_tid=0], [object_number=131, tag_tid=26, entry_tid=6295], [object_number=130, tag_tid=26, entry_tid=6294]]], state=connected] 2024-09-11T06:28:27.519 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=131, tag_tid=26, entry_tid=5703], [object_number=130, tag_tid=26, entry_tid=5702], [object_number=129, tag_tid=26, entry_tid=5701], [object_number=128, tag_tid=26, entry_tid=5700]]], state=connected] 2024-09-11T06:28:27.522 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:28:27.524 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=131, tag_tid=26, entry_tid=5703' 2024-09-11T06:28:27.524 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=129, tag_tid=27, entry_tid=1' -a 'object_number=129, tag_tid=27, entry_tid=1' = 'object_number=131, tag_tid=26, entry_tid=5703' 2024-09-11T06:28:27.524 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=131, tag_tid=26, entry_tid=5703' '!=' 'object_number=128, tag_tid=26, entry_tid=5628' 2024-09-11T06:28:27.524 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:28:27.524 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=131, tag_tid=26, entry_tid=5703' = 'object_number=128, tag_tid=26, entry_tid=5628' 2024-09-11T06:28:27.525 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=131, tag_tid=26, entry_tid=5703' 2024-09-11T06:28:27.525 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=129, tag_tid=27, entry_tid=1' 2024-09-11T06:28:27.525 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:28:27.525 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:28:27.525 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=27 2024-09-11T06:28:27.526 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=131, tag_tid=26, entry_tid=5703' 2024-09-11T06:28:27.526 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:28:27.526 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:28:27.527 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=26 2024-09-11T06:28:27.528 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=129, tag_tid=27, entry_tid=1' 2024-09-11T06:28:27.528 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:28:27.528 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:28:27.528 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:28:27.529 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=131, tag_tid=26, entry_tid=5703' 2024-09-11T06:28:27.529 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:28:27.529 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:28:27.531 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=5703 2024-09-11T06:28:27.531 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 27 = 26 -a 1 -le 5703 2024-09-11T06:28:27.531 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:28:27.531 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:28:27.531 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:28:27.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:28:27.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:28:27.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:28:27.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:28:27.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:28:27.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:28:27.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:28:27.734 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:28:27.734 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:28:27.734 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:28:27.734 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:28:27.734 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:28:27.734 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:28:27.734 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:28:27.734 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:28:27.736 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:28:27.736 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:28:27.737 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:28:27.737 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:28:27.737 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:28:27.737 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:28:27.737 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:28:27.737 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:28:27.737 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:28:27.737 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:28:27.737 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:28:27.737 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:28:27.737 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:28:27.737 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:28:27.737 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:28:27.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:28:27.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:28:27.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:28:27.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:28:27.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:28:27.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:28:27.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:28:27.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:28:27.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:28:27.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:28:27.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:28:27.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:28:27.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:28:27.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:28:27.739 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:28:27.739 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:28:27.739 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:28:27.739 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:28:27.739 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:28:27.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:28:27.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:28:27.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:28:27.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:28:27.900 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:28:27.902 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:28:27.902 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:28:27.902 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:28:27.902 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:28:27.902 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:28:27.902 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:28:27.902 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:28:27.902 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:28:27.902 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:28:27.902 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:28:27.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:28:27.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:28:27.905 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:28:27.905 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:28:27.905 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:28:27.952 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 32 2024-09-11T06:28:27.953 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 32 2024-09-11T06:28:27.953 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:28:27.953 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=129, tag_tid=27, entry_tid=1], [object_number=128, tag_tid=27, entry_tid=0], [object_number=131, tag_tid=26, entry_tid=6295], [object_number=130, tag_tid=26, entry_tid=6294]]], state=connected] 2024-09-11T06:28:27.953 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=128, tag_tid=26, entry_tid=5784], [object_number=131, tag_tid=26, entry_tid=5783], [object_number=130, tag_tid=26, entry_tid=5782], [object_number=129, tag_tid=26, entry_tid=5781]]], state=connected] 2024-09-11T06:28:27.956 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:28:27.958 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=129, tag_tid=27, entry_tid=1' 2024-09-11T06:28:27.959 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:28:27.959 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:28:27.959 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:28:27.959 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:28:27.959 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:28:27.959 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:28:27.959 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:28:27.959 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:28:27.959 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:28:27.959 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:28:27.959 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:28:27.960 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:28:27.960 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:28:27.960 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:28:27.960 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:28:28.008 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 32 2024-09-11T06:28:28.008 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 32 2024-09-11T06:28:28.008 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:28:28.008 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=129, tag_tid=27, entry_tid=1], [object_number=128, tag_tid=27, entry_tid=0], [object_number=131, tag_tid=26, entry_tid=6295], [object_number=130, tag_tid=26, entry_tid=6294]]], state=connected] 2024-09-11T06:28:28.008 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=128, tag_tid=26, entry_tid=5784], [object_number=131, tag_tid=26, entry_tid=5783], [object_number=130, tag_tid=26, entry_tid=5782], [object_number=129, tag_tid=26, entry_tid=5781]]], state=connected] 2024-09-11T06:28:28.011 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:28:28.013 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=128, tag_tid=26, entry_tid=5784' 2024-09-11T06:28:28.014 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=129, tag_tid=27, entry_tid=1' -a 'object_number=129, tag_tid=27, entry_tid=1' = 'object_number=128, tag_tid=26, entry_tid=5784' 2024-09-11T06:28:28.014 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=128, tag_tid=26, entry_tid=5784' '!=' 'object_number=131, tag_tid=26, entry_tid=5703' 2024-09-11T06:28:28.014 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:28:28.014 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=128, tag_tid=26, entry_tid=5784' = 'object_number=131, tag_tid=26, entry_tid=5703' 2024-09-11T06:28:28.014 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=128, tag_tid=26, entry_tid=5784' 2024-09-11T06:28:28.014 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=129, tag_tid=27, entry_tid=1' 2024-09-11T06:28:28.014 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:28:28.014 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:28:28.015 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=27 2024-09-11T06:28:28.015 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=128, tag_tid=26, entry_tid=5784' 2024-09-11T06:28:28.015 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:28:28.016 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:28:28.016 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=26 2024-09-11T06:28:28.017 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=129, tag_tid=27, entry_tid=1' 2024-09-11T06:28:28.017 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:28:28.017 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:28:28.018 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:28:28.019 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=128, tag_tid=26, entry_tid=5784' 2024-09-11T06:28:28.019 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:28:28.019 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:28:28.020 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=5784 2024-09-11T06:28:28.020 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 27 = 26 -a 1 -le 5784 2024-09-11T06:28:28.020 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:28:28.020 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:28:28.020 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:28:28.222 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:28:28.222 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:28:28.222 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:28:28.222 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:28:28.223 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:28:28.223 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:28:28.223 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:28:28.223 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:28:28.223 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:28:28.223 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:28:28.223 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:28:28.223 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:28:28.223 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:28:28.223 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:28:28.223 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:28:28.224 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:28:28.224 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:28:28.227 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:28:28.227 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:28:28.227 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:28:28.227 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:28:28.227 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:28:28.228 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:28:28.228 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:28:28.228 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:28:28.228 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:28:28.228 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:28:28.228 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:28:28.228 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:28:28.228 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:28:28.228 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:28:28.228 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:28:28.228 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:28:28.228 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:28:28.228 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:28:28.229 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:28:28.229 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:28:28.229 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:28:28.229 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:28:28.229 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:28:28.229 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:28:28.229 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:28:28.229 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:28:28.229 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:28:28.229 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:28:28.229 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:28:28.229 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:28:28.229 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:28:28.230 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:28:28.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:28:28.254 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:28:28.254 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:28:28.254 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:28:28.382 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:28:28.384 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:28:28.384 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:28:28.384 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:28:28.384 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:28:28.384 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:28:28.384 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:28:28.384 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:28:28.384 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:28:28.385 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:28:28.385 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:28:28.385 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:28:28.385 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:28:28.386 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:28:28.386 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:28:28.387 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:28:28.439 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 32 2024-09-11T06:28:28.439 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 32 2024-09-11T06:28:28.439 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:28:28.439 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=129, tag_tid=27, entry_tid=1], [object_number=128, tag_tid=27, entry_tid=0], [object_number=131, tag_tid=26, entry_tid=6295], [object_number=130, tag_tid=26, entry_tid=6294]]], state=connected] 2024-09-11T06:28:28.439 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=129, tag_tid=26, entry_tid=5857], [object_number=128, tag_tid=26, entry_tid=5856], [object_number=131, tag_tid=26, entry_tid=5855], [object_number=130, tag_tid=26, entry_tid=5854]]], state=connected] 2024-09-11T06:28:28.442 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:28:28.445 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=129, tag_tid=27, entry_tid=1' 2024-09-11T06:28:28.445 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:28:28.445 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:28:28.445 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:28:28.445 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:28:28.445 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:28:28.446 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:28:28.446 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:28:28.446 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:28:28.446 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:28:28.446 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:28:28.446 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:28:28.446 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:28:28.447 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:28:28.447 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:28:28.447 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:28:28.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 32 2024-09-11T06:28:28.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 32 2024-09-11T06:28:28.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:28:28.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=129, tag_tid=27, entry_tid=1], [object_number=128, tag_tid=27, entry_tid=0], [object_number=131, tag_tid=26, entry_tid=6295], [object_number=130, tag_tid=26, entry_tid=6294]]], state=connected] 2024-09-11T06:28:28.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=129, tag_tid=26, entry_tid=5857], [object_number=128, tag_tid=26, entry_tid=5856], [object_number=131, tag_tid=26, entry_tid=5855], [object_number=130, tag_tid=26, entry_tid=5854]]], state=connected] 2024-09-11T06:28:28.499 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:28:28.502 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=129, tag_tid=26, entry_tid=5857' 2024-09-11T06:28:28.502 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=129, tag_tid=27, entry_tid=1' -a 'object_number=129, tag_tid=27, entry_tid=1' = 'object_number=129, tag_tid=26, entry_tid=5857' 2024-09-11T06:28:28.502 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=129, tag_tid=26, entry_tid=5857' '!=' 'object_number=128, tag_tid=26, entry_tid=5784' 2024-09-11T06:28:28.502 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:28:28.502 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=129, tag_tid=26, entry_tid=5857' = 'object_number=128, tag_tid=26, entry_tid=5784' 2024-09-11T06:28:28.502 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=129, tag_tid=26, entry_tid=5857' 2024-09-11T06:28:28.502 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=129, tag_tid=27, entry_tid=1' 2024-09-11T06:28:28.502 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:28:28.502 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:28:28.503 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=27 2024-09-11T06:28:28.504 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=129, tag_tid=26, entry_tid=5857' 2024-09-11T06:28:28.504 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:28:28.504 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:28:28.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=26 2024-09-11T06:28:28.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=129, tag_tid=27, entry_tid=1' 2024-09-11T06:28:28.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:28:28.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:28:28.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:28:28.507 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=129, tag_tid=26, entry_tid=5857' 2024-09-11T06:28:28.507 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:28:28.507 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:28:28.508 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=5857 2024-09-11T06:28:28.508 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 27 = 26 -a 1 -le 5857 2024-09-11T06:28:28.508 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:28:28.509 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:28:28.509 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:28:28.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:28:28.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:28:28.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:28:28.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:28:28.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:28:28.711 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:28:28.711 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:28:28.711 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:28:28.711 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:28:28.711 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:28:28.711 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:28:28.711 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:28:28.711 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:28:28.711 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:28:28.711 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:28:28.712 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:28:28.712 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:28:28.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:28:28.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:28:28.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:28:28.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:28:28.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:28:28.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:28:28.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:28:28.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:28:28.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:28:28.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:28:28.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:28:28.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:28:28.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:28:28.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:28:28.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:28:28.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:28:28.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:28:28.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:28:28.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:28:28.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:28:28.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:28:28.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:28:28.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:28:28.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:28:28.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:28:28.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:28:28.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:28:28.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:28:28.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:28:28.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:28:28.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:28:28.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:28:28.741 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:28:28.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:28:28.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:28:28.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:28:28.869 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:28:28.870 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:28:28.870 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:28:28.870 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:28:28.870 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:28:28.870 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:28:28.870 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:28:28.870 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:28:28.870 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:28:28.870 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:28:28.871 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:28:28.871 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:28:28.871 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:28:28.872 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:28:28.873 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:28:28.873 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:28:29.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 32 2024-09-11T06:28:29.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 32 2024-09-11T06:28:29.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:28:29.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=129, tag_tid=27, entry_tid=1], [object_number=128, tag_tid=27, entry_tid=0], [object_number=131, tag_tid=26, entry_tid=6295], [object_number=130, tag_tid=26, entry_tid=6294]]], state=connected] 2024-09-11T06:28:29.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=128, tag_tid=26, entry_tid=5932], [object_number=131, tag_tid=26, entry_tid=5931], [object_number=130, tag_tid=26, entry_tid=5930], [object_number=129, tag_tid=26, entry_tid=5929]]], state=connected] 2024-09-11T06:28:29.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:28:29.143 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=129, tag_tid=27, entry_tid=1' 2024-09-11T06:28:29.143 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:28:29.144 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:28:29.144 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:28:29.144 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:28:29.144 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:28:29.144 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:28:29.144 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:28:29.144 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:28:29.144 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:28:29.145 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:28:29.145 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:28:29.145 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:28:29.147 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:28:29.147 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:28:29.147 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:28:29.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 32 2024-09-11T06:28:29.205 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 32 2024-09-11T06:28:29.205 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:28:29.205 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=129, tag_tid=27, entry_tid=1], [object_number=128, tag_tid=27, entry_tid=0], [object_number=131, tag_tid=26, entry_tid=6295], [object_number=130, tag_tid=26, entry_tid=6294]]], state=connected] 2024-09-11T06:28:29.205 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=128, tag_tid=26, entry_tid=5932], [object_number=131, tag_tid=26, entry_tid=5931], [object_number=130, tag_tid=26, entry_tid=5930], [object_number=129, tag_tid=26, entry_tid=5929]]], state=connected] 2024-09-11T06:28:29.208 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:28:29.210 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=128, tag_tid=26, entry_tid=5932' 2024-09-11T06:28:29.210 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=129, tag_tid=27, entry_tid=1' -a 'object_number=129, tag_tid=27, entry_tid=1' = 'object_number=128, tag_tid=26, entry_tid=5932' 2024-09-11T06:28:29.210 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=128, tag_tid=26, entry_tid=5932' '!=' 'object_number=129, tag_tid=26, entry_tid=5857' 2024-09-11T06:28:29.210 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:28:29.211 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=128, tag_tid=26, entry_tid=5932' = 'object_number=129, tag_tid=26, entry_tid=5857' 2024-09-11T06:28:29.211 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=128, tag_tid=26, entry_tid=5932' 2024-09-11T06:28:29.211 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=129, tag_tid=27, entry_tid=1' 2024-09-11T06:28:29.211 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:28:29.211 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:28:29.211 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=27 2024-09-11T06:28:29.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=128, tag_tid=26, entry_tid=5932' 2024-09-11T06:28:29.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:28:29.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:28:29.213 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=26 2024-09-11T06:28:29.214 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=129, tag_tid=27, entry_tid=1' 2024-09-11T06:28:29.214 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:28:29.214 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:28:29.215 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:28:29.216 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=128, tag_tid=26, entry_tid=5932' 2024-09-11T06:28:29.216 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:28:29.216 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:28:29.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=5932 2024-09-11T06:28:29.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 27 = 26 -a 1 -le 5932 2024-09-11T06:28:29.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:28:29.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:28:29.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:28:29.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:28:29.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:28:29.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:28:29.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:28:29.420 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:28:29.420 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:28:29.420 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:28:29.420 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:28:29.420 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:28:29.420 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:28:29.420 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:28:29.420 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:28:29.420 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:28:29.420 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:28:29.421 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:28:29.421 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:28:29.421 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:28:29.425 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:28:29.425 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:28:29.425 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:28:29.425 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:28:29.425 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:28:29.425 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:28:29.425 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:28:29.425 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:28:29.425 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:28:29.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:28:29.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:28:29.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:28:29.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:28:29.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:28:29.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:28:29.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:28:29.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:28:29.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:28:29.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:28:29.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:28:29.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:28:29.427 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:28:29.427 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:28:29.427 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:28:29.427 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:28:29.427 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:28:29.427 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:28:29.427 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:28:29.427 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:28:29.427 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:28:29.427 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:28:29.427 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:28:29.450 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:28:29.450 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:28:29.451 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:28:29.451 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:28:29.575 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:28:29.576 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:28:29.576 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:28:29.576 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:28:29.576 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:28:29.576 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:28:29.577 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:28:29.577 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:28:29.577 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:28:29.577 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:28:29.577 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:28:29.577 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:28:29.577 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:28:29.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:28:29.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:28:29.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:28:29.632 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 32 2024-09-11T06:28:29.632 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 32 2024-09-11T06:28:29.632 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:28:29.632 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=129, tag_tid=27, entry_tid=1], [object_number=128, tag_tid=27, entry_tid=0], [object_number=131, tag_tid=26, entry_tid=6295], [object_number=130, tag_tid=26, entry_tid=6294]]], state=connected] 2024-09-11T06:28:29.632 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=128, tag_tid=26, entry_tid=6040], [object_number=131, tag_tid=26, entry_tid=6039], [object_number=130, tag_tid=26, entry_tid=6038], [object_number=129, tag_tid=26, entry_tid=6037]]], state=connected] 2024-09-11T06:28:29.636 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:28:29.637 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=129, tag_tid=27, entry_tid=1' 2024-09-11T06:28:29.638 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:28:29.638 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:28:29.638 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:28:29.638 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:28:29.638 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:28:29.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:28:29.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:28:29.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:28:29.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:28:29.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:28:29.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:28:29.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:28:29.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:28:29.640 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:28:29.640 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:28:29.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 32 2024-09-11T06:28:29.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 32 2024-09-11T06:28:29.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:28:29.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=129, tag_tid=27, entry_tid=1], [object_number=128, tag_tid=27, entry_tid=0], [object_number=131, tag_tid=26, entry_tid=6295], [object_number=130, tag_tid=26, entry_tid=6294]]], state=connected] 2024-09-11T06:28:29.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=128, tag_tid=26, entry_tid=6040], [object_number=131, tag_tid=26, entry_tid=6039], [object_number=130, tag_tid=26, entry_tid=6038], [object_number=129, tag_tid=26, entry_tid=6037]]], state=connected] 2024-09-11T06:28:29.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:28:29.693 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=128, tag_tid=26, entry_tid=6040' 2024-09-11T06:28:29.693 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=129, tag_tid=27, entry_tid=1' -a 'object_number=129, tag_tid=27, entry_tid=1' = 'object_number=128, tag_tid=26, entry_tid=6040' 2024-09-11T06:28:29.693 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=128, tag_tid=26, entry_tid=6040' '!=' 'object_number=128, tag_tid=26, entry_tid=5932' 2024-09-11T06:28:29.694 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:28:29.694 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=128, tag_tid=26, entry_tid=6040' = 'object_number=128, tag_tid=26, entry_tid=5932' 2024-09-11T06:28:29.694 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=128, tag_tid=26, entry_tid=6040' 2024-09-11T06:28:29.694 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=129, tag_tid=27, entry_tid=1' 2024-09-11T06:28:29.694 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:28:29.694 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:28:29.694 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=27 2024-09-11T06:28:29.695 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=128, tag_tid=26, entry_tid=6040' 2024-09-11T06:28:29.695 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:28:29.695 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:28:29.696 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=26 2024-09-11T06:28:29.697 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=129, tag_tid=27, entry_tid=1' 2024-09-11T06:28:29.697 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:28:29.697 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:28:29.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:28:29.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=128, tag_tid=26, entry_tid=6040' 2024-09-11T06:28:29.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:28:29.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:28:29.700 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=6040 2024-09-11T06:28:29.700 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 27 = 26 -a 1 -le 6040 2024-09-11T06:28:29.700 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:28:29.700 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:28:29.700 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:28:29.902 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:28:29.902 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:28:29.902 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:28:29.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:28:29.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:28:29.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:28:29.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:28:29.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:28:29.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:28:29.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:28:29.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:28:29.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:28:29.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:28:29.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:28:29.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:28:29.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:28:29.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:28:29.907 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:28:29.907 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:28:29.908 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:28:29.908 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:28:29.908 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:28:29.908 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:28:29.908 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:28:29.908 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:28:29.908 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:28:29.908 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:28:29.908 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:28:29.908 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:28:29.908 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:28:29.908 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:28:29.909 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:28:29.909 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:28:29.909 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:28:29.909 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:28:29.909 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:28:29.909 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:28:29.909 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:28:29.909 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:28:29.909 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:28:29.909 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:28:29.909 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:28:29.910 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:28:29.910 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:28:29.910 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:28:29.910 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:28:29.910 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:28:29.910 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:28:29.910 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:28:29.934 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:28:29.934 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:28:29.934 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:28:29.934 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:28:30.064 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:28:30.065 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:28:30.065 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:28:30.065 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:28:30.065 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:28:30.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:28:30.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:28:30.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:28:30.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:28:30.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:28:30.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:28:30.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:28:30.067 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:28:30.069 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:28:30.069 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:28:30.069 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:28:30.117 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 32 2024-09-11T06:28:30.118 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 32 2024-09-11T06:28:30.118 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:28:30.118 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=129, tag_tid=27, entry_tid=1], [object_number=128, tag_tid=27, entry_tid=0], [object_number=131, tag_tid=26, entry_tid=6295], [object_number=130, tag_tid=26, entry_tid=6294]]], state=connected] 2024-09-11T06:28:30.118 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=130, tag_tid=26, entry_tid=6118], [object_number=129, tag_tid=26, entry_tid=6117], [object_number=128, tag_tid=26, entry_tid=6116], [object_number=131, tag_tid=26, entry_tid=6115]]], state=connected] 2024-09-11T06:28:30.121 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:28:30.122 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=129, tag_tid=27, entry_tid=1' 2024-09-11T06:28:30.123 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:28:30.123 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:28:30.124 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:28:30.124 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:28:30.124 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:28:30.124 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:28:30.124 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:28:30.124 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:28:30.124 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:28:30.124 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:28:30.124 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:28:30.124 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:28:30.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:28:30.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:28:30.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:28:30.172 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 32 2024-09-11T06:28:30.173 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 32 2024-09-11T06:28:30.173 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:28:30.173 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=129, tag_tid=27, entry_tid=1], [object_number=128, tag_tid=27, entry_tid=0], [object_number=131, tag_tid=26, entry_tid=6295], [object_number=130, tag_tid=26, entry_tid=6294]]], state=connected] 2024-09-11T06:28:30.173 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=130, tag_tid=26, entry_tid=6118], [object_number=129, tag_tid=26, entry_tid=6117], [object_number=128, tag_tid=26, entry_tid=6116], [object_number=131, tag_tid=26, entry_tid=6115]]], state=connected] 2024-09-11T06:28:30.176 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:28:30.178 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=130, tag_tid=26, entry_tid=6118' 2024-09-11T06:28:30.178 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=129, tag_tid=27, entry_tid=1' -a 'object_number=129, tag_tid=27, entry_tid=1' = 'object_number=130, tag_tid=26, entry_tid=6118' 2024-09-11T06:28:30.178 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=130, tag_tid=26, entry_tid=6118' '!=' 'object_number=128, tag_tid=26, entry_tid=6040' 2024-09-11T06:28:30.178 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:28:30.178 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=130, tag_tid=26, entry_tid=6118' = 'object_number=128, tag_tid=26, entry_tid=6040' 2024-09-11T06:28:30.178 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=130, tag_tid=26, entry_tid=6118' 2024-09-11T06:28:30.179 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=129, tag_tid=27, entry_tid=1' 2024-09-11T06:28:30.179 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:28:30.179 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:28:30.179 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=27 2024-09-11T06:28:30.180 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=130, tag_tid=26, entry_tid=6118' 2024-09-11T06:28:30.180 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:28:30.180 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:28:30.181 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=26 2024-09-11T06:28:30.181 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=129, tag_tid=27, entry_tid=1' 2024-09-11T06:28:30.181 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:28:30.182 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:28:30.182 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:28:30.183 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=130, tag_tid=26, entry_tid=6118' 2024-09-11T06:28:30.183 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:28:30.183 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:28:30.184 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=6118 2024-09-11T06:28:30.185 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 27 = 26 -a 1 -le 6118 2024-09-11T06:28:30.185 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:28:30.185 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:28:30.185 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:28:30.386 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:28:30.387 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:28:30.387 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:28:30.387 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:28:30.387 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:28:30.387 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:28:30.387 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:28:30.387 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:28:30.387 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:28:30.387 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:28:30.387 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:28:30.387 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:28:30.387 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:28:30.387 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:28:30.387 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:28:30.388 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:28:30.388 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:28:30.392 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:28:30.392 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:28:30.392 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:28:30.392 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:28:30.392 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:28:30.392 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:28:30.392 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:28:30.392 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:28:30.392 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:28:30.392 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:28:30.392 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:28:30.393 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:28:30.393 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:28:30.393 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:28:30.393 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:28:30.393 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:28:30.393 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:28:30.393 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:28:30.393 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:28:30.393 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:28:30.393 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:28:30.393 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:28:30.393 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:28:30.393 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:28:30.393 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:28:30.393 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:28:30.394 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:28:30.394 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:28:30.394 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:28:30.394 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:28:30.394 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:28:30.394 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:28:30.420 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:28:30.421 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:28:30.421 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:28:30.421 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:28:30.551 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:28:30.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:28:30.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:28:30.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:28:30.553 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:28:30.553 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:28:30.553 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:28:30.553 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:28:30.553 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:28:30.553 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:28:30.553 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:28:30.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:28:30.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:28:30.555 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:28:30.555 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:28:30.555 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:28:30.608 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 32 2024-09-11T06:28:30.608 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 32 2024-09-11T06:28:30.608 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:28:30.608 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=129, tag_tid=27, entry_tid=1], [object_number=128, tag_tid=27, entry_tid=0], [object_number=131, tag_tid=26, entry_tid=6295], [object_number=130, tag_tid=26, entry_tid=6294]]], state=connected] 2024-09-11T06:28:30.608 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=128, tag_tid=26, entry_tid=6192], [object_number=131, tag_tid=26, entry_tid=6191], [object_number=130, tag_tid=26, entry_tid=6190], [object_number=129, tag_tid=26, entry_tid=6189]]], state=connected] 2024-09-11T06:28:30.611 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:28:30.613 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=129, tag_tid=27, entry_tid=1' 2024-09-11T06:28:30.614 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:28:30.614 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:28:30.614 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:28:30.614 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:28:30.614 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:28:30.614 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:28:30.614 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:28:30.614 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:28:30.614 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:28:30.615 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:28:30.615 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:28:30.615 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:28:30.615 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:28:30.615 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:28:30.615 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:28:30.663 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 32 2024-09-11T06:28:30.663 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 32 2024-09-11T06:28:30.664 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:28:30.664 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=129, tag_tid=27, entry_tid=1], [object_number=128, tag_tid=27, entry_tid=0], [object_number=131, tag_tid=26, entry_tid=6295], [object_number=130, tag_tid=26, entry_tid=6294]]], state=connected] 2024-09-11T06:28:30.664 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=128, tag_tid=26, entry_tid=6192], [object_number=131, tag_tid=26, entry_tid=6191], [object_number=130, tag_tid=26, entry_tid=6190], [object_number=129, tag_tid=26, entry_tid=6189]]], state=connected] 2024-09-11T06:28:30.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:28:30.669 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=128, tag_tid=26, entry_tid=6192' 2024-09-11T06:28:30.669 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=129, tag_tid=27, entry_tid=1' -a 'object_number=129, tag_tid=27, entry_tid=1' = 'object_number=128, tag_tid=26, entry_tid=6192' 2024-09-11T06:28:30.669 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=128, tag_tid=26, entry_tid=6192' '!=' 'object_number=130, tag_tid=26, entry_tid=6118' 2024-09-11T06:28:30.669 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:28:30.669 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=128, tag_tid=26, entry_tid=6192' = 'object_number=130, tag_tid=26, entry_tid=6118' 2024-09-11T06:28:30.669 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=128, tag_tid=26, entry_tid=6192' 2024-09-11T06:28:30.669 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=129, tag_tid=27, entry_tid=1' 2024-09-11T06:28:30.669 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:28:30.670 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:28:30.670 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=27 2024-09-11T06:28:30.671 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=128, tag_tid=26, entry_tid=6192' 2024-09-11T06:28:30.671 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:28:30.671 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:28:30.672 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=26 2024-09-11T06:28:30.672 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=129, tag_tid=27, entry_tid=1' 2024-09-11T06:28:30.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:28:30.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:28:30.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:28:30.674 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=128, tag_tid=26, entry_tid=6192' 2024-09-11T06:28:30.674 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:28:30.674 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:28:30.675 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=6192 2024-09-11T06:28:30.676 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 27 = 26 -a 1 -le 6192 2024-09-11T06:28:30.676 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:28:30.676 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:28:30.676 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:28:30.878 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:28:30.878 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:28:30.878 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:28:30.878 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:28:30.878 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:28:30.878 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:28:30.878 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:28:30.878 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:28:30.878 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:28:30.878 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:28:30.878 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:28:30.879 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:28:30.879 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:28:30.879 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:28:30.879 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:28:30.879 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:28:30.879 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:28:30.883 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:28:30.883 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:28:30.883 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:28:30.883 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:28:30.883 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:28:30.883 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:28:30.883 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:28:30.883 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:28:30.883 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:28:30.883 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:28:30.883 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:28:30.884 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:28:30.884 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:28:30.884 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:28:30.884 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:28:30.884 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:28:30.884 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:28:30.884 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:28:30.884 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:28:30.884 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:28:30.884 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:28:30.884 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:28:30.884 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:28:30.884 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:28:30.885 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:28:30.885 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:28:30.885 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:28:30.885 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:28:30.885 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:28:30.885 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:28:30.885 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:28:30.885 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:28:30.908 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:28:30.908 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:28:30.908 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:28:30.908 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:28:31.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:28:31.043 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:28:31.043 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:28:31.043 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:28:31.043 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:28:31.043 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:28:31.043 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:28:31.044 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:28:31.044 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:28:31.044 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:28:31.044 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:28:31.044 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:28:31.044 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:28:31.046 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:28:31.047 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:28:31.047 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:28:31.101 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 32 2024-09-11T06:28:31.101 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 32 2024-09-11T06:28:31.101 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:28:31.101 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=129, tag_tid=27, entry_tid=1], [object_number=128, tag_tid=27, entry_tid=0], [object_number=131, tag_tid=26, entry_tid=6295], [object_number=130, tag_tid=26, entry_tid=6294]]], state=connected] 2024-09-11T06:28:31.101 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=130, tag_tid=26, entry_tid=6274], [object_number=129, tag_tid=26, entry_tid=6273], [object_number=128, tag_tid=26, entry_tid=6272], [object_number=131, tag_tid=26, entry_tid=6271]]], state=connected] 2024-09-11T06:28:31.104 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:28:31.105 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=129, tag_tid=27, entry_tid=1' 2024-09-11T06:28:31.107 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:28:31.107 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:28:31.107 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:28:31.108 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:28:31.108 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:28:31.108 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:28:31.108 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:28:31.108 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:28:31.108 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:28:31.108 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:28:31.108 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:28:31.108 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:28:31.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:28:31.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:28:31.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:28:31.156 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 32 2024-09-11T06:28:31.156 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 32 2024-09-11T06:28:31.156 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:28:31.156 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=129, tag_tid=27, entry_tid=1], [object_number=128, tag_tid=27, entry_tid=0], [object_number=131, tag_tid=26, entry_tid=6295], [object_number=130, tag_tid=26, entry_tid=6294]]], state=connected] 2024-09-11T06:28:31.156 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=130, tag_tid=26, entry_tid=6274], [object_number=129, tag_tid=26, entry_tid=6273], [object_number=128, tag_tid=26, entry_tid=6272], [object_number=131, tag_tid=26, entry_tid=6271]]], state=connected] 2024-09-11T06:28:31.160 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:28:31.162 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=130, tag_tid=26, entry_tid=6274' 2024-09-11T06:28:31.162 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=129, tag_tid=27, entry_tid=1' -a 'object_number=129, tag_tid=27, entry_tid=1' = 'object_number=130, tag_tid=26, entry_tid=6274' 2024-09-11T06:28:31.162 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=130, tag_tid=26, entry_tid=6274' '!=' 'object_number=128, tag_tid=26, entry_tid=6192' 2024-09-11T06:28:31.163 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:28:31.163 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=130, tag_tid=26, entry_tid=6274' = 'object_number=128, tag_tid=26, entry_tid=6192' 2024-09-11T06:28:31.163 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=130, tag_tid=26, entry_tid=6274' 2024-09-11T06:28:31.163 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=129, tag_tid=27, entry_tid=1' 2024-09-11T06:28:31.163 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:28:31.163 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:28:31.163 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=27 2024-09-11T06:28:31.164 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=130, tag_tid=26, entry_tid=6274' 2024-09-11T06:28:31.164 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:28:31.164 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:28:31.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=26 2024-09-11T06:28:31.166 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=129, tag_tid=27, entry_tid=1' 2024-09-11T06:28:31.166 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:28:31.166 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:28:31.167 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:28:31.167 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=130, tag_tid=26, entry_tid=6274' 2024-09-11T06:28:31.168 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:28:31.168 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:28:31.169 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=6274 2024-09-11T06:28:31.169 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 27 = 26 -a 1 -le 6274 2024-09-11T06:28:31.169 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:28:31.169 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:28:31.169 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:28:31.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:28:31.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:28:31.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:28:31.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:28:31.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:28:31.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:28:31.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:28:31.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:28:31.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:28:31.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:28:31.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:28:31.372 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:28:31.372 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:28:31.372 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:28:31.372 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:28:31.372 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:28:31.373 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:28:31.375 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:28:31.375 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:28:31.375 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:28:31.375 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:28:31.376 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:28:31.376 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:28:31.376 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:28:31.376 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:28:31.376 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:28:31.376 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:28:31.376 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:28:31.376 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:28:31.376 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:28:31.376 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:28:31.376 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:28:31.376 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:28:31.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:28:31.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:28:31.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:28:31.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:28:31.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:28:31.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:28:31.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:28:31.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:28:31.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:28:31.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:28:31.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:28:31.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:28:31.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:28:31.378 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:28:31.378 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:28:31.378 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:28:31.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:28:31.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:28:31.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:28:31.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:28:31.528 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:28:31.529 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:28:31.529 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:28:31.529 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:28:31.530 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:28:31.530 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:28:31.530 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:28:31.530 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:28:31.530 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:28:31.530 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:28:31.531 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:28:31.531 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:28:31.531 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:28:31.532 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:28:31.532 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:28:31.533 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:28:31.585 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 32 2024-09-11T06:28:31.585 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 32 2024-09-11T06:28:31.585 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:28:31.585 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=129, tag_tid=27, entry_tid=1], [object_number=128, tag_tid=27, entry_tid=0], [object_number=131, tag_tid=26, entry_tid=6295], [object_number=130, tag_tid=26, entry_tid=6294]]], state=connected] 2024-09-11T06:28:31.585 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=128, tag_tid=26, entry_tid=6296], [object_number=131, tag_tid=26, entry_tid=6295], [object_number=130, tag_tid=26, entry_tid=6294], [object_number=129, tag_tid=26, entry_tid=6293]]], state=connected] 2024-09-11T06:28:31.589 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:28:31.590 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=129, tag_tid=27, entry_tid=1' 2024-09-11T06:28:31.592 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:28:31.592 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:28:31.592 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:28:31.592 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:28:31.592 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:28:31.592 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:28:31.592 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:28:31.592 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:28:31.592 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:28:31.592 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:28:31.592 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:28:31.592 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:28:31.593 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:28:31.594 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:28:31.594 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:28:31.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 32 2024-09-11T06:28:31.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 32 2024-09-11T06:28:31.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:28:31.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=129, tag_tid=27, entry_tid=1], [object_number=128, tag_tid=27, entry_tid=0], [object_number=131, tag_tid=26, entry_tid=6295], [object_number=130, tag_tid=26, entry_tid=6294]]], state=connected] 2024-09-11T06:28:31.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=128, tag_tid=26, entry_tid=6296], [object_number=131, tag_tid=26, entry_tid=6295], [object_number=130, tag_tid=26, entry_tid=6294], [object_number=129, tag_tid=26, entry_tid=6293]]], state=connected] 2024-09-11T06:28:31.645 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:28:31.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=128, tag_tid=26, entry_tid=6296' 2024-09-11T06:28:31.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=129, tag_tid=27, entry_tid=1' -a 'object_number=129, tag_tid=27, entry_tid=1' = 'object_number=128, tag_tid=26, entry_tid=6296' 2024-09-11T06:28:31.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=128, tag_tid=26, entry_tid=6296' '!=' 'object_number=130, tag_tid=26, entry_tid=6274' 2024-09-11T06:28:31.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:28:31.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=128, tag_tid=26, entry_tid=6296' = 'object_number=130, tag_tid=26, entry_tid=6274' 2024-09-11T06:28:31.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=128, tag_tid=26, entry_tid=6296' 2024-09-11T06:28:31.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=129, tag_tid=27, entry_tid=1' 2024-09-11T06:28:31.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:28:31.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:28:31.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=27 2024-09-11T06:28:31.649 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=128, tag_tid=26, entry_tid=6296' 2024-09-11T06:28:31.649 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:28:31.649 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:28:31.649 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=26 2024-09-11T06:28:31.650 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=129, tag_tid=27, entry_tid=1' 2024-09-11T06:28:31.650 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:28:31.650 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:28:31.651 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:28:31.652 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=128, tag_tid=26, entry_tid=6296' 2024-09-11T06:28:31.652 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:28:31.652 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:28:31.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=6296 2024-09-11T06:28:31.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 27 = 26 -a 1 -le 6296 2024-09-11T06:28:31.654 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:28:31.654 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:28:31.654 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:28:31.855 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:28:31.855 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:28:31.855 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:28:31.855 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:28:31.855 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:28:31.855 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:28:31.855 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:28:31.855 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:28:31.855 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:28:31.855 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:28:31.855 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:28:31.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:28:31.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:28:31.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:28:31.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:28:31.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:28:31.857 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:28:31.859 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:28:31.859 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:28:31.859 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:28:31.859 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:28:31.859 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:28:31.859 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:28:31.859 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:28:31.859 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:28:31.860 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:28:31.860 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:28:31.860 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:28:31.860 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:28:31.860 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:28:31.860 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:28:31.860 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:28:31.860 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:28:31.860 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:28:31.860 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:28:31.860 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:28:31.860 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:28:31.860 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:28:31.861 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:28:31.861 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:28:31.861 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:28:31.861 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:28:31.861 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:28:31.861 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:28:31.861 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:28:31.861 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:28:31.861 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:28:31.861 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:28:31.861 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:28:31.885 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:28:31.885 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:28:31.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:28:31.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:28:32.011 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:28:32.012 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:28:32.013 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:28:32.013 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:28:32.013 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:28:32.013 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:28:32.013 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:28:32.013 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:28:32.013 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:28:32.013 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:28:32.014 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:28:32.014 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:28:32.014 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:28:32.016 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:28:32.017 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:28:32.017 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:28:32.074 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 32 2024-09-11T06:28:32.074 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 32 2024-09-11T06:28:32.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:28:32.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=129, tag_tid=27, entry_tid=1], [object_number=128, tag_tid=27, entry_tid=0], [object_number=131, tag_tid=26, entry_tid=6295], [object_number=130, tag_tid=26, entry_tid=6294]]], state=connected] 2024-09-11T06:28:32.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=129, tag_tid=27, entry_tid=1], [object_number=128, tag_tid=27, entry_tid=0], [object_number=131, tag_tid=26, entry_tid=6295], [object_number=130, tag_tid=26, entry_tid=6294]]], state=connected] 2024-09-11T06:28:32.078 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:28:32.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=129, tag_tid=27, entry_tid=1' 2024-09-11T06:28:32.081 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:28:32.081 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:28:32.081 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:28:32.081 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:28:32.081 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:28:32.081 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:28:32.081 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:28:32.081 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:28:32.081 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:28:32.081 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:28:32.082 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:28:32.082 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:28:32.082 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:28:32.082 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:28:32.083 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:28:32.133 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 32 2024-09-11T06:28:32.133 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 32 2024-09-11T06:28:32.133 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:28:32.133 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=129, tag_tid=27, entry_tid=1], [object_number=128, tag_tid=27, entry_tid=0], [object_number=131, tag_tid=26, entry_tid=6295], [object_number=130, tag_tid=26, entry_tid=6294]]], state=connected] 2024-09-11T06:28:32.133 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=129, tag_tid=27, entry_tid=1], [object_number=128, tag_tid=27, entry_tid=0], [object_number=131, tag_tid=26, entry_tid=6295], [object_number=130, tag_tid=26, entry_tid=6294]]], state=connected] 2024-09-11T06:28:32.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:28:32.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=129, tag_tid=27, entry_tid=1' 2024-09-11T06:28:32.139 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=129, tag_tid=27, entry_tid=1' -a 'object_number=129, tag_tid=27, entry_tid=1' = 'object_number=129, tag_tid=27, entry_tid=1' 2024-09-11T06:28:32.139 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:28:32.139 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_snap_present cluster1 mirror test snap9 2024-09-11T06:28:32.139 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:28:32.139 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:28:32.139 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:28:32.139 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap9 2024-09-11T06:28:32.139 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:28:32.139 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T06:28:32.139 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:28:33.139 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 info mirror/test@snap9 2024-09-11T06:28:33.211 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:rbd image 'test': 2024-09-11T06:28:33.211 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: size 512 MiB in 128 objects 2024-09-11T06:28:33.211 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: order 22 (4 MiB objects) 2024-09-11T06:28:33.211 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: snapshot_count: 17 2024-09-11T06:28:33.211 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: id: 10a618cfe337 2024-09-11T06:28:33.211 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: block_name_prefix: rbd_data.10a618cfe337 2024-09-11T06:28:33.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: format: 2 2024-09-11T06:28:33.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: features: layering, exclusive-lock, journaling 2024-09-11T06:28:33.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: op_features: 2024-09-11T06:28:33.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: flags: 2024-09-11T06:28:33.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: create_timestamp: Wed Sep 11 06:16:46 2024 2024-09-11T06:28:33.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: access_timestamp: Wed Sep 11 06:26:51 2024 2024-09-11T06:28:33.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: modify_timestamp: Wed Sep 11 06:28:02 2024 2024-09-11T06:28:33.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: protected: False 2024-09-11T06:28:33.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: journal: 10a618cfe337 2024-09-11T06:28:33.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring state: enabled 2024-09-11T06:28:33.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring mode: journal 2024-09-11T06:28:33.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring global id: a0f7d515-874e-4c67-a26b-4958c7ab3083 2024-09-11T06:28:33.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring primary: false 2024-09-11T06:28:33.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:28:33.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n snap8-clean ']' 2024-09-11T06:28:33.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ compare_image_snaps mirror test snap8-clean 2024-09-11T06:28:33.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:28:33.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:28:33.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap8-clean 2024-09-11T06:28:33.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local ret=0 2024-09-11T06:28:33.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local rmt_export=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export 2024-09-11T06:28:33.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loc_export=/tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:28:33.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:28:33.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8-clean /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export 2024-09-11T06:28:33.874 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T06:28:33.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8-clean /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:28:35.499 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T06:28:35.520 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmp /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:28:35.798 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:28:35.937 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:28:35.937 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ compare_image_snaps mirror test snap9 2024-09-11T06:28:35.937 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:28:35.937 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:28:35.937 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap9 2024-09-11T06:28:35.937 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local ret=0 2024-09-11T06:28:35.937 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local rmt_export=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export 2024-09-11T06:28:35.937 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loc_export=/tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:28:35.937 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:28:35.938 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9 /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export 2024-09-11T06:28:36.626 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T06:28:36.637 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9 /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:28:38.274 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T06:28:38.305 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmp /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:28:38.591 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:28:38.729 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:28:38.729 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ clean_snap_name=snap9-clean 2024-09-11T06:28:38.729 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_snap cluster2 mirror test snap9-clean 2024-09-11T06:28:38.729 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:28:38.729 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:28:38.729 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:28:38.729 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap9-clean 2024-09-11T06:28:38.729 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap9-clean --debug-rbd=20 --debug-journaler=20 2024-09-11T06:28:39.411 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 10` 2024-09-11T06:28:39.411 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ stress_write_image cluster2 mirror test 2024-09-11T06:28:39.411 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:28:39.411 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:28:39.411 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:28:39.412 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T06:28:39.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local duration=7 2024-09-11T06:28:39.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set +e 2024-09-11T06:28:39.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-11T06:28:41.897 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:100 IOs, 49810432 bytes 2024-09-11T06:28:42.813 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:200 IOs, 73216000 bytes 2024-09-11T06:28:44.655 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:300 IOs, 97083392 bytes 2024-09-11T06:28:46.425 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ error_code=124 2024-09-11T06:28:46.425 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set -e 2024-09-11T06:28:46.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T06:28:46.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:28:46.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-11T06:28:46.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:28:46.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:28:46.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:28:46.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state_pattern=up+replaying 2024-09-11T06:28:46.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local description_pattern= 2024-09-11T06:28:46.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local service_pattern= 2024-09-11T06:28:46.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:28:46.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:28:47.427 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-11T06:28:47.427 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:28:47.427 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:28:47.427 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:28:47.427 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state_pattern=up+replaying 2024-09-11T06:28:47.427 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local description_pattern= 2024-09-11T06:28:47.427 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local service_pattern= 2024-09-11T06:28:47.427 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local status 2024-09-11T06:28:47.428 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ CEPH_ARGS= 2024-09-11T06:28:47.428 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-11T06:28:47.529 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ status='test: 2024-09-11T06:28:47.529 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: global_id: a0f7d515-874e-4c67-a26b-4958c7ab3083 2024-09-11T06:28:47.530 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: state: up+replaying 2024-09-11T06:28:47.530 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: description: replaying, {"bytes_per_second":1929405.97,"entries_behind_primary":0,"entries_per_second":123.5,"non_primary_position":{"entry_tid":1,"object_number":129,"tag_tid":27},"primary_position":{"entry_tid":1,"object_number":129,"tag_tid":27}} 2024-09-11T06:28:47.530 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: service: mirror.0 on smithi112 2024-09-11T06:28:47.530 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: last_update: 2024-09-11 06:28:44 2024-09-11T06:28:47.530 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: peer_sites: 2024-09-11T06:28:47.530 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: name: cluster2 2024-09-11T06:28:47.530 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: state: up+stopped 2024-09-11T06:28:47.530 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: description: local image is primary 2024-09-11T06:28:47.530 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: last_update: 2024-09-11 06:28:44' 2024-09-11T06:28:47.530 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ grep '^ state: .*up+replaying' 2024-09-11T06:28:47.531 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: state: up+replaying 2024-09-11T06:28:47.531 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ grep '^ description: .*' 2024-09-11T06:28:47.532 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: description: replaying, {"bytes_per_second":1929405.97,"entries_behind_primary":0,"entries_per_second":123.5,"non_primary_position":{"entry_tid":1,"object_number":129,"tag_tid":27},"primary_position":{"entry_tid":1,"object_number":129,"tag_tid":27}} 2024-09-11T06:28:47.532 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n '' ']' 2024-09-11T06:28:47.532 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ echo up+replaying 2024-09-11T06:28:47.532 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ grep '^up+' 2024-09-11T06:28:47.533 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:up+replaying 2024-09-11T06:28:47.534 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ grep 'service: *mirror..* on ' 2024-09-11T06:28:47.534 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: service: mirror.0 on smithi112 2024-09-11T06:28:47.535 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local last_update 2024-09-11T06:28:47.535 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-11T06:28:47.538 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_update='2024-09-11 06:28:44' 2024-09-11T06:28:47.538 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-11 06:28:44' 2024-09-11T06:28:47.538 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:28:47.538 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:28:47.538 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:28:47.538 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state_pattern=up+replaying 2024-09-11T06:28:47.538 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'prev_last_update=2024-09-11 06:28:44' 2024-09-11T06:28:47.538 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local status 2024-09-11T06:28:47.538 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ CEPH_ARGS= 2024-09-11T06:28:47.538 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-11T06:28:47.623 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ status='OKOKOK142554262mirror.0smithi11219.1.1-298-gb53b6d3etrueOK42594265mirror.1smithi11219.1.1-298-gb53b6d3efalseOKtesta0f7d515-874e-4c67-a26b-4958c7ab3083up+replayingreplaying, {"bytes_per_second":1929405.97,"entries_behind_primary":0,"entries_per_second":123.5,"non_primary_position":{"entry_tid":1,"object_number":129,"tag_tid":27},"primary_position":{"entry_tid":1,"object_number":129,"tag_tid":27}}42554262mirror.0smithi1122024-09-11 06:28:44cluster20bfc6eda-f3c5-424a-8ff7-5c4d3d9fd488up+stoppedlocal image is primary2024-09-11 06:28:44' 2024-09-11T06:28:47.623 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local last_update state 2024-09-11T06:28:47.623 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-11T06:28:47.626 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_update='2024-09-11 06:28:44' 2024-09-11T06:28:47.626 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-11T06:28:47.629 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ state=up+replaying 2024-09-11T06:28:47.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ echo up+replaying 2024-09-11T06:28:47.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ grep up+replaying 2024-09-11T06:28:47.631 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:up+replaying 2024-09-11T06:28:47.632 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:28:47.632 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:28:47.632 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ snap_name=snap10 2024-09-11T06:28:47.632 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_snap cluster2 mirror test snap10 2024-09-11T06:28:47.632 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:28:47.632 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:28:47.633 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:28:47.633 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap10 2024-09-11T06:28:47.633 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap10 --debug-rbd=20 --debug-journaler=20 2024-09-11T06:29:16.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-11T06:29:16.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:29:16.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:29:16.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:29:16.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-11T06:29:16.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:29:16.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:29:16.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:29:16.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=started 2024-09-11T06:29:16.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:29:16.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:29:16.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:29:17.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-11T06:29:17.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:29:17.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:29:17.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:29:17.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=started 2024-09-11T06:29:17.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local status_result 2024-09-11T06:29:17.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=stopped 2024-09-11T06:29:17.484 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -i state 2024-09-11T06:29:17.484 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-11T06:29:17.484 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_instance=cluster1 2024-09-11T06:29:17.484 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:29:17.484 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:29:17.484 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance=cluster1 2024-09-11T06:29:17.485 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local loop_instance 2024-09-11T06:29:17.485 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:29:17.485 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sleep 0 2024-09-11T06:29:17.485 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:29:17.485 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ seq 0 1 2024-09-11T06:29:17.486 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:29:17.486 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-11T06:29:17.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1:0 2024-09-11T06:29:17.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:29:17.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:29:17.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:29:17.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:29:17.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=cluster 2024-09-11T06:29:17.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:29:17.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:29:17.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:29:17.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:29:17.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval cluster=cluster1 2024-09-11T06:29:17.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:29:17.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:29:17.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:29:17.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:29:17.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local local_cluster=cluster1:0 2024-09-11T06:29:17.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster=cluster1 2024-09-11T06:29:17.490 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance 2024-09-11T06:29:17.490 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:29:17.490 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local val=cluster1:0 2024-09-11T06:29:17.490 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T06:29:17.490 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance_var_name=instance 2024-09-11T06:29:17.490 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:29:17.490 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:29:17.490 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T06:29:17.490 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval local_cluster=cluster1 2024-09-11T06:29:17.490 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ local_cluster=cluster1 2024-09-11T06:29:17.490 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval instance=0 2024-09-11T06:29:17.490 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ instance=0 2024-09-11T06:29:17.490 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:29:17.509 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:29:17.509 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:29:17.509 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:29:17.510 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-11T06:29:17.638 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ return 0 2024-09-11T06:29:17.638 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T06:29:17.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ echo ' "state": "Replaying"' 2024-09-11T06:29:17.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ grep -i Replaying 2024-09-11T06:29:17.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: "state": "Replaying" 2024-09-11T06:29:17.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_state=started 2024-09-11T06:29:17.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test started = started 2024-09-11T06:29:17.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:29:17.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-11T06:29:17.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:29:17.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:29:17.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:29:17.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:29:17.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' journal = journal ']' 2024-09-11T06:29:17.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-09-11T06:29:17.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:29:17.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:29:17.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:29:17.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:29:17.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-11T06:29:17.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-11T06:29:17.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:29:17.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:29:17.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:29:17.844 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:29:17.844 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:29:17.844 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:29:17.844 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:29:17.844 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:29:17.844 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:29:17.844 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:29:17.844 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:29:17.845 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:29:17.845 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:29:17.845 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:29:17.845 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:29:17.845 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:29:17.845 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:29:17.845 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:29:17.845 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:29:17.846 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:29:17.848 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:29:17.848 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:29:17.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:29:17.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:29:17.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:29:17.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:29:17.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:29:17.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:29:17.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:29:17.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:29:17.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:29:17.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:29:17.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:29:17.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:29:17.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:29:17.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:29:17.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:29:17.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:29:17.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:29:17.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:29:17.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:29:17.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:29:17.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:29:17.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:29:17.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:29:17.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:29:17.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:29:17.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:29:17.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:29:17.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:29:17.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:29:17.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:29:17.877 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:29:17.877 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:29:17.877 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:29:17.877 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:29:18.005 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:29:18.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:29:18.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:18.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:18.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:18.007 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:29:18.007 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:18.007 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:18.007 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:18.007 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:29:18.007 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:29:18.007 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:29:18.008 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:29:18.009 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:18.010 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:29:18.010 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:18.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 33 2024-09-11T06:29:18.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 34 2024-09-11T06:29:18.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:29:18.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=8011], [object_number=138, tag_tid=29, entry_tid=8010]]], state=connected] 2024-09-11T06:29:18.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=133, tag_tid=29, entry_tid=5545], [object_number=132, tag_tid=29, entry_tid=5544], [object_number=135, tag_tid=29, entry_tid=5543], [object_number=138, tag_tid=29, entry_tid=5542]]], state=connected] 2024-09-11T06:29:18.069 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:18.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=137, tag_tid=30, entry_tid=1' 2024-09-11T06:29:18.071 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:29:18.071 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:18.072 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:18.072 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:18.072 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:29:18.072 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:18.072 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:18.072 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:18.072 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:29:18.072 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:29:18.072 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:29:18.072 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:29:18.073 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:18.073 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:29:18.073 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:18.122 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 33 2024-09-11T06:29:18.122 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 34 2024-09-11T06:29:18.122 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:29:18.122 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=8011], [object_number=138, tag_tid=29, entry_tid=8010]]], state=connected] 2024-09-11T06:29:18.122 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=133, tag_tid=29, entry_tid=5545], [object_number=132, tag_tid=29, entry_tid=5544], [object_number=135, tag_tid=29, entry_tid=5543], [object_number=138, tag_tid=29, entry_tid=5542]]], state=connected] 2024-09-11T06:29:18.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:18.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=133, tag_tid=29, entry_tid=5545' 2024-09-11T06:29:18.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=137, tag_tid=30, entry_tid=1' -a 'object_number=137, tag_tid=30, entry_tid=1' = 'object_number=133, tag_tid=29, entry_tid=5545' 2024-09-11T06:29:18.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=133, tag_tid=29, entry_tid=5545' '!=' '' 2024-09-11T06:29:18.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:29:18.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=133, tag_tid=29, entry_tid=5545' = '' 2024-09-11T06:29:18.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=133, tag_tid=29, entry_tid=5545' 2024-09-11T06:29:18.128 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-09-11T06:29:18.128 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:29:18.128 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:18.129 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=30 2024-09-11T06:29:18.129 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=133, tag_tid=29, entry_tid=5545' 2024-09-11T06:29:18.129 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:29:18.129 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:18.130 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=29 2024-09-11T06:29:18.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-09-11T06:29:18.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:29:18.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:18.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:29:18.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=133, tag_tid=29, entry_tid=5545' 2024-09-11T06:29:18.133 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:29:18.133 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:18.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=5545 2024-09-11T06:29:18.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 30 = 29 -a 1 -le 5545 2024-09-11T06:29:18.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:29:18.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:29:18.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:29:18.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:29:18.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:29:18.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:29:18.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:29:18.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:29:18.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:29:18.337 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:29:18.337 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:29:18.337 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:29:18.337 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:29:18.337 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:29:18.337 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:29:18.337 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:29:18.337 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:29:18.337 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:29:18.338 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:29:18.338 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:29:18.341 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:29:18.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:29:18.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:29:18.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:29:18.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:29:18.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:29:18.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:29:18.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:29:18.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:29:18.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:29:18.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:29:18.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:29:18.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:29:18.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:29:18.343 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:29:18.343 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:29:18.343 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:29:18.343 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:29:18.343 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:29:18.343 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:29:18.343 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:29:18.343 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:29:18.343 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:29:18.343 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:29:18.344 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:29:18.344 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:29:18.344 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:29:18.344 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:29:18.344 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:29:18.344 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:29:18.344 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:29:18.344 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:29:18.367 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:29:18.368 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:29:18.368 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:29:18.368 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:29:18.499 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:29:18.499 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:29:18.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:18.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:18.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:18.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:29:18.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:18.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:18.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:18.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:29:18.501 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:29:18.501 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:29:18.501 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:29:18.504 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:18.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:29:18.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:18.560 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 33 2024-09-11T06:29:18.560 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 34 2024-09-11T06:29:18.560 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:29:18.561 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=8011], [object_number=138, tag_tid=29, entry_tid=8010]]], state=connected] 2024-09-11T06:29:18.561 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=135, tag_tid=29, entry_tid=5627], [object_number=138, tag_tid=29, entry_tid=5626], [object_number=133, tag_tid=29, entry_tid=5625], [object_number=132, tag_tid=29, entry_tid=5624]]], state=connected] 2024-09-11T06:29:18.563 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:18.565 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=137, tag_tid=30, entry_tid=1' 2024-09-11T06:29:18.566 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:29:18.566 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:18.566 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:18.566 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:18.566 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:29:18.566 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:18.566 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:18.566 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:18.566 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:29:18.566 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:29:18.566 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:29:18.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:29:18.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:18.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:29:18.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:18.615 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 33 2024-09-11T06:29:18.615 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 34 2024-09-11T06:29:18.615 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:29:18.615 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=8011], [object_number=138, tag_tid=29, entry_tid=8010]]], state=connected] 2024-09-11T06:29:18.615 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=135, tag_tid=29, entry_tid=5627], [object_number=138, tag_tid=29, entry_tid=5626], [object_number=133, tag_tid=29, entry_tid=5625], [object_number=132, tag_tid=29, entry_tid=5624]]], state=connected] 2024-09-11T06:29:18.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:18.621 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=135, tag_tid=29, entry_tid=5627' 2024-09-11T06:29:18.621 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=137, tag_tid=30, entry_tid=1' -a 'object_number=137, tag_tid=30, entry_tid=1' = 'object_number=135, tag_tid=29, entry_tid=5627' 2024-09-11T06:29:18.621 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=135, tag_tid=29, entry_tid=5627' '!=' 'object_number=133, tag_tid=29, entry_tid=5545' 2024-09-11T06:29:18.621 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:29:18.621 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=135, tag_tid=29, entry_tid=5627' = 'object_number=133, tag_tid=29, entry_tid=5545' 2024-09-11T06:29:18.621 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=135, tag_tid=29, entry_tid=5627' 2024-09-11T06:29:18.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-09-11T06:29:18.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:29:18.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:18.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=30 2024-09-11T06:29:18.623 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=135, tag_tid=29, entry_tid=5627' 2024-09-11T06:29:18.623 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:29:18.623 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:18.624 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=29 2024-09-11T06:29:18.624 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-09-11T06:29:18.624 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:29:18.624 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:18.625 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:29:18.626 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=135, tag_tid=29, entry_tid=5627' 2024-09-11T06:29:18.626 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:29:18.626 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:18.627 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=5627 2024-09-11T06:29:18.627 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 30 = 29 -a 1 -le 5627 2024-09-11T06:29:18.627 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:29:18.627 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:29:18.628 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:29:18.829 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:29:18.829 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:29:18.829 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:29:18.829 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:29:18.830 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:29:18.830 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:29:18.830 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:29:18.830 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:29:18.830 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:29:18.830 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:29:18.830 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:29:18.830 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:29:18.830 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:29:18.830 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:29:18.830 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:29:18.831 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:29:18.831 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:29:18.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:29:18.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:29:18.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:29:18.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:29:18.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:29:18.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:29:18.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:29:18.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:29:18.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:29:18.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:29:18.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:29:18.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:29:18.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:29:18.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:29:18.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:29:18.835 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:29:18.836 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:29:18.836 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:29:18.836 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:29:18.836 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:29:18.836 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:29:18.836 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:29:18.836 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:29:18.836 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:29:18.836 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:29:18.836 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:29:18.837 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:29:18.837 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:29:18.837 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:29:18.837 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:29:18.837 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:29:18.837 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:29:18.862 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:29:18.862 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:29:18.862 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:29:18.863 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:29:18.990 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:29:18.991 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:29:18.991 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:18.991 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:18.992 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:18.992 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:29:18.992 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:18.992 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:18.992 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:18.992 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:29:18.992 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:29:18.992 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:29:18.992 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:29:18.992 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:18.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:29:18.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:19.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 33 2024-09-11T06:29:19.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 34 2024-09-11T06:29:19.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:29:19.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=8011], [object_number=138, tag_tid=29, entry_tid=8010]]], state=connected] 2024-09-11T06:29:19.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=136, tag_tid=29, entry_tid=5676], [object_number=139, tag_tid=29, entry_tid=5675], [object_number=138, tag_tid=29, entry_tid=5674], [object_number=133, tag_tid=29, entry_tid=5673]]], state=connected] 2024-09-11T06:29:19.059 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:19.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=137, tag_tid=30, entry_tid=1' 2024-09-11T06:29:19.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:29:19.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:19.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:19.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:19.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:29:19.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:19.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:19.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:19.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:29:19.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:29:19.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:29:19.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:29:19.063 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:19.063 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:29:19.063 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:19.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 33 2024-09-11T06:29:19.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 34 2024-09-11T06:29:19.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:29:19.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=8011], [object_number=138, tag_tid=29, entry_tid=8010]]], state=connected] 2024-09-11T06:29:19.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=136, tag_tid=29, entry_tid=5676], [object_number=139, tag_tid=29, entry_tid=5675], [object_number=138, tag_tid=29, entry_tid=5674], [object_number=133, tag_tid=29, entry_tid=5673]]], state=connected] 2024-09-11T06:29:19.114 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:19.116 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=136, tag_tid=29, entry_tid=5676' 2024-09-11T06:29:19.116 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=137, tag_tid=30, entry_tid=1' -a 'object_number=137, tag_tid=30, entry_tid=1' = 'object_number=136, tag_tid=29, entry_tid=5676' 2024-09-11T06:29:19.116 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=136, tag_tid=29, entry_tid=5676' '!=' 'object_number=135, tag_tid=29, entry_tid=5627' 2024-09-11T06:29:19.116 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:29:19.117 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=136, tag_tid=29, entry_tid=5676' = 'object_number=135, tag_tid=29, entry_tid=5627' 2024-09-11T06:29:19.117 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=136, tag_tid=29, entry_tid=5676' 2024-09-11T06:29:19.117 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-09-11T06:29:19.117 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:29:19.117 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:19.117 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=30 2024-09-11T06:29:19.118 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=136, tag_tid=29, entry_tid=5676' 2024-09-11T06:29:19.118 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:29:19.118 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:19.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=29 2024-09-11T06:29:19.120 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-09-11T06:29:19.120 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:29:19.120 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:19.121 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:29:19.121 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=136, tag_tid=29, entry_tid=5676' 2024-09-11T06:29:19.121 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:29:19.121 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:19.123 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=5676 2024-09-11T06:29:19.123 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 30 = 29 -a 1 -le 5676 2024-09-11T06:29:19.123 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:29:19.123 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:29:19.123 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:29:19.324 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:29:19.325 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:29:19.325 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:29:19.325 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:29:19.325 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:29:19.325 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:29:19.325 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:29:19.325 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:29:19.325 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:29:19.325 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:29:19.325 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:29:19.325 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:29:19.326 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:29:19.326 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:29:19.326 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:29:19.326 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:29:19.326 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:29:19.330 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:29:19.330 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:29:19.330 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:29:19.330 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:29:19.330 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:29:19.330 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:29:19.330 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:29:19.330 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:29:19.330 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:29:19.330 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:29:19.330 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:29:19.330 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:29:19.331 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:29:19.331 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:29:19.331 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:29:19.331 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:29:19.331 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:29:19.331 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:29:19.331 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:29:19.331 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:29:19.331 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:29:19.331 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:29:19.331 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:29:19.331 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:29:19.331 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:29:19.331 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:29:19.332 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:29:19.332 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:29:19.332 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:29:19.332 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:29:19.332 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:29:19.332 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:29:19.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:29:19.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:29:19.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:29:19.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:29:19.499 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:29:19.501 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:29:19.501 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:19.501 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:19.501 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:19.501 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:29:19.501 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:19.501 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:19.501 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:19.501 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:29:19.501 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:29:19.502 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:29:19.502 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:29:19.504 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:19.504 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:29:19.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:19.563 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 34 2024-09-11T06:29:19.563 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 34 2024-09-11T06:29:19.563 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:29:19.563 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=8011], [object_number=138, tag_tid=29, entry_tid=8010]]], state=connected] 2024-09-11T06:29:19.563 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=137, tag_tid=29, entry_tid=5741], [object_number=136, tag_tid=29, entry_tid=5740], [object_number=139, tag_tid=29, entry_tid=5739], [object_number=138, tag_tid=29, entry_tid=5738]]], state=connected] 2024-09-11T06:29:19.566 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:19.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=137, tag_tid=30, entry_tid=1' 2024-09-11T06:29:19.569 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:29:19.569 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:19.569 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:19.569 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:19.569 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:29:19.569 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:19.569 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:19.570 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:19.570 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:29:19.570 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:29:19.570 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:29:19.570 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:29:19.570 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:19.570 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:29:19.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:19.618 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 34 2024-09-11T06:29:19.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 34 2024-09-11T06:29:19.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:29:19.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=8011], [object_number=138, tag_tid=29, entry_tid=8010]]], state=connected] 2024-09-11T06:29:19.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=137, tag_tid=29, entry_tid=5741], [object_number=136, tag_tid=29, entry_tid=5740], [object_number=139, tag_tid=29, entry_tid=5739], [object_number=138, tag_tid=29, entry_tid=5738]]], state=connected] 2024-09-11T06:29:19.621 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:19.624 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=137, tag_tid=29, entry_tid=5741' 2024-09-11T06:29:19.624 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=137, tag_tid=30, entry_tid=1' -a 'object_number=137, tag_tid=30, entry_tid=1' = 'object_number=137, tag_tid=29, entry_tid=5741' 2024-09-11T06:29:19.624 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=137, tag_tid=29, entry_tid=5741' '!=' 'object_number=136, tag_tid=29, entry_tid=5676' 2024-09-11T06:29:19.625 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:29:19.625 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=137, tag_tid=29, entry_tid=5741' = 'object_number=136, tag_tid=29, entry_tid=5676' 2024-09-11T06:29:19.625 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=137, tag_tid=29, entry_tid=5741' 2024-09-11T06:29:19.625 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-09-11T06:29:19.625 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:29:19.625 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:19.627 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=30 2024-09-11T06:29:19.627 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=137, tag_tid=29, entry_tid=5741' 2024-09-11T06:29:19.628 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:29:19.628 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:19.628 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=29 2024-09-11T06:29:19.628 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-09-11T06:29:19.628 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:29:19.628 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:19.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:29:19.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=137, tag_tid=29, entry_tid=5741' 2024-09-11T06:29:19.631 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:29:19.631 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:19.631 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=5741 2024-09-11T06:29:19.631 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 30 = 29 -a 1 -le 5741 2024-09-11T06:29:19.631 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:29:19.631 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:29:19.632 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:29:19.831 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:29:19.831 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:29:19.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:29:19.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:29:19.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:29:19.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:29:19.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:29:19.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:29:19.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:29:19.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:29:19.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:29:19.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:29:19.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:29:19.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:29:19.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:29:19.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:29:19.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:29:19.837 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:29:19.837 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:29:19.837 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:29:19.837 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:29:19.837 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:29:19.838 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:29:19.838 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:29:19.838 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:29:19.838 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:29:19.838 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:29:19.838 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:29:19.838 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:29:19.839 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:29:19.839 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:29:19.839 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:29:19.839 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:29:19.839 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:29:19.839 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:29:19.840 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:29:19.840 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:29:19.840 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:29:19.840 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:29:19.840 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:29:19.840 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:29:19.841 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:29:19.841 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:29:19.841 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:29:19.841 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:29:19.841 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:29:19.841 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:29:19.841 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:29:19.842 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:29:19.861 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:29:19.861 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:29:19.861 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:29:19.862 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:29:19.999 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:29:20.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:29:20.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:20.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:20.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:20.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:29:20.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:20.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:20.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:20.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:29:20.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:29:20.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:29:20.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:29:20.003 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:20.003 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:20.003 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:29:20.059 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 34 2024-09-11T06:29:20.059 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 34 2024-09-11T06:29:20.059 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:29:20.059 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=8011], [object_number=138, tag_tid=29, entry_tid=8010]]], state=connected] 2024-09-11T06:29:20.059 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=137, tag_tid=29, entry_tid=5837], [object_number=136, tag_tid=29, entry_tid=5836], [object_number=139, tag_tid=29, entry_tid=5835], [object_number=138, tag_tid=29, entry_tid=5834]]], state=connected] 2024-09-11T06:29:20.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:20.063 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=137, tag_tid=30, entry_tid=1' 2024-09-11T06:29:20.065 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:29:20.065 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:20.065 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:20.065 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:20.065 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:29:20.065 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:20.065 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:20.065 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:20.065 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:29:20.065 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:29:20.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:29:20.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:29:20.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:20.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:29:20.067 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:20.114 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 34 2024-09-11T06:29:20.114 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 34 2024-09-11T06:29:20.114 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:29:20.114 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=8011], [object_number=138, tag_tid=29, entry_tid=8010]]], state=connected] 2024-09-11T06:29:20.114 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=137, tag_tid=29, entry_tid=5837], [object_number=136, tag_tid=29, entry_tid=5836], [object_number=139, tag_tid=29, entry_tid=5835], [object_number=138, tag_tid=29, entry_tid=5834]]], state=connected] 2024-09-11T06:29:20.117 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:20.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=137, tag_tid=29, entry_tid=5837' 2024-09-11T06:29:20.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=137, tag_tid=30, entry_tid=1' -a 'object_number=137, tag_tid=30, entry_tid=1' = 'object_number=137, tag_tid=29, entry_tid=5837' 2024-09-11T06:29:20.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=137, tag_tid=29, entry_tid=5837' '!=' 'object_number=137, tag_tid=29, entry_tid=5741' 2024-09-11T06:29:20.120 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:29:20.120 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=137, tag_tid=29, entry_tid=5837' = 'object_number=137, tag_tid=29, entry_tid=5741' 2024-09-11T06:29:20.120 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=137, tag_tid=29, entry_tid=5837' 2024-09-11T06:29:20.120 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-09-11T06:29:20.120 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:29:20.121 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:20.121 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=30 2024-09-11T06:29:20.121 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=137, tag_tid=29, entry_tid=5837' 2024-09-11T06:29:20.121 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:29:20.122 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:20.122 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=29 2024-09-11T06:29:20.123 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-09-11T06:29:20.123 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:29:20.123 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:20.124 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:29:20.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=137, tag_tid=29, entry_tid=5837' 2024-09-11T06:29:20.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:29:20.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:20.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=5837 2024-09-11T06:29:20.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 30 = 29 -a 1 -le 5837 2024-09-11T06:29:20.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:29:20.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:29:20.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:29:20.328 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:29:20.328 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:29:20.328 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:29:20.328 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:29:20.328 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:29:20.328 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:29:20.328 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:29:20.328 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:29:20.328 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:29:20.328 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:29:20.328 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:29:20.329 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:29:20.329 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:29:20.329 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:29:20.329 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:29:20.330 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:29:20.331 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:29:20.334 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:29:20.334 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:29:20.334 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:29:20.334 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:29:20.334 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:29:20.334 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:29:20.334 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:29:20.334 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:29:20.334 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:29:20.335 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:29:20.335 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:29:20.335 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:29:20.335 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:29:20.335 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:29:20.335 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:29:20.335 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:29:20.335 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:29:20.335 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:29:20.335 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:29:20.335 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:29:20.335 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:29:20.335 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:29:20.335 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:29:20.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:29:20.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:29:20.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:29:20.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:29:20.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:29:20.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:29:20.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:29:20.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:29:20.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:29:20.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:29:20.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:29:20.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:29:20.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:29:20.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:29:20.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:29:20.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:20.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:20.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:20.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:29:20.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:20.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:20.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:20.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:29:20.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:29:20.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:29:20.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:29:20.499 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:20.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:29:20.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:20.548 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 34 2024-09-11T06:29:20.548 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 34 2024-09-11T06:29:20.549 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:29:20.549 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=8011], [object_number=138, tag_tid=29, entry_tid=8010]]], state=connected] 2024-09-11T06:29:20.549 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=138, tag_tid=29, entry_tid=5938], [object_number=137, tag_tid=29, entry_tid=5937], [object_number=136, tag_tid=29, entry_tid=5936], [object_number=139, tag_tid=29, entry_tid=5935]]], state=connected] 2024-09-11T06:29:20.606 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:20.609 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=137, tag_tid=30, entry_tid=1' 2024-09-11T06:29:20.610 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:29:20.610 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:20.610 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:20.610 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:20.610 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:29:20.610 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:20.610 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:20.610 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:20.610 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:29:20.610 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:29:20.611 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:29:20.611 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:29:20.613 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:20.613 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:29:20.613 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:20.663 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 34 2024-09-11T06:29:20.664 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 34 2024-09-11T06:29:20.664 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:29:20.664 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=8011], [object_number=138, tag_tid=29, entry_tid=8010]]], state=connected] 2024-09-11T06:29:20.664 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=138, tag_tid=29, entry_tid=5938], [object_number=137, tag_tid=29, entry_tid=5937], [object_number=136, tag_tid=29, entry_tid=5936], [object_number=139, tag_tid=29, entry_tid=5935]]], state=connected] 2024-09-11T06:29:20.722 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:20.724 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=138, tag_tid=29, entry_tid=5938' 2024-09-11T06:29:20.724 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=137, tag_tid=30, entry_tid=1' -a 'object_number=137, tag_tid=30, entry_tid=1' = 'object_number=138, tag_tid=29, entry_tid=5938' 2024-09-11T06:29:20.724 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=138, tag_tid=29, entry_tid=5938' '!=' 'object_number=137, tag_tid=29, entry_tid=5837' 2024-09-11T06:29:20.724 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:29:20.724 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=138, tag_tid=29, entry_tid=5938' = 'object_number=137, tag_tid=29, entry_tid=5837' 2024-09-11T06:29:20.724 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=138, tag_tid=29, entry_tid=5938' 2024-09-11T06:29:20.725 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-09-11T06:29:20.725 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:29:20.725 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:20.727 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=30 2024-09-11T06:29:20.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=138, tag_tid=29, entry_tid=5938' 2024-09-11T06:29:20.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:29:20.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:20.730 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=29 2024-09-11T06:29:20.730 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-09-11T06:29:20.731 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:29:20.731 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:20.732 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:29:20.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=138, tag_tid=29, entry_tid=5938' 2024-09-11T06:29:20.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:29:20.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:20.735 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=5938 2024-09-11T06:29:20.735 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 30 = 29 -a 1 -le 5938 2024-09-11T06:29:20.735 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:29:20.735 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:29:20.735 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:29:20.937 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:29:20.937 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:29:20.937 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:29:20.937 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:29:20.937 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:29:20.937 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:29:20.937 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:29:20.937 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:29:20.938 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:29:20.938 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:29:20.938 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:29:20.938 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:29:20.938 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:29:20.938 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:29:20.938 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:29:20.938 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:29:20.939 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:29:20.942 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:29:20.942 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:29:20.942 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:29:20.943 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:29:20.943 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:29:20.943 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:29:20.943 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:29:20.943 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:29:20.943 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:29:20.943 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:29:20.943 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:29:20.943 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:29:20.943 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:29:20.943 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:29:20.943 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:29:20.943 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:29:20.943 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:29:20.943 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:29:20.944 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:29:20.944 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:29:20.944 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:29:20.944 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:29:20.944 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:29:20.944 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:29:20.944 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:29:20.944 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:29:20.944 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:29:20.944 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:29:20.944 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:29:20.944 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:29:20.944 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:29:20.944 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:29:20.968 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:29:20.968 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:29:20.968 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:29:20.968 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:29:21.097 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:29:21.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:29:21.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:21.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:21.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:21.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:29:21.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:21.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:21.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:21.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:29:21.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:29:21.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:29:21.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:29:21.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:21.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:29:21.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:21.152 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 34 2024-09-11T06:29:21.152 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 34 2024-09-11T06:29:21.152 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:29:21.152 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=8011], [object_number=138, tag_tid=29, entry_tid=8010]]], state=connected] 2024-09-11T06:29:21.152 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=136, tag_tid=29, entry_tid=6048], [object_number=139, tag_tid=29, entry_tid=6047], [object_number=138, tag_tid=29, entry_tid=6046], [object_number=137, tag_tid=29, entry_tid=6045]]], state=connected] 2024-09-11T06:29:21.155 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:21.157 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=137, tag_tid=30, entry_tid=1' 2024-09-11T06:29:21.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:29:21.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:21.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:21.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:21.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:29:21.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:21.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:21.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:21.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:29:21.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:29:21.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:29:21.159 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:29:21.159 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:21.160 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:29:21.160 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:21.413 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 34 2024-09-11T06:29:21.413 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 34 2024-09-11T06:29:21.413 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:29:21.413 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=8011], [object_number=138, tag_tid=29, entry_tid=8010]]], state=connected] 2024-09-11T06:29:21.413 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=136, tag_tid=29, entry_tid=6048], [object_number=139, tag_tid=29, entry_tid=6047], [object_number=138, tag_tid=29, entry_tid=6046], [object_number=137, tag_tid=29, entry_tid=6045]]], state=connected] 2024-09-11T06:29:21.420 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:21.421 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=136, tag_tid=29, entry_tid=6048' 2024-09-11T06:29:21.421 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=137, tag_tid=30, entry_tid=1' -a 'object_number=137, tag_tid=30, entry_tid=1' = 'object_number=136, tag_tid=29, entry_tid=6048' 2024-09-11T06:29:21.421 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=136, tag_tid=29, entry_tid=6048' '!=' 'object_number=138, tag_tid=29, entry_tid=5938' 2024-09-11T06:29:21.422 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:29:21.422 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=136, tag_tid=29, entry_tid=6048' = 'object_number=138, tag_tid=29, entry_tid=5938' 2024-09-11T06:29:21.422 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=136, tag_tid=29, entry_tid=6048' 2024-09-11T06:29:21.422 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-09-11T06:29:21.422 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:21.422 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:29:21.424 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=30 2024-09-11T06:29:21.425 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=136, tag_tid=29, entry_tid=6048' 2024-09-11T06:29:21.425 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:29:21.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:21.427 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=29 2024-09-11T06:29:21.428 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-09-11T06:29:21.428 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:29:21.428 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:21.430 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:29:21.431 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=136, tag_tid=29, entry_tid=6048' 2024-09-11T06:29:21.431 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:29:21.431 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:21.433 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=6048 2024-09-11T06:29:21.433 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 30 = 29 -a 1 -le 6048 2024-09-11T06:29:21.433 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:29:21.433 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:29:21.434 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:29:21.635 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:29:21.635 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:29:21.635 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:29:21.635 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:29:21.635 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:29:21.636 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:29:21.636 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:29:21.636 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:29:21.636 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:29:21.636 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:29:21.636 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:29:21.636 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:29:21.636 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:29:21.636 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:29:21.636 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:29:21.636 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:29:21.637 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:29:21.640 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:29:21.640 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:29:21.640 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:29:21.640 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:29:21.640 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:29:21.640 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:29:21.640 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:29:21.640 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:29:21.640 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:29:21.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:29:21.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:29:21.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:29:21.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:29:21.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:29:21.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:29:21.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:29:21.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:29:21.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:29:21.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:29:21.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:29:21.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:29:21.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:29:21.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:29:21.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:29:21.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:29:21.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:29:21.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:29:21.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:29:21.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:29:21.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:29:21.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:29:21.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:29:21.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:29:21.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:29:21.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:29:21.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:29:21.810 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:29:21.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:29:21.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:21.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:21.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:21.812 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:29:21.812 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:21.812 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:21.812 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:21.812 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:29:21.812 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:29:21.812 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:29:21.813 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:29:21.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:21.815 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:29:21.815 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:21.868 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 34 2024-09-11T06:29:21.868 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 34 2024-09-11T06:29:21.868 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:29:21.868 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=8011], [object_number=138, tag_tid=29, entry_tid=8010]]], state=connected] 2024-09-11T06:29:21.868 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=137, tag_tid=29, entry_tid=6161], [object_number=136, tag_tid=29, entry_tid=6160], [object_number=139, tag_tid=29, entry_tid=6159], [object_number=138, tag_tid=29, entry_tid=6158]]], state=connected] 2024-09-11T06:29:21.871 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:21.873 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=137, tag_tid=30, entry_tid=1' 2024-09-11T06:29:21.874 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:29:21.874 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:21.874 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:21.874 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:21.874 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:29:21.874 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:21.874 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:21.874 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:21.874 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:29:21.874 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:29:21.874 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:29:21.875 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:29:21.875 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:21.875 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:29:21.875 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:21.923 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 34 2024-09-11T06:29:21.923 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 34 2024-09-11T06:29:21.923 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:29:21.923 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=8011], [object_number=138, tag_tid=29, entry_tid=8010]]], state=connected] 2024-09-11T06:29:21.923 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=137, tag_tid=29, entry_tid=6161], [object_number=136, tag_tid=29, entry_tid=6160], [object_number=139, tag_tid=29, entry_tid=6159], [object_number=138, tag_tid=29, entry_tid=6158]]], state=connected] 2024-09-11T06:29:21.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:21.929 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=137, tag_tid=29, entry_tid=6161' 2024-09-11T06:29:21.929 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=137, tag_tid=30, entry_tid=1' -a 'object_number=137, tag_tid=30, entry_tid=1' = 'object_number=137, tag_tid=29, entry_tid=6161' 2024-09-11T06:29:21.929 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=137, tag_tid=29, entry_tid=6161' '!=' 'object_number=136, tag_tid=29, entry_tid=6048' 2024-09-11T06:29:21.929 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:29:21.929 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=137, tag_tid=29, entry_tid=6161' = 'object_number=136, tag_tid=29, entry_tid=6048' 2024-09-11T06:29:21.929 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=137, tag_tid=29, entry_tid=6161' 2024-09-11T06:29:21.929 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-09-11T06:29:21.929 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:29:21.930 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:21.930 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=30 2024-09-11T06:29:21.931 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=137, tag_tid=29, entry_tid=6161' 2024-09-11T06:29:21.931 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:29:21.931 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:21.932 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=29 2024-09-11T06:29:21.932 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-09-11T06:29:21.932 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:29:21.932 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:21.933 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:29:21.934 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=137, tag_tid=29, entry_tid=6161' 2024-09-11T06:29:21.934 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:29:21.934 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:21.935 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=6161 2024-09-11T06:29:21.935 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 30 = 29 -a 1 -le 6161 2024-09-11T06:29:21.935 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:29:21.935 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:29:21.935 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:29:22.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:29:22.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:29:22.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:29:22.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:29:22.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:29:22.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:29:22.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:29:22.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:29:22.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:29:22.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:29:22.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:29:22.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:29:22.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:29:22.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:29:22.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:29:22.139 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:29:22.139 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:29:22.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:29:22.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:29:22.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:29:22.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:29:22.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:29:22.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:29:22.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:29:22.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:29:22.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:29:22.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:29:22.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:29:22.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:29:22.143 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:29:22.143 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:29:22.143 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:29:22.143 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:29:22.144 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:29:22.144 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:29:22.144 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:29:22.144 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:29:22.144 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:29:22.144 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:29:22.144 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:29:22.144 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:29:22.144 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:29:22.144 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:29:22.144 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:29:22.144 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:29:22.144 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:29:22.145 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:29:22.145 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:29:22.145 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:29:22.169 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:29:22.169 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:29:22.169 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:29:22.169 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:29:22.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:29:22.300 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:29:22.301 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:22.301 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:22.301 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:22.301 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:29:22.301 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:22.301 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:22.301 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:22.301 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:29:22.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:29:22.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:29:22.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:29:22.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:22.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:29:22.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:22.560 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 34 2024-09-11T06:29:22.561 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 34 2024-09-11T06:29:22.561 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:29:22.561 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=8011], [object_number=138, tag_tid=29, entry_tid=8010]]], state=connected] 2024-09-11T06:29:22.561 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=139, tag_tid=29, entry_tid=6247], [object_number=138, tag_tid=29, entry_tid=6246], [object_number=137, tag_tid=29, entry_tid=6245], [object_number=136, tag_tid=29, entry_tid=6244]]], state=connected] 2024-09-11T06:29:22.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:22.569 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=137, tag_tid=30, entry_tid=1' 2024-09-11T06:29:22.570 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:29:22.570 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:22.570 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:22.570 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:22.570 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:29:22.570 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:22.570 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:22.570 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:22.570 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:29:22.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:29:22.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:29:22.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:29:22.573 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:22.573 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:29:22.573 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:22.631 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 34 2024-09-11T06:29:22.631 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 34 2024-09-11T06:29:22.631 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:29:22.631 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=8011], [object_number=138, tag_tid=29, entry_tid=8010]]], state=connected] 2024-09-11T06:29:22.631 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=139, tag_tid=29, entry_tid=6247], [object_number=138, tag_tid=29, entry_tid=6246], [object_number=137, tag_tid=29, entry_tid=6245], [object_number=136, tag_tid=29, entry_tid=6244]]], state=connected] 2024-09-11T06:29:22.634 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:22.636 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=139, tag_tid=29, entry_tid=6247' 2024-09-11T06:29:22.636 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=137, tag_tid=30, entry_tid=1' -a 'object_number=137, tag_tid=30, entry_tid=1' = 'object_number=139, tag_tid=29, entry_tid=6247' 2024-09-11T06:29:22.637 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=139, tag_tid=29, entry_tid=6247' '!=' 'object_number=137, tag_tid=29, entry_tid=6161' 2024-09-11T06:29:22.637 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:29:22.637 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=139, tag_tid=29, entry_tid=6247' = 'object_number=137, tag_tid=29, entry_tid=6161' 2024-09-11T06:29:22.637 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=139, tag_tid=29, entry_tid=6247' 2024-09-11T06:29:22.638 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-09-11T06:29:22.638 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:29:22.638 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:22.638 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=30 2024-09-11T06:29:22.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=139, tag_tid=29, entry_tid=6247' 2024-09-11T06:29:22.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:29:22.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:22.640 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=29 2024-09-11T06:29:22.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-09-11T06:29:22.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:29:22.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:22.645 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:29:22.645 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=139, tag_tid=29, entry_tid=6247' 2024-09-11T06:29:22.646 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:29:22.646 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:22.646 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=6247 2024-09-11T06:29:22.646 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 30 = 29 -a 1 -le 6247 2024-09-11T06:29:22.646 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:29:22.646 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:29:22.646 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:29:22.846 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:29:22.846 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:29:22.846 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:29:22.846 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:29:22.846 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:29:22.846 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:29:22.846 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:29:22.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:29:22.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:29:22.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:29:22.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:29:22.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:29:22.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:29:22.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:29:22.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:29:22.848 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:29:22.848 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:29:22.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:29:22.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:29:22.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:29:22.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:29:22.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:29:22.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:29:22.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:29:22.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:29:22.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:29:22.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:29:22.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:29:22.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:29:22.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:29:22.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:29:22.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:29:22.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:29:22.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:29:22.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:29:22.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:29:22.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:29:22.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:29:22.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:29:22.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:29:22.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:29:22.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:29:22.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:29:22.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:29:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:29:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:29:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:29:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:29:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:29:22.877 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:29:22.878 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:29:22.878 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:29:22.878 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:29:23.004 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:29:23.005 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:29:23.005 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:23.005 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:23.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:23.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:29:23.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:23.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:23.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:23.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:29:23.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:29:23.007 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:29:23.007 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:29:23.007 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:23.008 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:29:23.008 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:23.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 34 2024-09-11T06:29:23.067 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 34 2024-09-11T06:29:23.067 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:29:23.067 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=8011], [object_number=138, tag_tid=29, entry_tid=8010]]], state=connected] 2024-09-11T06:29:23.067 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=138, tag_tid=29, entry_tid=6366], [object_number=137, tag_tid=29, entry_tid=6365], [object_number=136, tag_tid=29, entry_tid=6364], [object_number=139, tag_tid=29, entry_tid=6363]]], state=connected] 2024-09-11T06:29:23.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:23.071 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=137, tag_tid=30, entry_tid=1' 2024-09-11T06:29:23.072 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:29:23.073 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:23.073 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:23.073 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:23.073 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:29:23.073 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:23.073 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:23.073 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:23.073 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:29:23.073 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:29:23.073 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:29:23.073 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:29:23.074 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:23.074 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:29:23.074 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:23.123 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 34 2024-09-11T06:29:23.123 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 34 2024-09-11T06:29:23.123 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:29:23.123 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=8011], [object_number=138, tag_tid=29, entry_tid=8010]]], state=connected] 2024-09-11T06:29:23.123 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=138, tag_tid=29, entry_tid=6366], [object_number=137, tag_tid=29, entry_tid=6365], [object_number=136, tag_tid=29, entry_tid=6364], [object_number=139, tag_tid=29, entry_tid=6363]]], state=connected] 2024-09-11T06:29:23.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:23.129 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=138, tag_tid=29, entry_tid=6366' 2024-09-11T06:29:23.129 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=137, tag_tid=30, entry_tid=1' -a 'object_number=137, tag_tid=30, entry_tid=1' = 'object_number=138, tag_tid=29, entry_tid=6366' 2024-09-11T06:29:23.129 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=138, tag_tid=29, entry_tid=6366' '!=' 'object_number=139, tag_tid=29, entry_tid=6247' 2024-09-11T06:29:23.129 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:29:23.129 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=138, tag_tid=29, entry_tid=6366' = 'object_number=139, tag_tid=29, entry_tid=6247' 2024-09-11T06:29:23.129 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=138, tag_tid=29, entry_tid=6366' 2024-09-11T06:29:23.130 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-09-11T06:29:23.130 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:29:23.130 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:23.130 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=30 2024-09-11T06:29:23.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=138, tag_tid=29, entry_tid=6366' 2024-09-11T06:29:23.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:29:23.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:23.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=29 2024-09-11T06:29:23.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-09-11T06:29:23.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:29:23.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:23.133 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:29:23.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=138, tag_tid=29, entry_tid=6366' 2024-09-11T06:29:23.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:29:23.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:23.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=6366 2024-09-11T06:29:23.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 30 = 29 -a 1 -le 6366 2024-09-11T06:29:23.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:29:23.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:29:23.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:29:23.337 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:29:23.337 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:29:23.338 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:29:23.338 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:29:23.338 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:29:23.338 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:29:23.338 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:29:23.338 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:29:23.338 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:29:23.338 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:29:23.338 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:29:23.338 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:29:23.338 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:29:23.338 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:29:23.338 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:29:23.339 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:29:23.339 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:29:23.343 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:29:23.343 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:29:23.343 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:29:23.343 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:29:23.343 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:29:23.343 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:29:23.343 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:29:23.343 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:29:23.343 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:29:23.343 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:29:23.344 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:29:23.344 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:29:23.344 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:29:23.344 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:29:23.344 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:29:23.344 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:29:23.344 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:29:23.344 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:29:23.344 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:29:23.344 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:29:23.344 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:29:23.344 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:29:23.344 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:29:23.344 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:29:23.345 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:29:23.345 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:29:23.345 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:29:23.345 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:29:23.345 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:29:23.345 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:29:23.345 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:29:23.345 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:29:23.368 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:29:23.368 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:29:23.368 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:29:23.368 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:29:23.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:29:23.498 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:29:23.498 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:23.498 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:23.498 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:23.498 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:29:23.498 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:23.498 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:23.498 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:23.498 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:29:23.499 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:29:23.499 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:29:23.499 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:29:23.501 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:23.502 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:29:23.502 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:23.561 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 34 2024-09-11T06:29:23.561 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 34 2024-09-11T06:29:23.561 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:29:23.561 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=8011], [object_number=138, tag_tid=29, entry_tid=8010]]], state=connected] 2024-09-11T06:29:23.561 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=136, tag_tid=29, entry_tid=6452], [object_number=139, tag_tid=29, entry_tid=6451], [object_number=138, tag_tid=29, entry_tid=6450], [object_number=137, tag_tid=29, entry_tid=6449]]], state=connected] 2024-09-11T06:29:23.564 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:23.565 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=137, tag_tid=30, entry_tid=1' 2024-09-11T06:29:23.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:29:23.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:23.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:23.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:23.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:29:23.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:23.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:23.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:23.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:29:23.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:29:23.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:29:23.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:29:23.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:23.569 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:29:23.569 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:23.617 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 34 2024-09-11T06:29:23.617 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 34 2024-09-11T06:29:23.618 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:29:23.618 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=8011], [object_number=138, tag_tid=29, entry_tid=8010]]], state=connected] 2024-09-11T06:29:23.618 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=136, tag_tid=29, entry_tid=6452], [object_number=139, tag_tid=29, entry_tid=6451], [object_number=138, tag_tid=29, entry_tid=6450], [object_number=137, tag_tid=29, entry_tid=6449]]], state=connected] 2024-09-11T06:29:23.621 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:23.623 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=136, tag_tid=29, entry_tid=6452' 2024-09-11T06:29:23.623 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=137, tag_tid=30, entry_tid=1' -a 'object_number=137, tag_tid=30, entry_tid=1' = 'object_number=136, tag_tid=29, entry_tid=6452' 2024-09-11T06:29:23.623 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=136, tag_tid=29, entry_tid=6452' '!=' 'object_number=138, tag_tid=29, entry_tid=6366' 2024-09-11T06:29:23.623 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:29:23.623 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=136, tag_tid=29, entry_tid=6452' = 'object_number=138, tag_tid=29, entry_tid=6366' 2024-09-11T06:29:23.623 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=136, tag_tid=29, entry_tid=6452' 2024-09-11T06:29:23.624 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-09-11T06:29:23.624 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:29:23.624 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:23.624 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=30 2024-09-11T06:29:23.625 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=136, tag_tid=29, entry_tid=6452' 2024-09-11T06:29:23.625 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:29:23.625 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:23.626 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=29 2024-09-11T06:29:23.627 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-09-11T06:29:23.627 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:29:23.627 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:23.628 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:29:23.628 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=136, tag_tid=29, entry_tid=6452' 2024-09-11T06:29:23.629 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:29:23.629 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:23.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=6452 2024-09-11T06:29:23.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 30 = 29 -a 1 -le 6452 2024-09-11T06:29:23.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:29:23.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:29:23.631 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:29:23.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:29:23.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:29:23.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:29:23.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:29:23.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:29:23.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:29:23.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:29:23.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:29:23.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:29:23.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:29:23.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:29:23.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:29:23.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:29:23.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:29:23.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:29:23.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:29:23.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:29:23.836 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:29:23.836 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:29:23.836 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:29:23.836 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:29:23.836 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:29:23.837 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:29:23.837 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:29:23.837 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:29:23.837 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:29:23.837 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:29:23.837 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:29:23.837 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:29:23.837 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:29:23.837 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:29:23.837 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:29:23.837 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:29:23.837 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:29:23.837 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:29:23.838 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:29:23.838 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:29:23.838 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:29:23.838 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:29:23.838 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:29:23.838 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:29:23.838 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:29:23.838 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:29:23.838 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:29:23.838 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:29:23.838 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:29:23.838 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:29:23.838 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:29:23.839 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:29:23.863 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:29:23.863 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:29:23.863 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:29:23.863 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:29:23.992 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:29:23.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:29:23.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:23.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:23.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:23.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:29:23.994 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:23.994 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:23.994 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:23.994 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:29:23.994 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:29:23.994 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:29:23.995 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:29:23.997 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:23.997 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:29:23.997 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:24.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 34 2024-09-11T06:29:24.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 34 2024-09-11T06:29:24.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:29:24.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=8011], [object_number=138, tag_tid=29, entry_tid=8010]]], state=connected] 2024-09-11T06:29:24.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=138, tag_tid=29, entry_tid=6526], [object_number=137, tag_tid=29, entry_tid=6525], [object_number=136, tag_tid=29, entry_tid=6524], [object_number=139, tag_tid=29, entry_tid=6523]]], state=connected] 2024-09-11T06:29:24.058 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:24.060 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=137, tag_tid=30, entry_tid=1' 2024-09-11T06:29:24.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:29:24.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:24.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:24.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:24.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:29:24.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:24.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:24.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:24.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:29:24.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:29:24.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:29:24.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:29:24.063 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:24.063 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:29:24.063 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:24.113 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 34 2024-09-11T06:29:24.113 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 34 2024-09-11T06:29:24.113 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:29:24.113 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=8011], [object_number=138, tag_tid=29, entry_tid=8010]]], state=connected] 2024-09-11T06:29:24.113 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=138, tag_tid=29, entry_tid=6526], [object_number=137, tag_tid=29, entry_tid=6525], [object_number=136, tag_tid=29, entry_tid=6524], [object_number=139, tag_tid=29, entry_tid=6523]]], state=connected] 2024-09-11T06:29:24.117 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:24.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=138, tag_tid=29, entry_tid=6526' 2024-09-11T06:29:24.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=137, tag_tid=30, entry_tid=1' -a 'object_number=137, tag_tid=30, entry_tid=1' = 'object_number=138, tag_tid=29, entry_tid=6526' 2024-09-11T06:29:24.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=138, tag_tid=29, entry_tid=6526' '!=' 'object_number=136, tag_tid=29, entry_tid=6452' 2024-09-11T06:29:24.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:29:24.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=138, tag_tid=29, entry_tid=6526' = 'object_number=136, tag_tid=29, entry_tid=6452' 2024-09-11T06:29:24.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=138, tag_tid=29, entry_tid=6526' 2024-09-11T06:29:24.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-09-11T06:29:24.120 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:29:24.120 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:24.120 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=30 2024-09-11T06:29:24.121 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=138, tag_tid=29, entry_tid=6526' 2024-09-11T06:29:24.121 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:29:24.121 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:24.122 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=29 2024-09-11T06:29:24.122 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-09-11T06:29:24.123 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:29:24.123 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:24.123 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:29:24.124 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=138, tag_tid=29, entry_tid=6526' 2024-09-11T06:29:24.124 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:29:24.124 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:24.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=6526 2024-09-11T06:29:24.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 30 = 29 -a 1 -le 6526 2024-09-11T06:29:24.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:29:24.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:29:24.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:29:24.327 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:29:24.327 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:29:24.327 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:29:24.328 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:29:24.328 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:29:24.328 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:29:24.328 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:29:24.328 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:29:24.328 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:29:24.328 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:29:24.328 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:29:24.328 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:29:24.328 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:29:24.328 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:29:24.328 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:29:24.329 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:29:24.329 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:29:24.333 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:29:24.333 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:29:24.333 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:29:24.333 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:29:24.333 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:29:24.333 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:29:24.334 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:29:24.334 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:29:24.334 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:29:24.334 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:29:24.334 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:29:24.334 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:29:24.334 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:29:24.334 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:29:24.334 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:29:24.334 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:29:24.334 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:29:24.334 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:29:24.334 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:29:24.334 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:29:24.335 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:29:24.335 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:29:24.335 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:29:24.335 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:29:24.335 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:29:24.335 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:29:24.335 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:29:24.335 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:29:24.335 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:29:24.335 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:29:24.335 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:29:24.335 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:29:24.357 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:29:24.358 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:29:24.358 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:29:24.358 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:29:24.484 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:29:24.486 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:29:24.486 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:24.486 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:24.486 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:24.486 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:29:24.486 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:24.486 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:24.486 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:24.486 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:29:24.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:29:24.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:29:24.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:29:24.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:24.490 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:29:24.490 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:24.541 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 34 2024-09-11T06:29:24.541 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 34 2024-09-11T06:29:24.541 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:29:24.541 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=8011], [object_number=138, tag_tid=29, entry_tid=8010]]], state=connected] 2024-09-11T06:29:24.541 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=137, tag_tid=29, entry_tid=6609], [object_number=136, tag_tid=29, entry_tid=6608], [object_number=139, tag_tid=29, entry_tid=6607], [object_number=138, tag_tid=29, entry_tid=6606]]], state=connected] 2024-09-11T06:29:24.544 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:24.545 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=137, tag_tid=30, entry_tid=1' 2024-09-11T06:29:24.546 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:29:24.547 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:24.547 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:24.547 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:24.547 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:29:24.547 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:24.547 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:24.547 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:24.547 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:29:24.547 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:29:24.547 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:29:24.547 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:29:24.548 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:24.548 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:29:24.548 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:24.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 34 2024-09-11T06:29:24.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 34 2024-09-11T06:29:24.596 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:29:24.596 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=8011], [object_number=138, tag_tid=29, entry_tid=8010]]], state=connected] 2024-09-11T06:29:24.596 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=137, tag_tid=29, entry_tid=6609], [object_number=136, tag_tid=29, entry_tid=6608], [object_number=139, tag_tid=29, entry_tid=6607], [object_number=138, tag_tid=29, entry_tid=6606]]], state=connected] 2024-09-11T06:29:24.599 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:24.601 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=137, tag_tid=29, entry_tid=6609' 2024-09-11T06:29:24.601 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=137, tag_tid=30, entry_tid=1' -a 'object_number=137, tag_tid=30, entry_tid=1' = 'object_number=137, tag_tid=29, entry_tid=6609' 2024-09-11T06:29:24.601 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=137, tag_tid=29, entry_tid=6609' '!=' 'object_number=138, tag_tid=29, entry_tid=6526' 2024-09-11T06:29:24.601 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:29:24.601 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=137, tag_tid=29, entry_tid=6609' = 'object_number=138, tag_tid=29, entry_tid=6526' 2024-09-11T06:29:24.601 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=137, tag_tid=29, entry_tid=6609' 2024-09-11T06:29:24.601 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-09-11T06:29:24.602 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:29:24.602 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:24.602 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=30 2024-09-11T06:29:24.603 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=137, tag_tid=29, entry_tid=6609' 2024-09-11T06:29:24.603 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:29:24.603 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:24.604 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=29 2024-09-11T06:29:24.604 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-09-11T06:29:24.604 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:29:24.604 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:24.605 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:29:24.606 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=137, tag_tid=29, entry_tid=6609' 2024-09-11T06:29:24.606 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:29:24.606 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:24.607 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=6609 2024-09-11T06:29:24.608 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 30 = 29 -a 1 -le 6609 2024-09-11T06:29:24.608 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:29:24.608 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:29:24.608 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:29:24.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:29:24.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:29:24.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:29:24.810 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:29:24.810 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:29:24.810 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:29:24.810 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:29:24.810 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:29:24.810 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:29:24.810 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:29:24.810 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:29:24.810 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:29:24.810 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:29:24.810 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:29:24.810 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:29:24.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:29:24.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:29:24.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:29:24.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:29:24.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:29:24.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:29:24.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:29:24.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:29:24.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:29:24.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:29:24.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:29:24.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:29:24.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:29:24.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:29:24.815 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:29:24.815 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:29:24.815 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:29:24.815 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:29:24.815 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:29:24.815 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:29:24.815 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:29:24.815 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:29:24.815 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:29:24.815 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:29:24.815 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:29:24.815 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:29:24.815 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:29:24.815 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:29:24.816 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:29:24.816 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:29:24.816 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:29:24.816 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:29:24.816 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:29:24.816 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:29:24.838 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:29:24.839 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:29:24.839 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:29:24.839 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:29:24.967 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:29:24.969 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:29:24.969 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:24.969 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:24.969 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:24.969 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:29:24.969 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:24.969 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:24.969 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:24.969 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:29:24.969 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:29:24.970 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:29:24.970 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:29:24.971 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:24.971 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:29:24.971 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:25.240 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 34 2024-09-11T06:29:25.240 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 34 2024-09-11T06:29:25.240 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:29:25.240 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=8011], [object_number=138, tag_tid=29, entry_tid=8010]]], state=connected] 2024-09-11T06:29:25.240 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=137, tag_tid=29, entry_tid=6697], [object_number=136, tag_tid=29, entry_tid=6696], [object_number=139, tag_tid=29, entry_tid=6695], [object_number=138, tag_tid=29, entry_tid=6694]]], state=connected] 2024-09-11T06:29:25.244 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:25.246 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=137, tag_tid=30, entry_tid=1' 2024-09-11T06:29:25.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:29:25.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:25.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:25.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:25.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:29:25.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:25.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:25.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:25.248 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:29:25.248 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:29:25.248 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:29:25.248 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:29:25.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:25.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:29:25.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:25.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 34 2024-09-11T06:29:25.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 34 2024-09-11T06:29:25.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:29:25.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=8011], [object_number=138, tag_tid=29, entry_tid=8010]]], state=connected] 2024-09-11T06:29:25.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=137, tag_tid=29, entry_tid=6697], [object_number=136, tag_tid=29, entry_tid=6696], [object_number=139, tag_tid=29, entry_tid=6695], [object_number=138, tag_tid=29, entry_tid=6694]]], state=connected] 2024-09-11T06:29:25.306 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:25.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=137, tag_tid=29, entry_tid=6697' 2024-09-11T06:29:25.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=137, tag_tid=30, entry_tid=1' -a 'object_number=137, tag_tid=30, entry_tid=1' = 'object_number=137, tag_tid=29, entry_tid=6697' 2024-09-11T06:29:25.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=137, tag_tid=29, entry_tid=6697' '!=' 'object_number=137, tag_tid=29, entry_tid=6609' 2024-09-11T06:29:25.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:29:25.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=137, tag_tid=29, entry_tid=6697' = 'object_number=137, tag_tid=29, entry_tid=6609' 2024-09-11T06:29:25.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=137, tag_tid=29, entry_tid=6697' 2024-09-11T06:29:25.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-09-11T06:29:25.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:29:25.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:25.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=30 2024-09-11T06:29:25.310 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=137, tag_tid=29, entry_tid=6697' 2024-09-11T06:29:25.310 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:29:25.310 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:25.311 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=29 2024-09-11T06:29:25.312 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-09-11T06:29:25.312 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:29:25.312 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:25.313 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:29:25.314 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=137, tag_tid=29, entry_tid=6697' 2024-09-11T06:29:25.314 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:29:25.314 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:25.315 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=6697 2024-09-11T06:29:25.315 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 30 = 29 -a 1 -le 6697 2024-09-11T06:29:25.315 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:29:25.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:29:25.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:29:25.517 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:29:25.517 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:29:25.517 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:29:25.517 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:29:25.518 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:29:25.518 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:29:25.518 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:29:25.518 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:29:25.518 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:29:25.518 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:29:25.518 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:29:25.518 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:29:25.518 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:29:25.518 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:29:25.518 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:29:25.519 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:29:25.519 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:29:25.522 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:29:25.522 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:29:25.522 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:29:25.522 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:29:25.522 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:29:25.522 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:29:25.522 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:29:25.523 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:29:25.523 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:29:25.523 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:29:25.523 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:29:25.523 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:29:25.523 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:29:25.523 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:29:25.523 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:29:25.523 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:29:25.523 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:29:25.523 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:29:25.523 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:29:25.523 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:29:25.523 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:29:25.524 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:29:25.524 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:29:25.524 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:29:25.524 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:29:25.524 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:29:25.524 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:29:25.524 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:29:25.524 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:29:25.524 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:29:25.524 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:29:25.524 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:29:25.547 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:29:25.548 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:29:25.548 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:29:25.548 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:29:25.680 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:29:25.681 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:29:25.681 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:25.681 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:25.681 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:25.681 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:29:25.681 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:25.681 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:25.681 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:25.681 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:29:25.682 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:29:25.682 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:29:25.682 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:29:25.683 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:25.683 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:29:25.683 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:25.732 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 34 2024-09-11T06:29:25.732 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 34 2024-09-11T06:29:25.732 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:29:25.732 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=8011], [object_number=138, tag_tid=29, entry_tid=8010]]], state=connected] 2024-09-11T06:29:25.732 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=138, tag_tid=29, entry_tid=6834], [object_number=137, tag_tid=29, entry_tid=6833], [object_number=136, tag_tid=29, entry_tid=6832], [object_number=139, tag_tid=29, entry_tid=6831]]], state=connected] 2024-09-11T06:29:25.735 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:25.736 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=137, tag_tid=30, entry_tid=1' 2024-09-11T06:29:25.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:29:25.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:25.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:25.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:25.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:29:25.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:25.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:25.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:25.739 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:29:25.739 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:29:25.739 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:29:25.739 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:29:25.739 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:25.739 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:29:25.740 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:25.788 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 34 2024-09-11T06:29:25.788 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 34 2024-09-11T06:29:25.788 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:29:25.789 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=8011], [object_number=138, tag_tid=29, entry_tid=8010]]], state=connected] 2024-09-11T06:29:25.789 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=138, tag_tid=29, entry_tid=6834], [object_number=137, tag_tid=29, entry_tid=6833], [object_number=136, tag_tid=29, entry_tid=6832], [object_number=139, tag_tid=29, entry_tid=6831]]], state=connected] 2024-09-11T06:29:25.791 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:25.794 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=138, tag_tid=29, entry_tid=6834' 2024-09-11T06:29:25.794 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=137, tag_tid=30, entry_tid=1' -a 'object_number=137, tag_tid=30, entry_tid=1' = 'object_number=138, tag_tid=29, entry_tid=6834' 2024-09-11T06:29:25.794 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=138, tag_tid=29, entry_tid=6834' '!=' 'object_number=137, tag_tid=29, entry_tid=6697' 2024-09-11T06:29:25.794 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:29:25.794 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=138, tag_tid=29, entry_tid=6834' = 'object_number=137, tag_tid=29, entry_tid=6697' 2024-09-11T06:29:25.794 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=138, tag_tid=29, entry_tid=6834' 2024-09-11T06:29:25.794 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-09-11T06:29:25.794 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:29:25.794 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:25.795 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=30 2024-09-11T06:29:25.795 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=138, tag_tid=29, entry_tid=6834' 2024-09-11T06:29:25.796 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:29:25.796 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:25.796 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=29 2024-09-11T06:29:25.797 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-09-11T06:29:25.797 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:29:25.797 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:25.798 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:29:25.798 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=138, tag_tid=29, entry_tid=6834' 2024-09-11T06:29:25.799 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:29:25.799 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:25.800 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=6834 2024-09-11T06:29:25.800 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 30 = 29 -a 1 -le 6834 2024-09-11T06:29:25.800 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:29:25.800 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:29:25.800 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:29:26.002 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:29:26.002 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:29:26.002 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:29:26.002 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:29:26.002 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:29:26.002 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:29:26.002 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:29:26.002 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:29:26.002 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:29:26.003 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:29:26.003 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:29:26.003 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:29:26.003 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:29:26.003 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:29:26.003 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:29:26.003 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:29:26.004 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:29:26.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:29:26.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:29:26.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:29:26.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:29:26.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:29:26.007 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:29:26.007 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:29:26.007 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:29:26.007 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:29:26.007 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:29:26.007 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:29:26.007 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:29:26.007 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:29:26.007 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:29:26.007 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:29:26.007 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:29:26.008 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:29:26.008 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:29:26.008 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:29:26.009 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:29:26.009 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:29:26.009 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:29:26.009 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:29:26.009 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:29:26.009 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:29:26.009 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:29:26.009 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:29:26.009 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:29:26.009 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:29:26.009 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:29:26.009 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:29:26.010 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:29:26.029 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:29:26.030 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:29:26.030 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:29:26.030 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:29:26.156 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:29:26.157 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:29:26.157 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:26.157 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:26.157 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:26.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:29:26.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:26.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:26.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:26.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:29:26.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:29:26.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:29:26.159 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:29:26.160 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:26.160 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:29:26.160 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:26.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 34 2024-09-11T06:29:26.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 34 2024-09-11T06:29:26.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:29:26.221 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=8011], [object_number=138, tag_tid=29, entry_tid=8010]]], state=connected] 2024-09-11T06:29:26.221 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=139, tag_tid=29, entry_tid=6923], [object_number=138, tag_tid=29, entry_tid=6922], [object_number=137, tag_tid=29, entry_tid=6921], [object_number=136, tag_tid=29, entry_tid=6920]]], state=connected] 2024-09-11T06:29:26.224 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:26.225 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=137, tag_tid=30, entry_tid=1' 2024-09-11T06:29:26.226 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:29:26.226 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:26.226 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:26.226 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:26.226 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:29:26.227 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:26.227 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:26.227 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:26.227 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:29:26.227 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:29:26.227 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:29:26.227 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:29:26.228 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:26.228 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:29:26.228 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:26.276 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 34 2024-09-11T06:29:26.276 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 34 2024-09-11T06:29:26.276 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:29:26.276 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=8011], [object_number=138, tag_tid=29, entry_tid=8010]]], state=connected] 2024-09-11T06:29:26.276 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=139, tag_tid=29, entry_tid=6923], [object_number=138, tag_tid=29, entry_tid=6922], [object_number=137, tag_tid=29, entry_tid=6921], [object_number=136, tag_tid=29, entry_tid=6920]]], state=connected] 2024-09-11T06:29:26.279 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:26.282 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=139, tag_tid=29, entry_tid=6923' 2024-09-11T06:29:26.282 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=137, tag_tid=30, entry_tid=1' -a 'object_number=137, tag_tid=30, entry_tid=1' = 'object_number=139, tag_tid=29, entry_tid=6923' 2024-09-11T06:29:26.282 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=139, tag_tid=29, entry_tid=6923' '!=' 'object_number=138, tag_tid=29, entry_tid=6834' 2024-09-11T06:29:26.282 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:29:26.282 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=139, tag_tid=29, entry_tid=6923' = 'object_number=138, tag_tid=29, entry_tid=6834' 2024-09-11T06:29:26.282 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=139, tag_tid=29, entry_tid=6923' 2024-09-11T06:29:26.282 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-09-11T06:29:26.282 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:29:26.282 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:26.283 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=30 2024-09-11T06:29:26.284 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=139, tag_tid=29, entry_tid=6923' 2024-09-11T06:29:26.284 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:29:26.284 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:26.284 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=29 2024-09-11T06:29:26.285 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-09-11T06:29:26.285 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:29:26.285 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:26.286 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:29:26.287 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=139, tag_tid=29, entry_tid=6923' 2024-09-11T06:29:26.287 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:29:26.287 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:26.288 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=6923 2024-09-11T06:29:26.288 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 30 = 29 -a 1 -le 6923 2024-09-11T06:29:26.288 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:29:26.288 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:29:26.289 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:29:26.490 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:29:26.490 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:29:26.490 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:29:26.491 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:29:26.491 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:29:26.491 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:29:26.491 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:29:26.491 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:29:26.491 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:29:26.491 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:29:26.491 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:29:26.491 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:29:26.491 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:29:26.491 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:29:26.492 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:29:26.492 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:29:26.492 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:29:26.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:29:26.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:29:26.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:29:26.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:29:26.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:29:26.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:29:26.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:29:26.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:29:26.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:29:26.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:29:26.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:29:26.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:29:26.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:29:26.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:29:26.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:29:26.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:29:26.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:29:26.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:29:26.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:29:26.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:29:26.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:29:26.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:29:26.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:29:26.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:29:26.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:29:26.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:29:26.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:29:26.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:29:26.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:29:26.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:29:26.498 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:29:26.498 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:29:26.522 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:29:26.522 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:29:26.522 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:29:26.522 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:29:26.652 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:29:26.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:29:26.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:26.654 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:26.654 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:26.654 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:29:26.654 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:26.654 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:26.654 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:26.654 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:29:26.655 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:29:26.655 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:29:26.655 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:29:26.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:26.658 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:29:26.658 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:26.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 34 2024-09-11T06:29:26.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 34 2024-09-11T06:29:26.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:29:26.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=8011], [object_number=138, tag_tid=29, entry_tid=8010]]], state=connected] 2024-09-11T06:29:26.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=138, tag_tid=29, entry_tid=7002], [object_number=137, tag_tid=29, entry_tid=7001], [object_number=136, tag_tid=29, entry_tid=7000], [object_number=139, tag_tid=29, entry_tid=6999]]], state=connected] 2024-09-11T06:29:26.708 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:26.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=137, tag_tid=30, entry_tid=1' 2024-09-11T06:29:26.711 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:29:26.711 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:26.711 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:26.711 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:26.711 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:29:26.711 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:26.711 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:26.711 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:26.711 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:29:26.711 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:29:26.712 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:29:26.712 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:29:26.712 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:26.712 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:29:26.712 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:26.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 34 2024-09-11T06:29:26.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 34 2024-09-11T06:29:26.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:29:26.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=8011], [object_number=138, tag_tid=29, entry_tid=8010]]], state=connected] 2024-09-11T06:29:26.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=138, tag_tid=29, entry_tid=7002], [object_number=137, tag_tid=29, entry_tid=7001], [object_number=136, tag_tid=29, entry_tid=7000], [object_number=139, tag_tid=29, entry_tid=6999]]], state=connected] 2024-09-11T06:29:26.765 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:26.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=138, tag_tid=29, entry_tid=7002' 2024-09-11T06:29:26.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=137, tag_tid=30, entry_tid=1' -a 'object_number=137, tag_tid=30, entry_tid=1' = 'object_number=138, tag_tid=29, entry_tid=7002' 2024-09-11T06:29:26.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=138, tag_tid=29, entry_tid=7002' '!=' 'object_number=139, tag_tid=29, entry_tid=6923' 2024-09-11T06:29:26.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:29:26.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=138, tag_tid=29, entry_tid=7002' = 'object_number=139, tag_tid=29, entry_tid=6923' 2024-09-11T06:29:26.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=138, tag_tid=29, entry_tid=7002' 2024-09-11T06:29:26.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-09-11T06:29:26.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:29:26.768 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:26.768 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=30 2024-09-11T06:29:26.769 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=138, tag_tid=29, entry_tid=7002' 2024-09-11T06:29:26.769 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:29:26.769 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:26.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=29 2024-09-11T06:29:26.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-09-11T06:29:26.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:29:26.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:26.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:29:26.772 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=138, tag_tid=29, entry_tid=7002' 2024-09-11T06:29:26.772 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:29:26.772 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:26.773 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=7002 2024-09-11T06:29:26.774 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 30 = 29 -a 1 -le 7002 2024-09-11T06:29:26.774 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:29:26.774 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:29:26.774 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:29:26.975 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:29:26.976 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:29:26.976 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:29:26.976 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:29:26.976 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:29:26.976 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:29:26.976 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:29:26.976 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:29:26.976 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:29:26.976 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:29:26.976 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:29:26.977 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:29:26.977 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:29:26.977 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:29:26.977 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:29:26.977 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:29:26.977 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:29:26.980 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:29:26.980 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:29:26.980 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:29:26.980 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:29:26.980 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:29:26.980 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:29:26.981 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:29:26.981 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:29:26.981 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:29:26.981 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:29:26.981 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:29:26.981 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:29:26.981 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:29:26.981 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:29:26.981 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:29:26.981 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:29:26.981 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:29:26.981 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:29:26.982 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:29:26.982 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:29:26.982 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:29:26.982 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:29:26.982 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:29:26.982 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:29:26.982 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:29:26.982 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:29:26.982 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:29:26.982 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:29:26.982 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:29:26.982 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:29:26.982 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:29:26.982 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:29:27.005 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:29:27.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:29:27.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:29:27.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:29:27.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:29:27.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:29:27.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:27.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:27.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:27.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:29:27.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:27.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:27.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:27.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:29:27.139 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:29:27.139 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:29:27.139 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:29:27.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:27.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:29:27.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:27.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 34 2024-09-11T06:29:27.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 34 2024-09-11T06:29:27.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:29:27.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=8011], [object_number=138, tag_tid=29, entry_tid=8010]]], state=connected] 2024-09-11T06:29:27.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=138, tag_tid=29, entry_tid=7082], [object_number=137, tag_tid=29, entry_tid=7081], [object_number=136, tag_tid=29, entry_tid=7080], [object_number=139, tag_tid=29, entry_tid=7079]]], state=connected] 2024-09-11T06:29:27.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:27.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=137, tag_tid=30, entry_tid=1' 2024-09-11T06:29:27.196 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:29:27.197 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:27.197 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:27.197 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:27.197 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:29:27.197 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:27.197 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:27.197 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:27.197 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:29:27.197 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:29:27.197 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:29:27.197 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:29:27.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:27.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:29:27.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:27.246 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 34 2024-09-11T06:29:27.246 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 34 2024-09-11T06:29:27.246 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:29:27.246 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=8011], [object_number=138, tag_tid=29, entry_tid=8010]]], state=connected] 2024-09-11T06:29:27.246 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=138, tag_tid=29, entry_tid=7082], [object_number=137, tag_tid=29, entry_tid=7081], [object_number=136, tag_tid=29, entry_tid=7080], [object_number=139, tag_tid=29, entry_tid=7079]]], state=connected] 2024-09-11T06:29:27.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:27.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=138, tag_tid=29, entry_tid=7082' 2024-09-11T06:29:27.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=137, tag_tid=30, entry_tid=1' -a 'object_number=137, tag_tid=30, entry_tid=1' = 'object_number=138, tag_tid=29, entry_tid=7082' 2024-09-11T06:29:27.252 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=138, tag_tid=29, entry_tid=7082' '!=' 'object_number=138, tag_tid=29, entry_tid=7002' 2024-09-11T06:29:27.252 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:29:27.252 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=138, tag_tid=29, entry_tid=7082' = 'object_number=138, tag_tid=29, entry_tid=7002' 2024-09-11T06:29:27.252 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=138, tag_tid=29, entry_tid=7082' 2024-09-11T06:29:27.252 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-09-11T06:29:27.252 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:29:27.252 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:27.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=30 2024-09-11T06:29:27.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=138, tag_tid=29, entry_tid=7082' 2024-09-11T06:29:27.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:29:27.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:27.254 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=29 2024-09-11T06:29:27.255 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-09-11T06:29:27.255 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:29:27.255 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:27.256 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:29:27.257 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=138, tag_tid=29, entry_tid=7082' 2024-09-11T06:29:27.257 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:29:27.257 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:27.258 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=7082 2024-09-11T06:29:27.258 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 30 = 29 -a 1 -le 7082 2024-09-11T06:29:27.258 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:29:27.258 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:29:27.258 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:29:27.460 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:29:27.460 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:29:27.460 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:29:27.460 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:29:27.460 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:29:27.460 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:29:27.461 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:29:27.461 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:29:27.461 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:29:27.461 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:29:27.461 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:29:27.461 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:29:27.461 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:29:27.461 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:29:27.461 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:29:27.461 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:29:27.462 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:29:27.464 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:29:27.464 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:29:27.464 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:29:27.464 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:29:27.464 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:29:27.464 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:29:27.464 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:29:27.464 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:29:27.464 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:29:27.464 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:29:27.464 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:29:27.465 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:29:27.465 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:29:27.465 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:29:27.465 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:29:27.465 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:29:27.466 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:29:27.466 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:29:27.466 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:29:27.466 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:29:27.466 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:29:27.467 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:29:27.467 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:29:27.467 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:29:27.467 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:29:27.467 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:29:27.467 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:29:27.467 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:29:27.467 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:29:27.467 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:29:27.467 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:29:27.467 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:29:27.491 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:29:27.491 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:29:27.492 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:29:27.492 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:29:27.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:29:27.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:29:27.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:27.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:27.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:27.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:29:27.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:27.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:27.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:27.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:29:27.623 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:29:27.623 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:29:27.623 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:29:27.626 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:27.626 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:29:27.626 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:27.675 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 34 2024-09-11T06:29:27.676 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 34 2024-09-11T06:29:27.676 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:29:27.676 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=8011], [object_number=138, tag_tid=29, entry_tid=8010]]], state=connected] 2024-09-11T06:29:27.676 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=137, tag_tid=29, entry_tid=7157], [object_number=136, tag_tid=29, entry_tid=7156], [object_number=139, tag_tid=29, entry_tid=7155], [object_number=138, tag_tid=29, entry_tid=7154]]], state=connected] 2024-09-11T06:29:27.679 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:27.680 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=137, tag_tid=30, entry_tid=1' 2024-09-11T06:29:27.681 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:29:27.681 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:27.682 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:27.682 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:27.682 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:29:27.682 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:27.682 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:27.682 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:27.682 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:29:27.682 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:29:27.682 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:29:27.682 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:29:27.683 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:27.683 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:29:27.683 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:27.731 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 34 2024-09-11T06:29:27.731 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 34 2024-09-11T06:29:27.731 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:29:27.731 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=8011], [object_number=138, tag_tid=29, entry_tid=8010]]], state=connected] 2024-09-11T06:29:27.731 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=137, tag_tid=29, entry_tid=7157], [object_number=136, tag_tid=29, entry_tid=7156], [object_number=139, tag_tid=29, entry_tid=7155], [object_number=138, tag_tid=29, entry_tid=7154]]], state=connected] 2024-09-11T06:29:27.734 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:27.736 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=137, tag_tid=29, entry_tid=7157' 2024-09-11T06:29:27.736 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=137, tag_tid=30, entry_tid=1' -a 'object_number=137, tag_tid=30, entry_tid=1' = 'object_number=137, tag_tid=29, entry_tid=7157' 2024-09-11T06:29:27.736 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=137, tag_tid=29, entry_tid=7157' '!=' 'object_number=138, tag_tid=29, entry_tid=7082' 2024-09-11T06:29:27.736 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:29:27.737 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=137, tag_tid=29, entry_tid=7157' = 'object_number=138, tag_tid=29, entry_tid=7082' 2024-09-11T06:29:27.737 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=137, tag_tid=29, entry_tid=7157' 2024-09-11T06:29:27.737 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-09-11T06:29:27.737 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:29:27.737 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:27.737 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=30 2024-09-11T06:29:27.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=137, tag_tid=29, entry_tid=7157' 2024-09-11T06:29:27.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:29:27.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:27.739 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=29 2024-09-11T06:29:27.740 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-09-11T06:29:27.740 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:29:27.740 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:27.741 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:29:27.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=137, tag_tid=29, entry_tid=7157' 2024-09-11T06:29:27.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:29:27.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:27.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=7157 2024-09-11T06:29:27.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 30 = 29 -a 1 -le 7157 2024-09-11T06:29:27.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:29:27.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:29:27.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:29:27.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:29:27.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:29:27.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:29:27.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:29:27.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:29:27.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:29:27.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:29:27.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:29:27.946 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:29:27.946 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:29:27.946 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:29:27.946 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:29:27.946 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:29:27.946 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:29:27.946 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:29:27.946 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:29:27.947 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:29:27.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:29:27.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:29:27.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:29:27.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:29:27.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:29:27.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:29:27.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:29:27.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:29:27.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:29:27.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:29:27.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:29:27.952 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:29:27.952 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:29:27.952 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:29:27.952 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:29:27.952 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:29:27.952 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:29:27.952 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:29:27.952 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:29:27.952 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:29:27.952 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:29:27.952 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:29:27.952 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:29:27.952 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:29:27.952 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:29:27.952 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:29:27.953 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:29:27.953 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:29:27.953 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:29:27.953 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:29:27.953 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:29:27.953 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:29:27.976 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:29:27.977 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:29:27.977 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:29:27.977 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:29:28.108 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:29:28.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:29:28.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:28.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:28.110 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:28.110 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:29:28.110 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:28.110 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:28.110 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:28.110 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:29:28.110 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:29:28.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:29:28.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:29:28.113 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:28.113 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:29:28.113 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:28.174 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 34 2024-09-11T06:29:28.174 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 34 2024-09-11T06:29:28.174 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:29:28.174 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=8011], [object_number=138, tag_tid=29, entry_tid=8010]]], state=connected] 2024-09-11T06:29:28.174 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=137, tag_tid=29, entry_tid=7233], [object_number=136, tag_tid=29, entry_tid=7232], [object_number=139, tag_tid=29, entry_tid=7231], [object_number=138, tag_tid=29, entry_tid=7230]]], state=connected] 2024-09-11T06:29:28.177 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:28.179 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=137, tag_tid=30, entry_tid=1' 2024-09-11T06:29:28.180 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:29:28.180 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:28.180 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:28.180 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:28.180 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:29:28.180 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:28.180 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:28.180 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:28.180 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:29:28.180 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:29:28.180 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:29:28.181 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:29:28.181 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:28.182 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:29:28.182 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:28.230 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 34 2024-09-11T06:29:28.230 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 34 2024-09-11T06:29:28.230 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:29:28.230 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=8011], [object_number=138, tag_tid=29, entry_tid=8010]]], state=connected] 2024-09-11T06:29:28.230 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=137, tag_tid=29, entry_tid=7233], [object_number=136, tag_tid=29, entry_tid=7232], [object_number=139, tag_tid=29, entry_tid=7231], [object_number=138, tag_tid=29, entry_tid=7230]]], state=connected] 2024-09-11T06:29:28.233 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:28.235 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=137, tag_tid=29, entry_tid=7233' 2024-09-11T06:29:28.235 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=137, tag_tid=30, entry_tid=1' -a 'object_number=137, tag_tid=30, entry_tid=1' = 'object_number=137, tag_tid=29, entry_tid=7233' 2024-09-11T06:29:28.235 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=137, tag_tid=29, entry_tid=7233' '!=' 'object_number=137, tag_tid=29, entry_tid=7157' 2024-09-11T06:29:28.235 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:29:28.235 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=137, tag_tid=29, entry_tid=7233' = 'object_number=137, tag_tid=29, entry_tid=7157' 2024-09-11T06:29:28.235 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=137, tag_tid=29, entry_tid=7233' 2024-09-11T06:29:28.236 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-09-11T06:29:28.236 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:29:28.236 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:28.237 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=30 2024-09-11T06:29:28.237 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=137, tag_tid=29, entry_tid=7233' 2024-09-11T06:29:28.237 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:29:28.237 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:28.238 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=29 2024-09-11T06:29:28.239 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-09-11T06:29:28.239 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:29:28.239 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:28.240 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:29:28.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=137, tag_tid=29, entry_tid=7233' 2024-09-11T06:29:28.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:29:28.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:28.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=7233 2024-09-11T06:29:28.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 30 = 29 -a 1 -le 7233 2024-09-11T06:29:28.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:29:28.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:29:28.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:29:28.444 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:29:28.444 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:29:28.444 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:29:28.444 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:29:28.444 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:29:28.445 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:29:28.445 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:29:28.445 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:29:28.445 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:29:28.445 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:29:28.445 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:29:28.445 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:29:28.445 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:29:28.445 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:29:28.445 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:29:28.446 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:29:28.446 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:29:28.448 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:29:28.448 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:29:28.448 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:29:28.448 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:29:28.448 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:29:28.448 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:29:28.448 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:29:28.448 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:29:28.448 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:29:28.448 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:29:28.449 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:29:28.449 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:29:28.449 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:29:28.449 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:29:28.449 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:29:28.449 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:29:28.451 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:29:28.451 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:29:28.451 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:29:28.451 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:29:28.451 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:29:28.451 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:29:28.451 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:29:28.451 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:29:28.451 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:29:28.451 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:29:28.451 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:29:28.451 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:29:28.452 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:29:28.452 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:29:28.452 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:29:28.452 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:29:28.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:29:28.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:29:28.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:29:28.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:29:28.606 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:29:28.607 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:29:28.607 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:28.607 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:28.607 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:28.607 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:29:28.607 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:28.607 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:28.607 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:28.607 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:29:28.607 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:29:28.608 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:29:28.608 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:29:28.611 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:28.611 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:29:28.611 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:28.664 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 34 2024-09-11T06:29:28.664 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 34 2024-09-11T06:29:28.664 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:29:28.664 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=8011], [object_number=138, tag_tid=29, entry_tid=8010]]], state=connected] 2024-09-11T06:29:28.664 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=136, tag_tid=29, entry_tid=7316], [object_number=139, tag_tid=29, entry_tid=7315], [object_number=138, tag_tid=29, entry_tid=7314], [object_number=137, tag_tid=29, entry_tid=7313]]], state=connected] 2024-09-11T06:29:28.668 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:28.669 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=137, tag_tid=30, entry_tid=1' 2024-09-11T06:29:28.670 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:29:28.670 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:28.671 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:28.671 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:28.671 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:29:28.671 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:28.671 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:28.671 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:28.671 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:29:28.671 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:29:28.671 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:29:28.671 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:29:28.672 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:28.672 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:29:28.672 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:28.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 34 2024-09-11T06:29:28.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 34 2024-09-11T06:29:28.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:29:28.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=8011], [object_number=138, tag_tid=29, entry_tid=8010]]], state=connected] 2024-09-11T06:29:28.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=136, tag_tid=29, entry_tid=7316], [object_number=139, tag_tid=29, entry_tid=7315], [object_number=138, tag_tid=29, entry_tid=7314], [object_number=137, tag_tid=29, entry_tid=7313]]], state=connected] 2024-09-11T06:29:28.723 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:28.725 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=136, tag_tid=29, entry_tid=7316' 2024-09-11T06:29:28.725 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=137, tag_tid=30, entry_tid=1' -a 'object_number=137, tag_tid=30, entry_tid=1' = 'object_number=136, tag_tid=29, entry_tid=7316' 2024-09-11T06:29:28.725 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=136, tag_tid=29, entry_tid=7316' '!=' 'object_number=137, tag_tid=29, entry_tid=7233' 2024-09-11T06:29:28.725 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:29:28.725 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=136, tag_tid=29, entry_tid=7316' = 'object_number=137, tag_tid=29, entry_tid=7233' 2024-09-11T06:29:28.725 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=136, tag_tid=29, entry_tid=7316' 2024-09-11T06:29:28.726 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-09-11T06:29:28.726 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:29:28.726 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:28.726 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=30 2024-09-11T06:29:28.727 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=136, tag_tid=29, entry_tid=7316' 2024-09-11T06:29:28.727 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:29:28.727 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:28.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=29 2024-09-11T06:29:28.729 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-09-11T06:29:28.729 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:29:28.729 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:28.730 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:29:28.730 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=136, tag_tid=29, entry_tid=7316' 2024-09-11T06:29:28.730 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:29:28.730 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:28.732 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=7316 2024-09-11T06:29:28.732 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 30 = 29 -a 1 -le 7316 2024-09-11T06:29:28.732 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:29:28.732 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:29:28.732 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:29:28.934 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:29:28.934 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:29:28.934 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:29:28.934 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:29:28.934 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:29:28.934 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:29:28.934 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:29:28.934 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:29:28.934 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:29:28.934 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:29:28.935 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:29:28.935 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:29:28.935 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:29:28.935 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:29:28.935 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:29:28.935 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:29:28.935 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:29:28.937 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:29:28.937 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:29:28.937 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:29:28.938 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:29:28.938 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:29:28.938 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:29:28.938 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:29:28.938 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:29:28.938 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:29:28.938 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:29:28.938 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:29:28.938 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:29:28.938 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:29:28.938 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:29:28.938 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:29:28.938 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:29:28.938 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:29:28.939 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:29:28.939 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:29:28.939 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:29:28.939 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:29:28.939 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:29:28.939 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:29:28.939 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:29:28.939 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:29:28.939 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:29:28.939 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:29:28.939 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:29:28.939 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:29:28.939 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:29:28.940 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:29:28.940 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:29:28.964 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:29:28.964 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:29:28.964 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:29:28.964 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:29:29.098 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:29:29.099 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:29:29.099 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:29.099 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:29.099 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:29.099 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:29:29.100 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:29.100 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:29.100 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:29.100 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:29:29.100 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:29:29.100 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:29:29.100 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:29:29.102 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:29.103 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:29:29.103 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:29.151 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 34 2024-09-11T06:29:29.151 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 34 2024-09-11T06:29:29.151 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:29:29.151 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=8011], [object_number=138, tag_tid=29, entry_tid=8010]]], state=connected] 2024-09-11T06:29:29.151 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=138, tag_tid=29, entry_tid=7402], [object_number=137, tag_tid=29, entry_tid=7401], [object_number=136, tag_tid=29, entry_tid=7400], [object_number=139, tag_tid=29, entry_tid=7399]]], state=connected] 2024-09-11T06:29:29.154 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:29.155 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=137, tag_tid=30, entry_tid=1' 2024-09-11T06:29:29.157 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:29:29.157 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:29.157 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:29.157 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:29.157 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:29:29.157 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:29.157 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:29.157 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:29.157 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:29:29.157 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:29:29.157 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:29:29.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:29:29.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:29.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:29:29.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:29.209 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 34 2024-09-11T06:29:29.209 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 34 2024-09-11T06:29:29.209 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:29:29.210 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=8011], [object_number=138, tag_tid=29, entry_tid=8010]]], state=connected] 2024-09-11T06:29:29.210 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=138, tag_tid=29, entry_tid=7402], [object_number=137, tag_tid=29, entry_tid=7401], [object_number=136, tag_tid=29, entry_tid=7400], [object_number=139, tag_tid=29, entry_tid=7399]]], state=connected] 2024-09-11T06:29:29.213 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:29.214 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=138, tag_tid=29, entry_tid=7402' 2024-09-11T06:29:29.214 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=137, tag_tid=30, entry_tid=1' -a 'object_number=137, tag_tid=30, entry_tid=1' = 'object_number=138, tag_tid=29, entry_tid=7402' 2024-09-11T06:29:29.214 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=138, tag_tid=29, entry_tid=7402' '!=' 'object_number=136, tag_tid=29, entry_tid=7316' 2024-09-11T06:29:29.215 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:29:29.215 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=138, tag_tid=29, entry_tid=7402' = 'object_number=136, tag_tid=29, entry_tid=7316' 2024-09-11T06:29:29.215 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=138, tag_tid=29, entry_tid=7402' 2024-09-11T06:29:29.215 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-09-11T06:29:29.215 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:29:29.216 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:29.216 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=30 2024-09-11T06:29:29.216 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=138, tag_tid=29, entry_tid=7402' 2024-09-11T06:29:29.216 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:29:29.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:29.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=29 2024-09-11T06:29:29.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-09-11T06:29:29.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:29:29.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:29.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:29:29.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=138, tag_tid=29, entry_tid=7402' 2024-09-11T06:29:29.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:29:29.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:29.221 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=7402 2024-09-11T06:29:29.221 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 30 = 29 -a 1 -le 7402 2024-09-11T06:29:29.221 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:29:29.221 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:29:29.222 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:29:29.423 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:29:29.423 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:29:29.423 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:29:29.424 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:29:29.424 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:29:29.424 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:29:29.424 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:29:29.424 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:29:29.424 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:29:29.424 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:29:29.424 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:29:29.424 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:29:29.424 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:29:29.424 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:29:29.424 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:29:29.425 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:29:29.425 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:29:29.427 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:29:29.427 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:29:29.427 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:29:29.427 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:29:29.428 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:29:29.428 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:29:29.428 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:29:29.428 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:29:29.428 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:29:29.428 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:29:29.428 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:29:29.428 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:29:29.428 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:29:29.428 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:29:29.428 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:29:29.428 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:29:29.428 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:29:29.429 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:29:29.429 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:29:29.429 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:29:29.429 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:29:29.429 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:29:29.429 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:29:29.429 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:29:29.429 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:29:29.429 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:29:29.429 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:29:29.429 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:29:29.429 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:29:29.429 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:29:29.429 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:29:29.430 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:29:29.455 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:29:29.455 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:29:29.455 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:29:29.455 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:29:29.590 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:29:29.591 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:29:29.591 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:29.592 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:29.592 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:29.592 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:29:29.592 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:29.592 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:29.592 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:29.592 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:29:29.592 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:29:29.593 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:29:29.593 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:29:29.594 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:29.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:29:29.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:29.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 34 2024-09-11T06:29:29.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 34 2024-09-11T06:29:29.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:29:29.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=8011], [object_number=138, tag_tid=29, entry_tid=8010]]], state=connected] 2024-09-11T06:29:29.649 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=138, tag_tid=29, entry_tid=7490], [object_number=137, tag_tid=29, entry_tid=7489], [object_number=136, tag_tid=29, entry_tid=7488], [object_number=139, tag_tid=29, entry_tid=7487]]], state=connected] 2024-09-11T06:29:29.652 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:29.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=137, tag_tid=30, entry_tid=1' 2024-09-11T06:29:29.654 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:29:29.655 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:29.655 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:29.655 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:29.655 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:29:29.655 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:29.655 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:29.655 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:29.655 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:29:29.655 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:29:29.655 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:29:29.655 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:29:29.656 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:29.656 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:29:29.656 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:29.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 34 2024-09-11T06:29:29.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 34 2024-09-11T06:29:29.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:29:29.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=8011], [object_number=138, tag_tid=29, entry_tid=8010]]], state=connected] 2024-09-11T06:29:29.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=138, tag_tid=29, entry_tid=7490], [object_number=137, tag_tid=29, entry_tid=7489], [object_number=136, tag_tid=29, entry_tid=7488], [object_number=139, tag_tid=29, entry_tid=7487]]], state=connected] 2024-09-11T06:29:29.707 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:29.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=138, tag_tid=29, entry_tid=7490' 2024-09-11T06:29:29.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=137, tag_tid=30, entry_tid=1' -a 'object_number=137, tag_tid=30, entry_tid=1' = 'object_number=138, tag_tid=29, entry_tid=7490' 2024-09-11T06:29:29.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=138, tag_tid=29, entry_tid=7490' '!=' 'object_number=138, tag_tid=29, entry_tid=7402' 2024-09-11T06:29:29.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:29:29.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=138, tag_tid=29, entry_tid=7490' = 'object_number=138, tag_tid=29, entry_tid=7402' 2024-09-11T06:29:29.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=138, tag_tid=29, entry_tid=7490' 2024-09-11T06:29:29.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-09-11T06:29:29.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:29:29.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:29.711 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=30 2024-09-11T06:29:29.711 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=138, tag_tid=29, entry_tid=7490' 2024-09-11T06:29:29.711 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:29:29.711 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:29.712 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=29 2024-09-11T06:29:29.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-09-11T06:29:29.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:29:29.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:29.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:29:29.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=138, tag_tid=29, entry_tid=7490' 2024-09-11T06:29:29.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:29:29.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:29.716 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=7490 2024-09-11T06:29:29.716 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 30 = 29 -a 1 -le 7490 2024-09-11T06:29:29.716 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:29:29.716 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:29:29.716 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:29:29.918 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:29:29.918 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:29:29.918 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:29:29.918 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:29:29.918 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:29:29.918 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:29:29.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:29:29.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:29:29.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:29:29.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:29:29.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:29:29.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:29:29.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:29:29.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:29:29.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:29:29.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:29:29.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:29:29.923 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:29:29.923 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:29:29.923 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:29:29.923 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:29:29.924 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:29:29.924 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:29:29.924 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:29:29.924 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:29:29.924 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:29:29.924 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:29:29.924 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:29:29.924 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:29:29.924 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:29:29.924 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:29:29.924 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:29:29.924 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:29:29.924 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:29:29.925 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:29:29.925 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:29:29.925 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:29:29.925 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:29:29.925 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:29:29.925 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:29:29.925 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:29:29.925 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:29:29.925 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:29:29.925 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:29:29.925 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:29:29.925 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:29:29.925 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:29:29.925 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:29:29.926 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:29:29.949 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:29:29.949 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:29:29.949 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:29:29.949 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:29:30.076 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:29:30.077 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:29:30.078 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:30.078 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:30.078 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:30.078 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:29:30.078 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:30.078 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:30.078 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:30.078 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:29:30.078 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:29:30.079 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:29:30.079 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:29:30.081 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:30.081 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:29:30.081 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:30.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 34 2024-09-11T06:29:30.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 34 2024-09-11T06:29:30.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:29:30.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=8011], [object_number=138, tag_tid=29, entry_tid=8010]]], state=connected] 2024-09-11T06:29:30.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=137, tag_tid=29, entry_tid=7569], [object_number=136, tag_tid=29, entry_tid=7568], [object_number=139, tag_tid=29, entry_tid=7567], [object_number=138, tag_tid=29, entry_tid=7566]]], state=connected] 2024-09-11T06:29:30.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:30.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=137, tag_tid=30, entry_tid=1' 2024-09-11T06:29:30.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:29:30.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:30.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:30.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:30.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:29:30.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:30.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:30.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:30.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:29:30.139 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:29:30.139 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:29:30.139 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:29:30.139 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:30.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:29:30.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:30.187 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 34 2024-09-11T06:29:30.188 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 34 2024-09-11T06:29:30.188 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:29:30.188 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=8011], [object_number=138, tag_tid=29, entry_tid=8010]]], state=connected] 2024-09-11T06:29:30.188 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=137, tag_tid=29, entry_tid=7569], [object_number=136, tag_tid=29, entry_tid=7568], [object_number=139, tag_tid=29, entry_tid=7567], [object_number=138, tag_tid=29, entry_tid=7566]]], state=connected] 2024-09-11T06:29:30.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:30.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=137, tag_tid=29, entry_tid=7569' 2024-09-11T06:29:30.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=137, tag_tid=30, entry_tid=1' -a 'object_number=137, tag_tid=30, entry_tid=1' = 'object_number=137, tag_tid=29, entry_tid=7569' 2024-09-11T06:29:30.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=137, tag_tid=29, entry_tid=7569' '!=' 'object_number=138, tag_tid=29, entry_tid=7490' 2024-09-11T06:29:30.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:29:30.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=137, tag_tid=29, entry_tid=7569' = 'object_number=138, tag_tid=29, entry_tid=7490' 2024-09-11T06:29:30.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=137, tag_tid=29, entry_tid=7569' 2024-09-11T06:29:30.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-09-11T06:29:30.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:29:30.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:30.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=30 2024-09-11T06:29:30.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=137, tag_tid=29, entry_tid=7569' 2024-09-11T06:29:30.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:29:30.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:30.196 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=29 2024-09-11T06:29:30.196 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-09-11T06:29:30.196 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:29:30.197 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:30.197 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:29:30.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=137, tag_tid=29, entry_tid=7569' 2024-09-11T06:29:30.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:29:30.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:30.199 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=7569 2024-09-11T06:29:30.200 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 30 = 29 -a 1 -le 7569 2024-09-11T06:29:30.200 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:29:30.200 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:29:30.200 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:29:30.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:29:30.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:29:30.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:29:30.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:29:30.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:29:30.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:29:30.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:29:30.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:29:30.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:29:30.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:29:30.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:29:30.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:29:30.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:29:30.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:29:30.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:29:30.403 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:29:30.403 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:29:30.405 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:29:30.405 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:29:30.406 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:29:30.406 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:29:30.406 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:29:30.406 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:29:30.406 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:29:30.406 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:29:30.406 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:29:30.406 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:29:30.406 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:29:30.406 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:29:30.406 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:29:30.406 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:29:30.406 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:29:30.406 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:29:30.407 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:29:30.407 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:29:30.407 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:29:30.407 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:29:30.407 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:29:30.407 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:29:30.407 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:29:30.407 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:29:30.407 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:29:30.407 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:29:30.407 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:29:30.407 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:29:30.407 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:29:30.408 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:29:30.408 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:29:30.408 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:29:30.433 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:29:30.434 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:29:30.434 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:29:30.434 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:29:30.559 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:29:30.561 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:29:30.561 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:30.561 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:30.561 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:30.561 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:29:30.561 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:30.561 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:30.561 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:30.562 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:29:30.562 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:29:30.562 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:29:30.562 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:29:30.563 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:30.563 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:29:30.564 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:30.612 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 34 2024-09-11T06:29:30.612 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 34 2024-09-11T06:29:30.612 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:29:30.612 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=8011], [object_number=138, tag_tid=29, entry_tid=8010]]], state=connected] 2024-09-11T06:29:30.612 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=136, tag_tid=29, entry_tid=7660], [object_number=139, tag_tid=29, entry_tid=7659], [object_number=138, tag_tid=29, entry_tid=7658], [object_number=137, tag_tid=29, entry_tid=7657]]], state=connected] 2024-09-11T06:29:30.615 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:30.616 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=137, tag_tid=30, entry_tid=1' 2024-09-11T06:29:30.617 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:29:30.618 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:30.618 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:30.618 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:30.618 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:29:30.618 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:30.618 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:30.618 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:30.618 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:29:30.618 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:29:30.618 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:29:30.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:29:30.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:30.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:29:30.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:30.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 34 2024-09-11T06:29:30.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 34 2024-09-11T06:29:30.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:29:30.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=8011], [object_number=138, tag_tid=29, entry_tid=8010]]], state=connected] 2024-09-11T06:29:30.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=136, tag_tid=29, entry_tid=7660], [object_number=139, tag_tid=29, entry_tid=7659], [object_number=138, tag_tid=29, entry_tid=7658], [object_number=137, tag_tid=29, entry_tid=7657]]], state=connected] 2024-09-11T06:29:30.670 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:30.672 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=136, tag_tid=29, entry_tid=7660' 2024-09-11T06:29:30.672 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=137, tag_tid=30, entry_tid=1' -a 'object_number=137, tag_tid=30, entry_tid=1' = 'object_number=136, tag_tid=29, entry_tid=7660' 2024-09-11T06:29:30.672 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=136, tag_tid=29, entry_tid=7660' '!=' 'object_number=137, tag_tid=29, entry_tid=7569' 2024-09-11T06:29:30.672 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:29:30.672 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=136, tag_tid=29, entry_tid=7660' = 'object_number=137, tag_tid=29, entry_tid=7569' 2024-09-11T06:29:30.672 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=136, tag_tid=29, entry_tid=7660' 2024-09-11T06:29:30.672 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-09-11T06:29:30.672 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:29:30.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:30.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=30 2024-09-11T06:29:30.674 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=136, tag_tid=29, entry_tid=7660' 2024-09-11T06:29:30.674 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:29:30.674 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:30.675 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=29 2024-09-11T06:29:30.675 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-09-11T06:29:30.676 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:29:30.676 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:30.676 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:29:30.677 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=136, tag_tid=29, entry_tid=7660' 2024-09-11T06:29:30.677 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:29:30.677 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:30.678 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=7660 2024-09-11T06:29:30.678 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 30 = 29 -a 1 -le 7660 2024-09-11T06:29:30.679 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:29:30.679 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:29:30.679 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:29:30.880 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:29:30.881 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:29:30.881 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:29:30.881 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:29:30.881 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:29:30.881 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:29:30.881 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:29:30.881 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:29:30.881 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:29:30.881 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:29:30.881 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:29:30.882 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:29:30.882 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:29:30.882 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:29:30.882 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:29:30.882 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:29:30.882 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:29:30.885 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:29:30.885 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:29:30.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:29:30.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:29:30.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:29:30.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:29:30.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:29:30.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:29:30.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:29:30.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:29:30.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:29:30.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:29:30.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:29:30.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:29:30.887 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:29:30.887 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:29:30.887 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:29:30.887 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:29:30.887 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:29:30.887 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:29:30.887 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:29:30.887 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:29:30.887 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:29:30.887 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:29:30.887 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:29:30.887 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:29:30.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:29:30.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:29:30.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:29:30.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:29:30.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:29:30.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:29:30.911 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:29:30.911 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:29:30.911 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:29:30.911 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:29:31.046 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:29:31.047 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:29:31.047 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:31.047 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:31.047 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:31.047 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:29:31.047 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:31.047 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:31.047 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:31.047 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:29:31.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:29:31.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:29:31.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:29:31.050 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:31.051 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:29:31.051 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:31.099 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 34 2024-09-11T06:29:31.099 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 34 2024-09-11T06:29:31.099 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:29:31.099 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=8011], [object_number=138, tag_tid=29, entry_tid=8010]]], state=connected] 2024-09-11T06:29:31.100 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=138, tag_tid=29, entry_tid=7762], [object_number=137, tag_tid=29, entry_tid=7761], [object_number=136, tag_tid=29, entry_tid=7760], [object_number=139, tag_tid=29, entry_tid=7759]]], state=connected] 2024-09-11T06:29:31.102 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:31.104 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=137, tag_tid=30, entry_tid=1' 2024-09-11T06:29:31.105 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:29:31.105 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:31.105 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:31.106 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:31.106 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:29:31.106 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:31.106 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:31.106 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:31.106 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:29:31.106 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:29:31.106 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:29:31.106 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:29:31.107 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:31.107 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:29:31.107 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:31.155 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 34 2024-09-11T06:29:31.156 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 34 2024-09-11T06:29:31.156 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:29:31.156 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=8011], [object_number=138, tag_tid=29, entry_tid=8010]]], state=connected] 2024-09-11T06:29:31.156 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=138, tag_tid=29, entry_tid=7762], [object_number=137, tag_tid=29, entry_tid=7761], [object_number=136, tag_tid=29, entry_tid=7760], [object_number=139, tag_tid=29, entry_tid=7759]]], state=connected] 2024-09-11T06:29:31.159 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:31.161 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=138, tag_tid=29, entry_tid=7762' 2024-09-11T06:29:31.161 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=137, tag_tid=30, entry_tid=1' -a 'object_number=137, tag_tid=30, entry_tid=1' = 'object_number=138, tag_tid=29, entry_tid=7762' 2024-09-11T06:29:31.161 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=138, tag_tid=29, entry_tid=7762' '!=' 'object_number=136, tag_tid=29, entry_tid=7660' 2024-09-11T06:29:31.161 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:29:31.161 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=138, tag_tid=29, entry_tid=7762' = 'object_number=136, tag_tid=29, entry_tid=7660' 2024-09-11T06:29:31.161 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=138, tag_tid=29, entry_tid=7762' 2024-09-11T06:29:31.162 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-09-11T06:29:31.162 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:29:31.162 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:31.162 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=30 2024-09-11T06:29:31.163 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=138, tag_tid=29, entry_tid=7762' 2024-09-11T06:29:31.163 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:29:31.163 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:31.164 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=29 2024-09-11T06:29:31.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-09-11T06:29:31.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:29:31.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:31.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:29:31.166 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=138, tag_tid=29, entry_tid=7762' 2024-09-11T06:29:31.166 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:29:31.166 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:31.168 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=7762 2024-09-11T06:29:31.168 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 30 = 29 -a 1 -le 7762 2024-09-11T06:29:31.168 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:29:31.168 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:29:31.168 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:29:31.369 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:29:31.369 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:29:31.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:29:31.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:29:31.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:29:31.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:29:31.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:29:31.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:29:31.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:29:31.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:29:31.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:29:31.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:29:31.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:29:31.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:29:31.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:29:31.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:29:31.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:29:31.375 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:29:31.375 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:29:31.375 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:29:31.375 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:29:31.375 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:29:31.375 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:29:31.375 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:29:31.376 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:29:31.376 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:29:31.376 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:29:31.376 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:29:31.376 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:29:31.376 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:29:31.376 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:29:31.376 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:29:31.376 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:29:31.376 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:29:31.376 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:29:31.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:29:31.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:29:31.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:29:31.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:29:31.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:29:31.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:29:31.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:29:31.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:29:31.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:29:31.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:29:31.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:29:31.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:29:31.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:29:31.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:29:31.399 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:29:31.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:29:31.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:29:31.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:29:31.525 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:29:31.526 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:29:31.526 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:31.526 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:31.526 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:31.526 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:29:31.527 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:31.527 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:31.527 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:31.527 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:29:31.527 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:29:31.528 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:29:31.528 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:29:31.530 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:31.530 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:29:31.531 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:31.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 34 2024-09-11T06:29:31.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 34 2024-09-11T06:29:31.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:29:31.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=8011], [object_number=138, tag_tid=29, entry_tid=8010]]], state=connected] 2024-09-11T06:29:31.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=138, tag_tid=29, entry_tid=7850], [object_number=137, tag_tid=29, entry_tid=7849], [object_number=136, tag_tid=29, entry_tid=7848], [object_number=139, tag_tid=29, entry_tid=7847]]], state=connected] 2024-09-11T06:29:31.583 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:31.584 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=137, tag_tid=30, entry_tid=1' 2024-09-11T06:29:31.585 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:29:31.585 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:31.585 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:31.585 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:31.586 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:29:31.586 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:31.586 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:31.586 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:31.586 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:29:31.586 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:29:31.586 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:29:31.586 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:29:31.587 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:31.587 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:29:31.587 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:31.635 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 34 2024-09-11T06:29:31.635 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 34 2024-09-11T06:29:31.635 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:29:31.635 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=8011], [object_number=138, tag_tid=29, entry_tid=8010]]], state=connected] 2024-09-11T06:29:31.635 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=138, tag_tid=29, entry_tid=7850], [object_number=137, tag_tid=29, entry_tid=7849], [object_number=136, tag_tid=29, entry_tid=7848], [object_number=139, tag_tid=29, entry_tid=7847]]], state=connected] 2024-09-11T06:29:31.638 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:31.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=138, tag_tid=29, entry_tid=7850' 2024-09-11T06:29:31.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=137, tag_tid=30, entry_tid=1' -a 'object_number=137, tag_tid=30, entry_tid=1' = 'object_number=138, tag_tid=29, entry_tid=7850' 2024-09-11T06:29:31.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=138, tag_tid=29, entry_tid=7850' '!=' 'object_number=138, tag_tid=29, entry_tid=7762' 2024-09-11T06:29:31.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:29:31.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=138, tag_tid=29, entry_tid=7850' = 'object_number=138, tag_tid=29, entry_tid=7762' 2024-09-11T06:29:31.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=138, tag_tid=29, entry_tid=7850' 2024-09-11T06:29:31.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-09-11T06:29:31.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:29:31.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:31.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=30 2024-09-11T06:29:31.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=138, tag_tid=29, entry_tid=7850' 2024-09-11T06:29:31.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:29:31.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:31.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=29 2024-09-11T06:29:31.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-09-11T06:29:31.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:29:31.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:31.645 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:29:31.646 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=138, tag_tid=29, entry_tid=7850' 2024-09-11T06:29:31.646 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:29:31.646 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:31.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=7850 2024-09-11T06:29:31.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 30 = 29 -a 1 -le 7850 2024-09-11T06:29:31.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:29:31.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:29:31.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:29:31.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:29:31.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:29:31.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:29:31.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:29:31.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:29:31.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:29:31.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:29:31.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:29:31.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:29:31.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:29:31.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:29:31.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:29:31.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:29:31.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:29:31.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:29:31.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:29:31.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:29:31.854 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:29:31.854 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:29:31.854 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:29:31.855 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:29:31.855 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:29:31.855 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:29:31.855 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:29:31.855 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:29:31.855 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:29:31.855 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:29:31.855 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:29:31.855 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:29:31.855 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:29:31.855 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:29:31.855 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:29:31.855 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:29:31.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:29:31.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:29:31.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:29:31.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:29:31.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:29:31.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:29:31.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:29:31.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:29:31.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:29:31.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:29:31.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:29:31.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:29:31.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:29:31.857 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:29:31.857 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:29:31.860 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:29:31.881 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:29:31.882 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:29:31.882 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:29:31.882 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:29:32.007 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:29:32.008 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:29:32.008 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:32.008 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:32.009 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:32.009 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:29:32.009 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:32.009 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:32.009 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:32.009 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:29:32.009 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:29:32.010 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:29:32.010 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:29:32.012 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:32.012 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:29:32.012 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:32.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 34 2024-09-11T06:29:32.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 34 2024-09-11T06:29:32.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:29:32.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=8011], [object_number=138, tag_tid=29, entry_tid=8010]]], state=connected] 2024-09-11T06:29:32.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=139, tag_tid=29, entry_tid=7923], [object_number=138, tag_tid=29, entry_tid=7922], [object_number=137, tag_tid=29, entry_tid=7921], [object_number=136, tag_tid=29, entry_tid=7920]]], state=connected] 2024-09-11T06:29:32.064 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:32.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=137, tag_tid=30, entry_tid=1' 2024-09-11T06:29:32.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:29:32.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:32.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:32.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:32.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:29:32.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:32.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:32.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:32.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:29:32.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:29:32.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:29:32.071 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:29:32.071 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:32.071 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:29:32.071 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:32.116 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 34 2024-09-11T06:29:32.116 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 34 2024-09-11T06:29:32.117 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:29:32.117 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=8011], [object_number=138, tag_tid=29, entry_tid=8010]]], state=connected] 2024-09-11T06:29:32.117 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=139, tag_tid=29, entry_tid=7923], [object_number=138, tag_tid=29, entry_tid=7922], [object_number=137, tag_tid=29, entry_tid=7921], [object_number=136, tag_tid=29, entry_tid=7920]]], state=connected] 2024-09-11T06:29:32.120 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:32.121 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=139, tag_tid=29, entry_tid=7923' 2024-09-11T06:29:32.121 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=137, tag_tid=30, entry_tid=1' -a 'object_number=137, tag_tid=30, entry_tid=1' = 'object_number=139, tag_tid=29, entry_tid=7923' 2024-09-11T06:29:32.122 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=139, tag_tid=29, entry_tid=7923' '!=' 'object_number=138, tag_tid=29, entry_tid=7850' 2024-09-11T06:29:32.122 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:29:32.122 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=139, tag_tid=29, entry_tid=7923' = 'object_number=138, tag_tid=29, entry_tid=7850' 2024-09-11T06:29:32.122 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=139, tag_tid=29, entry_tid=7923' 2024-09-11T06:29:32.122 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-09-11T06:29:32.123 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:29:32.123 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:32.123 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=30 2024-09-11T06:29:32.124 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=139, tag_tid=29, entry_tid=7923' 2024-09-11T06:29:32.124 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:29:32.124 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:32.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=29 2024-09-11T06:29:32.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-09-11T06:29:32.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:29:32.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:32.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:29:32.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=139, tag_tid=29, entry_tid=7923' 2024-09-11T06:29:32.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:29:32.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:32.128 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=7923 2024-09-11T06:29:32.128 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 30 = 29 -a 1 -le 7923 2024-09-11T06:29:32.128 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:29:32.129 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:29:32.129 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:29:32.330 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:29:32.331 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:29:32.331 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:29:32.331 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:29:32.331 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:29:32.331 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:29:32.331 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:29:32.331 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:29:32.331 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:29:32.331 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:29:32.331 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:29:32.331 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:29:32.332 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:29:32.332 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:29:32.332 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:29:32.332 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:29:32.332 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:29:32.335 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:29:32.335 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:29:32.335 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:29:32.335 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:29:32.335 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:29:32.335 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:29:32.335 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:29:32.335 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:29:32.335 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:29:32.335 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:29:32.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:29:32.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:29:32.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:29:32.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:29:32.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:29:32.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:29:32.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:29:32.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:29:32.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:29:32.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:29:32.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:29:32.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:29:32.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:29:32.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:29:32.337 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:29:32.337 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:29:32.337 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:29:32.337 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:29:32.337 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:29:32.337 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:29:32.337 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:29:32.337 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:29:32.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:29:32.362 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:29:32.362 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:29:32.362 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:29:32.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:29:32.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:29:32.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:32.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:32.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:32.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:29:32.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:32.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:32.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:32.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:29:32.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:29:32.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:29:32.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:29:32.498 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:32.499 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:29:32.499 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:32.548 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 34 2024-09-11T06:29:32.548 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 34 2024-09-11T06:29:32.549 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:29:32.549 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=8011], [object_number=138, tag_tid=29, entry_tid=8010]]], state=connected] 2024-09-11T06:29:32.549 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=136, tag_tid=29, entry_tid=8004], [object_number=139, tag_tid=29, entry_tid=8003], [object_number=138, tag_tid=29, entry_tid=8002], [object_number=137, tag_tid=29, entry_tid=8001]]], state=connected] 2024-09-11T06:29:32.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:32.553 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=137, tag_tid=30, entry_tid=1' 2024-09-11T06:29:32.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:29:32.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:32.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:32.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:32.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:29:32.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:32.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:32.555 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:32.555 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:29:32.555 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:29:32.555 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:29:32.555 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:29:32.556 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:32.556 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:29:32.556 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:32.603 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 34 2024-09-11T06:29:32.603 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 34 2024-09-11T06:29:32.604 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:29:32.604 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=8011], [object_number=138, tag_tid=29, entry_tid=8010]]], state=connected] 2024-09-11T06:29:32.604 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=136, tag_tid=29, entry_tid=8004], [object_number=139, tag_tid=29, entry_tid=8003], [object_number=138, tag_tid=29, entry_tid=8002], [object_number=137, tag_tid=29, entry_tid=8001]]], state=connected] 2024-09-11T06:29:32.607 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:32.609 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=136, tag_tid=29, entry_tid=8004' 2024-09-11T06:29:32.609 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=137, tag_tid=30, entry_tid=1' -a 'object_number=137, tag_tid=30, entry_tid=1' = 'object_number=136, tag_tid=29, entry_tid=8004' 2024-09-11T06:29:32.609 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=136, tag_tid=29, entry_tid=8004' '!=' 'object_number=139, tag_tid=29, entry_tid=7923' 2024-09-11T06:29:32.609 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:29:32.609 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=136, tag_tid=29, entry_tid=8004' = 'object_number=139, tag_tid=29, entry_tid=7923' 2024-09-11T06:29:32.610 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=136, tag_tid=29, entry_tid=8004' 2024-09-11T06:29:32.610 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-09-11T06:29:32.610 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:29:32.610 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:32.610 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=30 2024-09-11T06:29:32.611 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=136, tag_tid=29, entry_tid=8004' 2024-09-11T06:29:32.611 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:29:32.611 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:32.612 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=29 2024-09-11T06:29:32.612 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-09-11T06:29:32.613 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:29:32.613 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:32.613 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:29:32.614 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=136, tag_tid=29, entry_tid=8004' 2024-09-11T06:29:32.614 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:29:32.614 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:32.615 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=8004 2024-09-11T06:29:32.616 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 30 = 29 -a 1 -le 8004 2024-09-11T06:29:32.616 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:29:32.616 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:29:32.616 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:29:32.817 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:29:32.818 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:29:32.818 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:29:32.818 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:29:32.818 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:29:32.818 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:29:32.818 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:29:32.818 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:29:32.818 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:29:32.818 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:29:32.818 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:29:32.818 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:29:32.818 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:29:32.818 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:29:32.819 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:29:32.819 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:29:32.819 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:29:32.822 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:29:32.822 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:29:32.822 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:29:32.822 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:29:32.822 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:29:32.822 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:29:32.822 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:29:32.822 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:29:32.823 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:29:32.823 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:29:32.823 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:29:32.823 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:29:32.823 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:29:32.823 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:29:32.823 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:29:32.823 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:29:32.825 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:29:32.826 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:29:32.826 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:29:32.826 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:29:32.826 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:29:32.826 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:29:32.826 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:29:32.827 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:29:32.827 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:29:32.827 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:29:32.827 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:29:32.827 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:29:32.827 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:29:32.827 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:29:32.828 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:29:32.828 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:29:32.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:29:32.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:29:32.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:29:32.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:29:32.975 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:29:32.976 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:29:32.977 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:32.977 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:32.977 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:32.977 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:29:32.977 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:32.977 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:32.977 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:32.977 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:29:32.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:29:32.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:29:32.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:29:32.979 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:32.979 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:29:32.979 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:33.026 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 34 2024-09-11T06:29:33.026 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 34 2024-09-11T06:29:33.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:29:33.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=8011], [object_number=138, tag_tid=29, entry_tid=8010]]], state=connected] 2024-09-11T06:29:33.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=137, tag_tid=29, entry_tid=8013], [object_number=136, tag_tid=29, entry_tid=8012], [object_number=139, tag_tid=29, entry_tid=8011], [object_number=138, tag_tid=29, entry_tid=8010]]], state=connected] 2024-09-11T06:29:33.030 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:33.031 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=137, tag_tid=30, entry_tid=1' 2024-09-11T06:29:33.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:29:33.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:33.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:33.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:33.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:29:33.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:33.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:33.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:33.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:29:33.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:29:33.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:29:33.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:29:33.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:33.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:29:33.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:33.083 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 34 2024-09-11T06:29:33.083 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 34 2024-09-11T06:29:33.083 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:29:33.083 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=8011], [object_number=138, tag_tid=29, entry_tid=8010]]], state=connected] 2024-09-11T06:29:33.083 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=137, tag_tid=29, entry_tid=8013], [object_number=136, tag_tid=29, entry_tid=8012], [object_number=139, tag_tid=29, entry_tid=8011], [object_number=138, tag_tid=29, entry_tid=8010]]], state=connected] 2024-09-11T06:29:33.092 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:33.092 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=137, tag_tid=29, entry_tid=8013' 2024-09-11T06:29:33.093 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=137, tag_tid=30, entry_tid=1' -a 'object_number=137, tag_tid=30, entry_tid=1' = 'object_number=137, tag_tid=29, entry_tid=8013' 2024-09-11T06:29:33.093 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=137, tag_tid=29, entry_tid=8013' '!=' 'object_number=136, tag_tid=29, entry_tid=8004' 2024-09-11T06:29:33.093 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:29:33.093 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=137, tag_tid=29, entry_tid=8013' = 'object_number=136, tag_tid=29, entry_tid=8004' 2024-09-11T06:29:33.093 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=137, tag_tid=29, entry_tid=8013' 2024-09-11T06:29:33.093 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-09-11T06:29:33.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:29:33.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:33.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=30 2024-09-11T06:29:33.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=137, tag_tid=29, entry_tid=8013' 2024-09-11T06:29:33.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:29:33.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:33.095 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=29 2024-09-11T06:29:33.095 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-09-11T06:29:33.095 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:29:33.095 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:33.095 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:29:33.097 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=137, tag_tid=29, entry_tid=8013' 2024-09-11T06:29:33.097 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:29:33.097 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:33.097 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=8013 2024-09-11T06:29:33.098 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 30 = 29 -a 1 -le 8013 2024-09-11T06:29:33.098 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:29:33.098 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:29:33.098 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:29:33.296 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:29:33.296 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:29:33.296 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:29:33.296 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:29:33.296 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:29:33.297 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:29:33.297 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:29:33.297 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:29:33.297 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:29:33.297 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:29:33.297 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:29:33.297 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:29:33.297 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:29:33.297 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:29:33.297 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:29:33.298 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:29:33.298 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:29:33.301 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:29:33.301 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:29:33.301 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:29:33.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:29:33.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:29:33.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:29:33.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:29:33.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:29:33.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:29:33.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:29:33.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:29:33.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:29:33.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:29:33.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:29:33.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:29:33.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:29:33.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:29:33.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:29:33.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:29:33.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:29:33.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:29:33.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:29:33.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:29:33.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:29:33.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:29:33.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:29:33.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:29:33.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:29:33.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:29:33.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:29:33.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:29:33.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:29:33.326 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:29:33.326 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:29:33.326 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:29:33.326 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:29:33.449 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:29:33.450 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:29:33.450 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:33.450 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:33.450 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:33.450 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:29:33.450 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:33.451 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:33.451 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:33.451 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:29:33.452 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:29:33.452 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:29:33.452 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:29:33.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:33.454 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:29:33.454 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:33.514 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 34 2024-09-11T06:29:33.515 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 34 2024-09-11T06:29:33.515 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:29:33.515 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=8011], [object_number=138, tag_tid=29, entry_tid=8010]]], state=connected] 2024-09-11T06:29:33.515 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=137, tag_tid=29, entry_tid=8013], [object_number=136, tag_tid=29, entry_tid=8012], [object_number=139, tag_tid=29, entry_tid=8011], [object_number=138, tag_tid=29, entry_tid=8010]]], state=connected] 2024-09-11T06:29:33.518 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:33.520 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=137, tag_tid=30, entry_tid=1' 2024-09-11T06:29:33.521 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:29:33.521 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:33.521 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:33.521 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:33.522 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:29:33.522 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:33.522 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:33.522 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:33.522 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:29:33.522 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:29:33.522 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:29:33.522 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:29:33.523 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:33.523 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:29:33.523 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:33.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 34 2024-09-11T06:29:33.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 34 2024-09-11T06:29:33.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:29:33.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=8011], [object_number=138, tag_tid=29, entry_tid=8010]]], state=connected] 2024-09-11T06:29:33.572 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=137, tag_tid=29, entry_tid=8013], [object_number=136, tag_tid=29, entry_tid=8012], [object_number=139, tag_tid=29, entry_tid=8011], [object_number=138, tag_tid=29, entry_tid=8010]]], state=connected] 2024-09-11T06:29:33.575 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:33.576 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=137, tag_tid=29, entry_tid=8013' 2024-09-11T06:29:33.576 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=137, tag_tid=30, entry_tid=1' -a 'object_number=137, tag_tid=30, entry_tid=1' = 'object_number=137, tag_tid=29, entry_tid=8013' 2024-09-11T06:29:33.577 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=137, tag_tid=29, entry_tid=8013' '!=' 'object_number=137, tag_tid=29, entry_tid=8013' 2024-09-11T06:29:33.577 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:29:33.577 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.4 2024-09-11T06:29:33.979 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:29:33.979 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:29:33.979 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:29:33.979 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:29:33.979 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:29:33.979 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:29:33.979 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:29:33.979 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:29:33.979 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:29:33.979 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:29:33.979 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:29:33.980 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:29:33.980 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:29:33.980 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:29:33.980 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:29:33.980 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:29:33.980 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:29:33.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:29:33.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:29:33.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:29:33.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:29:33.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:29:33.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:29:33.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:29:33.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:29:33.985 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:29:33.985 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:29:33.985 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:29:33.985 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:29:33.985 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:29:33.985 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:29:33.985 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:29:33.985 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:29:33.985 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:29:33.985 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:29:33.985 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:29:33.985 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:29:33.985 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:29:33.986 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:29:33.986 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:29:33.986 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:29:33.986 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:29:33.986 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:29:33.986 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:29:33.986 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:29:33.986 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:29:33.986 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:29:33.986 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:29:33.986 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:29:34.010 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:29:34.010 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:29:34.010 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:29:34.010 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:29:34.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:29:34.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:29:34.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:34.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:34.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:34.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:29:34.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:34.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:34.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:34.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:29:34.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:29:34.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:29:34.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:29:34.139 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:34.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:29:34.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:34.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 34 2024-09-11T06:29:34.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 34 2024-09-11T06:29:34.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:29:34.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=8011], [object_number=138, tag_tid=29, entry_tid=8010]]], state=connected] 2024-09-11T06:29:34.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=8011], [object_number=138, tag_tid=29, entry_tid=8010]]], state=connected] 2024-09-11T06:29:34.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:34.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=137, tag_tid=30, entry_tid=1' 2024-09-11T06:29:34.196 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:29:34.197 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:34.197 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:34.197 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:34.197 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:29:34.197 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:34.197 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:34.197 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:34.197 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:29:34.197 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:29:34.197 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:29:34.197 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:29:34.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:34.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:29:34.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:34.246 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 34 2024-09-11T06:29:34.246 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 34 2024-09-11T06:29:34.246 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:29:34.246 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=8011], [object_number=138, tag_tid=29, entry_tid=8010]]], state=connected] 2024-09-11T06:29:34.246 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=8011], [object_number=138, tag_tid=29, entry_tid=8010]]], state=connected] 2024-09-11T06:29:34.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:34.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=137, tag_tid=30, entry_tid=1' 2024-09-11T06:29:34.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=137, tag_tid=30, entry_tid=1' -a 'object_number=137, tag_tid=30, entry_tid=1' = 'object_number=137, tag_tid=30, entry_tid=1' 2024-09-11T06:29:34.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:29:34.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_snap_present cluster1 mirror test snap10 2024-09-11T06:29:34.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:29:34.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:29:34.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:29:34.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap10 2024-09-11T06:29:34.252 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:29:34.252 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T06:29:34.252 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:29:35.252 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 info mirror/test@snap10 2024-09-11T06:29:35.326 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:rbd image 'test': 2024-09-11T06:29:35.326 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: size 512 MiB in 128 objects 2024-09-11T06:29:35.326 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: order 22 (4 MiB objects) 2024-09-11T06:29:35.326 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: snapshot_count: 19 2024-09-11T06:29:35.326 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: id: 10a618cfe337 2024-09-11T06:29:35.326 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: block_name_prefix: rbd_data.10a618cfe337 2024-09-11T06:29:35.326 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: format: 2 2024-09-11T06:29:35.327 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: features: layering, exclusive-lock, journaling 2024-09-11T06:29:35.327 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: op_features: 2024-09-11T06:29:35.327 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: flags: 2024-09-11T06:29:35.327 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: create_timestamp: Wed Sep 11 06:16:46 2024 2024-09-11T06:29:35.327 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: access_timestamp: Wed Sep 11 06:28:33 2024 2024-09-11T06:29:35.327 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: modify_timestamp: Wed Sep 11 06:29:04 2024 2024-09-11T06:29:35.327 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: protected: False 2024-09-11T06:29:35.327 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: journal: 10a618cfe337 2024-09-11T06:29:35.327 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring state: enabled 2024-09-11T06:29:35.327 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring mode: journal 2024-09-11T06:29:35.327 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring global id: a0f7d515-874e-4c67-a26b-4958c7ab3083 2024-09-11T06:29:35.327 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring primary: false 2024-09-11T06:29:35.330 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:29:35.331 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n snap9-clean ']' 2024-09-11T06:29:35.331 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ compare_image_snaps mirror test snap9-clean 2024-09-11T06:29:35.331 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:29:35.331 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:29:35.331 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap9-clean 2024-09-11T06:29:35.331 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local ret=0 2024-09-11T06:29:35.331 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local rmt_export=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export 2024-09-11T06:29:35.331 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loc_export=/tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:29:35.331 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:29:35.332 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9-clean /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export 2024-09-11T06:29:35.981 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T06:29:35.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9-clean /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:29:37.551 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T06:29:37.577 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmp /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:29:37.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:29:37.990 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:29:37.990 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ compare_image_snaps mirror test snap10 2024-09-11T06:29:37.990 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:29:37.990 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:29:37.990 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap10 2024-09-11T06:29:37.990 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local ret=0 2024-09-11T06:29:37.990 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local rmt_export=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export 2024-09-11T06:29:37.991 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loc_export=/tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:29:37.991 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:29:37.991 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10 /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export 2024-09-11T06:29:38.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T06:29:38.724 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10 /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:29:40.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T06:29:40.327 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmp /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:29:40.606 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:29:40.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:29:40.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ clean_snap_name=snap10-clean 2024-09-11T06:29:40.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_snap cluster2 mirror test snap10-clean 2024-09-11T06:29:40.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:29:40.745 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:29:40.745 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:29:40.745 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap10-clean 2024-09-11T06:29:40.745 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap10-clean --debug-rbd=20 --debug-journaler=20 2024-09-11T06:29:41.544 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-11T06:29:41.544 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:29:41.544 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:29:41.544 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:29:41.544 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-11T06:29:41.544 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:29:41.544 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:29:41.544 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:29:41.544 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=started 2024-09-11T06:29:41.544 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:29:41.545 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:29:41.545 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:29:42.546 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-11T06:29:42.546 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:29:42.546 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:29:42.546 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:29:42.546 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=started 2024-09-11T06:29:42.546 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local status_result 2024-09-11T06:29:42.546 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=stopped 2024-09-11T06:29:42.547 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-11T06:29:42.547 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -i state 2024-09-11T06:29:42.547 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_instance=cluster1 2024-09-11T06:29:42.547 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:29:42.548 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:29:42.548 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance=cluster1 2024-09-11T06:29:42.548 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local loop_instance 2024-09-11T06:29:42.548 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:29:42.548 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sleep 0 2024-09-11T06:29:42.548 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:29:42.549 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ seq 0 1 2024-09-11T06:29:42.551 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:29:42.551 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-11T06:29:42.551 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1:0 2024-09-11T06:29:42.551 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:29:42.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:29:42.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:29:42.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:29:42.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=cluster 2024-09-11T06:29:42.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:29:42.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:29:42.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:29:42.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:29:42.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval cluster=cluster1 2024-09-11T06:29:42.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:29:42.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:29:42.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:29:42.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:29:42.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local local_cluster=cluster1:0 2024-09-11T06:29:42.553 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster=cluster1 2024-09-11T06:29:42.553 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance 2024-09-11T06:29:42.553 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:29:42.553 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local val=cluster1:0 2024-09-11T06:29:42.553 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T06:29:42.553 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance_var_name=instance 2024-09-11T06:29:42.553 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:29:42.553 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:29:42.553 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T06:29:42.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval local_cluster=cluster1 2024-09-11T06:29:42.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ local_cluster=cluster1 2024-09-11T06:29:42.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval instance=0 2024-09-11T06:29:42.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ instance=0 2024-09-11T06:29:42.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:29:42.573 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:29:42.573 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:29:42.573 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:29:42.573 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-11T06:29:42.696 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ return 0 2024-09-11T06:29:42.696 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T06:29:42.697 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ grep -i Replaying 2024-09-11T06:29:42.697 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ echo ' "state": "Replaying"' 2024-09-11T06:29:42.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: "state": "Replaying" 2024-09-11T06:29:42.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_state=started 2024-09-11T06:29:42.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test started = started 2024-09-11T06:29:42.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:29:42.700 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-11T06:29:42.700 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:29:42.700 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:29:42.700 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:29:42.700 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:29:42.700 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' journal = journal ']' 2024-09-11T06:29:42.700 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2024-09-11T06:29:42.700 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:29:42.700 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:29:42.700 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:29:42.700 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:29:42.700 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-11T06:29:42.700 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-11T06:29:42.701 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:29:42.701 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:29:42.701 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:29:42.900 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:29:42.901 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:29:42.901 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:29:42.901 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:29:42.901 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:29:42.901 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:29:42.901 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:29:42.901 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:29:42.901 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:29:42.901 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:29:42.901 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:29:42.901 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:29:42.901 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:29:42.902 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:29:42.902 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:29:42.902 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:29:42.902 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:29:42.907 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:29:42.907 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:29:42.907 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:29:42.907 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:29:42.907 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:29:42.907 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:29:42.907 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:29:42.907 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:29:42.907 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:29:42.908 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:29:42.908 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:29:42.908 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:29:42.908 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:29:42.908 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:29:42.908 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:29:42.908 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:29:42.908 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:29:42.908 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:29:42.908 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:29:42.908 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:29:42.908 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:29:42.908 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:29:42.909 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:29:42.909 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:29:42.909 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:29:42.909 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:29:42.909 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:29:42.909 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:29:42.909 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:29:42.909 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:29:42.909 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:29:42.909 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:29:42.930 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:29:42.931 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:29:42.931 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:29:42.931 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:29:43.052 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:29:43.053 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:29:43.053 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:43.053 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:43.053 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:43.053 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:29:43.053 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:43.053 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:43.053 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:43.054 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:29:43.054 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:29:43.054 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:29:43.054 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:29:43.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:43.056 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:29:43.056 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:43.104 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 34 2024-09-11T06:29:43.104 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 34 2024-09-11T06:29:43.104 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:29:43.104 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=31, entry_tid=1], [object_number=136, tag_tid=31, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=8011], [object_number=138, tag_tid=29, entry_tid=8010]]], state=connected] 2024-09-11T06:29:43.104 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=8011], [object_number=138, tag_tid=29, entry_tid=8010]]], state=connected] 2024-09-11T06:29:43.107 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:43.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=137, tag_tid=31, entry_tid=1' 2024-09-11T06:29:43.110 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:29:43.110 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:43.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:43.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:43.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:29:43.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:43.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:43.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:43.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:29:43.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:29:43.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:29:43.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:29:43.112 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:43.112 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:29:43.112 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:43.159 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 34 2024-09-11T06:29:43.159 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 34 2024-09-11T06:29:43.159 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:29:43.160 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=31, entry_tid=1], [object_number=136, tag_tid=31, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=8011], [object_number=138, tag_tid=29, entry_tid=8010]]], state=connected] 2024-09-11T06:29:43.160 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=8011], [object_number=138, tag_tid=29, entry_tid=8010]]], state=connected] 2024-09-11T06:29:43.162 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:43.164 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=137, tag_tid=30, entry_tid=1' 2024-09-11T06:29:43.164 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=137, tag_tid=31, entry_tid=1' -a 'object_number=137, tag_tid=31, entry_tid=1' = 'object_number=137, tag_tid=30, entry_tid=1' 2024-09-11T06:29:43.164 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=137, tag_tid=30, entry_tid=1' '!=' '' 2024-09-11T06:29:43.164 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:29:43.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=137, tag_tid=30, entry_tid=1' = '' 2024-09-11T06:29:43.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=137, tag_tid=30, entry_tid=1' 2024-09-11T06:29:43.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=137, tag_tid=31, entry_tid=1' 2024-09-11T06:29:43.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:29:43.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:43.166 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=31 2024-09-11T06:29:43.166 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-09-11T06:29:43.166 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:29:43.167 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:43.167 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=30 2024-09-11T06:29:43.168 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=137, tag_tid=31, entry_tid=1' 2024-09-11T06:29:43.168 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:29:43.168 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:43.169 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:29:43.170 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=137, tag_tid=30, entry_tid=1' 2024-09-11T06:29:43.170 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:29:43.170 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:29:43.171 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=1 2024-09-11T06:29:43.171 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 31 = 30 -a 1 -le 1 2024-09-11T06:29:43.171 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:29:43.172 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:29:43.172 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:29:43.373 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:29:43.373 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:29:43.373 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:29:43.373 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:29:43.373 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:29:43.373 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:29:43.374 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:29:43.374 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:29:43.374 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:29:43.374 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:29:43.374 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:29:43.374 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:29:43.374 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:29:43.374 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:29:43.374 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:29:43.375 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:29:43.375 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:29:43.379 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:29:43.379 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:29:43.379 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:29:43.379 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:29:43.380 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:29:43.380 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:29:43.380 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:29:43.380 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:29:43.380 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:29:43.380 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:29:43.380 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:29:43.380 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:29:43.380 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:29:43.380 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:29:43.380 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:29:43.380 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:29:43.380 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:29:43.380 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:29:43.381 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:29:43.381 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:29:43.381 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:29:43.381 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:29:43.381 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:29:43.381 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:29:43.381 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:29:43.381 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:29:43.381 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:29:43.381 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:29:43.381 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:29:43.381 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:29:43.381 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:29:43.381 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:29:43.405 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:29:43.406 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:29:43.406 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:29:43.406 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:29:43.525 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:29:43.526 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:29:43.526 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:43.526 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:43.526 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:43.527 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:29:43.527 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:43.527 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:43.527 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:43.527 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:29:43.527 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:29:43.527 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:29:43.527 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:29:43.529 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:43.530 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:29:43.530 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:43.578 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 34 2024-09-11T06:29:43.578 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 34 2024-09-11T06:29:43.578 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:29:43.578 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=31, entry_tid=1], [object_number=136, tag_tid=31, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=8011], [object_number=138, tag_tid=29, entry_tid=8010]]], state=connected] 2024-09-11T06:29:43.578 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=8011], [object_number=138, tag_tid=29, entry_tid=8010]]], state=connected] 2024-09-11T06:29:43.581 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:43.583 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=137, tag_tid=31, entry_tid=1' 2024-09-11T06:29:43.584 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:29:43.584 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:43.584 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:43.584 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:43.584 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:29:43.584 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:43.584 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:43.584 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:43.584 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:29:43.584 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:29:43.585 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:29:43.585 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:29:43.585 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:43.585 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:29:43.585 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:43.635 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 34 2024-09-11T06:29:43.635 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 34 2024-09-11T06:29:43.635 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:29:43.636 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=31, entry_tid=1], [object_number=136, tag_tid=31, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=8011], [object_number=138, tag_tid=29, entry_tid=8010]]], state=connected] 2024-09-11T06:29:43.636 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=137, tag_tid=30, entry_tid=1], [object_number=136, tag_tid=30, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=8011], [object_number=138, tag_tid=29, entry_tid=8010]]], state=connected] 2024-09-11T06:29:43.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:43.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=137, tag_tid=30, entry_tid=1' 2024-09-11T06:29:43.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=137, tag_tid=31, entry_tid=1' -a 'object_number=137, tag_tid=31, entry_tid=1' = 'object_number=137, tag_tid=30, entry_tid=1' 2024-09-11T06:29:43.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=137, tag_tid=30, entry_tid=1' '!=' 'object_number=137, tag_tid=30, entry_tid=1' 2024-09-11T06:29:43.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:29:43.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.4 2024-09-11T06:29:44.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror test 2024-09-11T06:29:44.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:29:44.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:29:44.043 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:29:44.043 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:29:44.043 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n test ']' 2024-09-11T06:29:44.043 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/test' 2024-09-11T06:29:44.043 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2024-09-11T06:29:44.043 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:29:44.043 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:29:44.043 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:29:44.043 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:29:44.043 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:29:44.043 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:29:44.043 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:29:44.044 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:29:44.044 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:29:44.047 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:29:44.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2024-09-11T06:29:44.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:29:44.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:29:44.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:29:44.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:29:44.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:29:44.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:29:44.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:29:44.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:29:44.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:29:44.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:29:44.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:29:44.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:29:44.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:29:44.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:29:44.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:29:44.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:29:44.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:29:44.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:29:44.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:29:44.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:29:44.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:29:44.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:29:44.050 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:29:44.050 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:29:44.050 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:29:44.050 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:29:44.050 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:29:44.050 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:29:44.051 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:29:44.051 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:29:44.074 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:29:44.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:29:44.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:29:44.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2024-09-11T06:29:44.201 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:29:44.202 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror test 2024-09-11T06:29:44.202 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:44.202 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:44.202 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:44.203 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '' 2024-09-11T06:29:44.203 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:44.203 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:44.203 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:44.203 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:29:44.203 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:29:44.203 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:29:44.203 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:29:44.205 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:44.205 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:29:44.205 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:44.261 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 34 2024-09-11T06:29:44.262 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 34 2024-09-11T06:29:44.262 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:29:44.262 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=31, entry_tid=1], [object_number=136, tag_tid=31, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=8011], [object_number=138, tag_tid=29, entry_tid=8010]]], state=connected] 2024-09-11T06:29:44.262 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=137, tag_tid=31, entry_tid=1], [object_number=136, tag_tid=31, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=8011], [object_number=138, tag_tid=29, entry_tid=8010]]], state=connected] 2024-09-11T06:29:44.265 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:44.266 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=137, tag_tid=31, entry_tid=1' 2024-09-11T06:29:44.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror test 2024-09-11T06:29:44.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:44.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:44.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:44.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror test '..*' 2024-09-11T06:29:44.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:29:44.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:29:44.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:29:44.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:29:44.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-test.status 2024-09-11T06:29:44.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-test.status 2024-09-11T06:29:44.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:29:44.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:44.269 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/test 2024-09-11T06:29:44.269 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:44.317 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 34 2024-09-11T06:29:44.317 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 34 2024-09-11T06:29:44.317 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:29:44.317 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=137, tag_tid=31, entry_tid=1], [object_number=136, tag_tid=31, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=8011], [object_number=138, tag_tid=29, entry_tid=8010]]], state=connected] 2024-09-11T06:29:44.317 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=137, tag_tid=31, entry_tid=1], [object_number=136, tag_tid=31, entry_tid=0], [object_number=139, tag_tid=29, entry_tid=8011], [object_number=138, tag_tid=29, entry_tid=8010]]], state=connected] 2024-09-11T06:29:44.320 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.status 2024-09-11T06:29:44.322 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=137, tag_tid=31, entry_tid=1' 2024-09-11T06:29:44.322 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=137, tag_tid=31, entry_tid=1' -a 'object_number=137, tag_tid=31, entry_tid=1' = 'object_number=137, tag_tid=31, entry_tid=1' 2024-09-11T06:29:44.323 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:29:44.323 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_snap_present cluster1 mirror test snap10-clean 2024-09-11T06:29:44.323 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:29:44.323 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:29:44.323 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:29:44.323 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap10-clean 2024-09-11T06:29:44.323 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:29:44.323 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T06:29:44.323 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:29:45.323 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 info mirror/test@snap10-clean 2024-09-11T06:29:45.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:rbd image 'test': 2024-09-11T06:29:45.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: size 512 MiB in 128 objects 2024-09-11T06:29:45.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: order 22 (4 MiB objects) 2024-09-11T06:29:45.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: snapshot_count: 20 2024-09-11T06:29:45.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: id: 10a618cfe337 2024-09-11T06:29:45.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: block_name_prefix: rbd_data.10a618cfe337 2024-09-11T06:29:45.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: format: 2 2024-09-11T06:29:45.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: features: layering, exclusive-lock, journaling 2024-09-11T06:29:45.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: op_features: 2024-09-11T06:29:45.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: flags: 2024-09-11T06:29:45.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: create_timestamp: Wed Sep 11 06:16:46 2024 2024-09-11T06:29:45.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: access_timestamp: Wed Sep 11 06:29:36 2024 2024-09-11T06:29:45.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: modify_timestamp: Wed Sep 11 06:29:04 2024 2024-09-11T06:29:45.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: protected: False 2024-09-11T06:29:45.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: journal: 10a618cfe337 2024-09-11T06:29:45.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring state: enabled 2024-09-11T06:29:45.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring mode: journal 2024-09-11T06:29:45.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring global id: a0f7d515-874e-4c67-a26b-4958c7ab3083 2024-09-11T06:29:45.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring primary: false 2024-09-11T06:29:45.405 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:29:45.405 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 1 10 2024-09-11T06:29:45.407 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 10` 2024-09-11T06:29:45.407 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ snap_name=snap1 2024-09-11T06:29:45.407 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ compare_image_snaps mirror test snap1 2024-09-11T06:29:45.407 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:29:45.407 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:29:45.407 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap1 2024-09-11T06:29:45.407 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local ret=0 2024-09-11T06:29:45.407 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local rmt_export=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export 2024-09-11T06:29:45.407 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loc_export=/tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:29:45.407 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:29:45.408 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1 /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export 2024-09-11T06:29:45.948 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T06:29:45.958 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1 /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:29:47.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T06:29:47.211 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmp /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:29:47.493 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:29:47.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:29:47.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ snap_name=snap1-clean 2024-09-11T06:29:47.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ compare_image_snaps mirror test snap1-clean 2024-09-11T06:29:47.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:29:47.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:29:47.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap1-clean 2024-09-11T06:29:47.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local ret=0 2024-09-11T06:29:47.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local rmt_export=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export 2024-09-11T06:29:47.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loc_export=/tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:29:47.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:29:47.631 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1-clean /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export 2024-09-11T06:29:48.187 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T06:29:48.197 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1-clean /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:29:49.460 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T06:29:49.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmp /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:29:49.764 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:29:49.905 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:29:49.905 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 10` 2024-09-11T06:29:49.905 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ snap_name=snap2 2024-09-11T06:29:49.905 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ compare_image_snaps mirror test snap2 2024-09-11T06:29:49.905 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:29:49.905 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:29:49.905 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap2 2024-09-11T06:29:49.905 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local ret=0 2024-09-11T06:29:49.906 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local rmt_export=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export 2024-09-11T06:29:49.906 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loc_export=/tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:29:49.906 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:29:49.906 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2 /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export 2024-09-11T06:29:50.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T06:29:50.521 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2 /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:29:51.884 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T06:29:51.902 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmp /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:29:52.181 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:29:52.318 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:29:52.318 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ snap_name=snap2-clean 2024-09-11T06:29:52.318 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ compare_image_snaps mirror test snap2-clean 2024-09-11T06:29:52.318 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:29:52.318 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:29:52.318 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap2-clean 2024-09-11T06:29:52.319 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local ret=0 2024-09-11T06:29:52.319 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local rmt_export=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export 2024-09-11T06:29:52.319 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loc_export=/tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:29:52.319 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:29:52.319 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2-clean /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export 2024-09-11T06:29:52.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T06:29:52.928 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2-clean /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:29:54.284 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T06:29:54.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmp /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:29:54.583 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:29:54.722 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:29:54.722 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 10` 2024-09-11T06:29:54.722 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ snap_name=snap3 2024-09-11T06:29:54.723 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ compare_image_snaps mirror test snap3 2024-09-11T06:29:54.723 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:29:54.723 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:29:54.723 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap3 2024-09-11T06:29:54.723 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local ret=0 2024-09-11T06:29:54.723 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local rmt_export=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export 2024-09-11T06:29:54.723 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loc_export=/tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:29:54.723 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:29:54.723 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3 /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export 2024-09-11T06:29:55.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T06:29:55.353 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3 /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:29:56.757 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T06:29:56.778 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmp /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:29:57.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:29:57.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:29:57.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ snap_name=snap3-clean 2024-09-11T06:29:57.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ compare_image_snaps mirror test snap3-clean 2024-09-11T06:29:57.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:29:57.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:29:57.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap3-clean 2024-09-11T06:29:57.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local ret=0 2024-09-11T06:29:57.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local rmt_export=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export 2024-09-11T06:29:57.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loc_export=/tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:29:57.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:29:57.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3-clean /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export 2024-09-11T06:29:57.808 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T06:29:57.820 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3-clean /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:29:59.239 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T06:29:59.262 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmp /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:29:59.544 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:29:59.682 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:29:59.682 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 10` 2024-09-11T06:29:59.682 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ snap_name=snap4 2024-09-11T06:29:59.682 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ compare_image_snaps mirror test snap4 2024-09-11T06:29:59.682 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:29:59.683 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:29:59.683 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap4 2024-09-11T06:29:59.683 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local ret=0 2024-09-11T06:29:59.683 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local rmt_export=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export 2024-09-11T06:29:59.683 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loc_export=/tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:29:59.683 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:29:59.683 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4 /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export 2024-09-11T06:30:00.315 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T06:30:00.326 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4 /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:30:01.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T06:30:01.774 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmp /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:30:02.052 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:30:02.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:30:02.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ snap_name=snap4-clean 2024-09-11T06:30:02.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ compare_image_snaps mirror test snap4-clean 2024-09-11T06:30:02.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:30:02.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:30:02.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap4-clean 2024-09-11T06:30:02.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local ret=0 2024-09-11T06:30:02.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local rmt_export=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export 2024-09-11T06:30:02.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loc_export=/tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:30:02.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:30:02.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4-clean /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export 2024-09-11T06:30:02.854 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T06:30:02.867 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4-clean /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:30:04.406 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T06:30:04.429 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmp /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:30:04.708 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:30:04.848 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:30:04.848 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 10` 2024-09-11T06:30:04.848 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ snap_name=snap5 2024-09-11T06:30:04.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ compare_image_snaps mirror test snap5 2024-09-11T06:30:04.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:30:04.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:30:04.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap5 2024-09-11T06:30:04.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local ret=0 2024-09-11T06:30:04.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local rmt_export=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export 2024-09-11T06:30:04.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loc_export=/tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:30:04.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:30:04.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5 /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export 2024-09-11T06:30:05.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T06:30:05.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5 /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:30:07.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T06:30:07.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmp /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:30:07.346 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:30:07.478 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:30:07.478 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ snap_name=snap5-clean 2024-09-11T06:30:07.478 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ compare_image_snaps mirror test snap5-clean 2024-09-11T06:30:07.478 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:30:07.478 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:30:07.478 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap5-clean 2024-09-11T06:30:07.478 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local ret=0 2024-09-11T06:30:07.478 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local rmt_export=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export 2024-09-11T06:30:07.478 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loc_export=/tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:30:07.478 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:30:07.479 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5-clean /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export 2024-09-11T06:30:08.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T06:30:08.149 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5-clean /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:30:09.517 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T06:30:09.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmp /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:30:09.822 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:30:09.959 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:30:09.959 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 10` 2024-09-11T06:30:09.960 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ snap_name=snap6 2024-09-11T06:30:09.960 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ compare_image_snaps mirror test snap6 2024-09-11T06:30:09.960 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:30:09.960 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:30:09.960 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap6 2024-09-11T06:30:09.960 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local ret=0 2024-09-11T06:30:09.960 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local rmt_export=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export 2024-09-11T06:30:09.960 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loc_export=/tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:30:09.960 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:30:09.960 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6 /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export 2024-09-11T06:30:10.634 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T06:30:10.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6 /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:30:12.209 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T06:30:12.235 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmp /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:30:12.517 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:30:12.655 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:30:12.656 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ snap_name=snap6-clean 2024-09-11T06:30:12.656 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ compare_image_snaps mirror test snap6-clean 2024-09-11T06:30:12.656 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:30:12.656 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:30:12.656 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap6-clean 2024-09-11T06:30:12.656 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local ret=0 2024-09-11T06:30:12.656 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local rmt_export=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export 2024-09-11T06:30:12.656 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loc_export=/tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:30:12.656 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:30:12.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6-clean /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export 2024-09-11T06:30:13.597 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T06:30:13.608 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6-clean /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:30:15.117 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T06:30:15.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmp /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:30:15.421 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:30:15.562 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:30:15.562 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 10` 2024-09-11T06:30:15.563 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ snap_name=snap7 2024-09-11T06:30:15.563 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ compare_image_snaps mirror test snap7 2024-09-11T06:30:15.563 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:30:15.563 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:30:15.563 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap7 2024-09-11T06:30:15.563 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local ret=0 2024-09-11T06:30:15.564 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local rmt_export=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export 2024-09-11T06:30:15.564 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loc_export=/tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:30:15.564 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:30:15.564 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7 /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export 2024-09-11T06:30:16.262 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T06:30:16.274 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7 /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:30:17.879 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T06:30:17.908 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmp /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:30:18.186 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:30:18.324 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:30:18.324 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ snap_name=snap7-clean 2024-09-11T06:30:18.324 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ compare_image_snaps mirror test snap7-clean 2024-09-11T06:30:18.324 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:30:18.324 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:30:18.324 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap7-clean 2024-09-11T06:30:18.324 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local ret=0 2024-09-11T06:30:18.324 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local rmt_export=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export 2024-09-11T06:30:18.325 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loc_export=/tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:30:18.325 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:30:18.325 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7-clean /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export 2024-09-11T06:30:19.050 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T06:30:19.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7-clean /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:30:20.741 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T06:30:20.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmp /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:30:21.043 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:30:21.183 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:30:21.183 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 10` 2024-09-11T06:30:21.183 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ snap_name=snap8 2024-09-11T06:30:21.183 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ compare_image_snaps mirror test snap8 2024-09-11T06:30:21.183 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:30:21.183 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:30:21.184 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap8 2024-09-11T06:30:21.184 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local ret=0 2024-09-11T06:30:21.184 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local rmt_export=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export 2024-09-11T06:30:21.184 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loc_export=/tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:30:21.184 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:30:21.184 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8 /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export 2024-09-11T06:30:21.883 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T06:30:21.898 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8 /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:30:23.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T06:30:23.566 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmp /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:30:23.845 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:30:23.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:30:23.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ snap_name=snap8-clean 2024-09-11T06:30:23.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ compare_image_snaps mirror test snap8-clean 2024-09-11T06:30:23.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:30:23.985 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:30:23.985 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap8-clean 2024-09-11T06:30:23.985 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local ret=0 2024-09-11T06:30:23.985 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local rmt_export=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export 2024-09-11T06:30:23.985 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loc_export=/tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:30:23.985 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:30:23.986 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8-clean /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export 2024-09-11T06:30:24.662 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T06:30:24.674 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8-clean /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:30:26.186 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T06:30:26.206 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmp /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:30:26.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:30:26.628 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:30:26.628 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 10` 2024-09-11T06:30:26.628 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ snap_name=snap9 2024-09-11T06:30:26.628 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ compare_image_snaps mirror test snap9 2024-09-11T06:30:26.628 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:30:26.629 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:30:26.629 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap9 2024-09-11T06:30:26.629 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local ret=0 2024-09-11T06:30:26.629 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local rmt_export=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export 2024-09-11T06:30:26.629 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loc_export=/tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:30:26.629 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:30:26.629 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9 /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export 2024-09-11T06:30:27.337 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T06:30:27.350 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9 /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:30:28.899 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T06:30:28.928 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmp /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:30:29.208 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:30:29.349 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:30:29.349 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ snap_name=snap9-clean 2024-09-11T06:30:29.349 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ compare_image_snaps mirror test snap9-clean 2024-09-11T06:30:29.350 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:30:29.350 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:30:29.350 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap9-clean 2024-09-11T06:30:29.350 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local ret=0 2024-09-11T06:30:29.350 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local rmt_export=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export 2024-09-11T06:30:29.350 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loc_export=/tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:30:29.350 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:30:29.350 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9-clean /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export 2024-09-11T06:30:30.050 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T06:30:30.065 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9-clean /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:30:31.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T06:30:31.326 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmp /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:30:31.603 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:30:31.745 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:30:31.745 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 10` 2024-09-11T06:30:31.745 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ snap_name=snap10 2024-09-11T06:30:31.745 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ compare_image_snaps mirror test snap10 2024-09-11T06:30:31.745 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:30:31.745 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:30:31.745 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap10 2024-09-11T06:30:31.745 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local ret=0 2024-09-11T06:30:31.745 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local rmt_export=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export 2024-09-11T06:30:31.745 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loc_export=/tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:30:31.746 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:30:31.746 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10 /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export 2024-09-11T06:30:32.435 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T06:30:32.449 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10 /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:30:34.074 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T06:30:34.102 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmp /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:30:34.381 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:30:34.524 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:30:34.524 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ snap_name=snap10-clean 2024-09-11T06:30:34.524 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ compare_image_snaps mirror test snap10-clean 2024-09-11T06:30:34.524 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:30:34.524 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:30:34.525 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap10-clean 2024-09-11T06:30:34.525 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local ret=0 2024-09-11T06:30:34.525 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local rmt_export=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export 2024-09-11T06:30:34.525 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loc_export=/tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:30:34.525 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:30:34.525 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10-clean /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export 2024-09-11T06:30:35.215 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T06:30:35.228 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10-clean /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:30:36.877 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T06:30:36.899 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmp /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:30:37.177 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-test.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-test.export 2024-09-11T06:30:37.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:30:37.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 1 10 2024-09-11T06:30:37.318 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 10` 2024-09-11T06:30:37.318 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ snap_name=snap1 2024-09-11T06:30:37.318 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_snapshot cluster2 mirror test snap1 2024-09-11T06:30:37.318 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:30:37.318 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:30:37.318 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:30:37.318 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap=snap1 2024-09-11T06:30:37.318 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap1 2024-09-11T06:30:38.420 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing snap: 100% complete...done. 2024-09-11T06:30:38.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ snap_name=snap1-clean 2024-09-11T06:30:38.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_snapshot cluster2 mirror test snap1-clean 2024-09-11T06:30:38.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:30:38.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:30:38.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:30:38.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap=snap1-clean 2024-09-11T06:30:38.454 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap1-clean 2024-09-11T06:30:39.435 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing snap: 100% complete...done. 2024-09-11T06:30:39.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 10` 2024-09-11T06:30:39.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ snap_name=snap2 2024-09-11T06:30:39.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_snapshot cluster2 mirror test snap2 2024-09-11T06:30:39.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:30:39.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:30:39.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:30:39.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap=snap2 2024-09-11T06:30:39.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap2 2024-09-11T06:30:40.429 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing snap: 100% complete...done. 2024-09-11T06:30:40.468 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ snap_name=snap2-clean 2024-09-11T06:30:40.468 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_snapshot cluster2 mirror test snap2-clean 2024-09-11T06:30:40.468 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:30:40.468 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:30:40.468 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:30:40.468 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap=snap2-clean 2024-09-11T06:30:40.468 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap2-clean 2024-09-11T06:30:41.455 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing snap: 100% complete...done. 2024-09-11T06:30:41.492 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 10` 2024-09-11T06:30:41.492 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ snap_name=snap3 2024-09-11T06:30:41.492 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_snapshot cluster2 mirror test snap3 2024-09-11T06:30:41.492 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:30:41.492 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:30:41.493 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:30:41.493 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap=snap3 2024-09-11T06:30:41.493 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap3 2024-09-11T06:30:42.454 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing snap: 100% complete...done. 2024-09-11T06:30:42.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ snap_name=snap3-clean 2024-09-11T06:30:42.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_snapshot cluster2 mirror test snap3-clean 2024-09-11T06:30:42.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:30:42.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:30:42.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:30:42.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap=snap3-clean 2024-09-11T06:30:42.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap3-clean 2024-09-11T06:30:42.579 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:30:42.581+0000 7efd60c6e640 -1 bdev(0x560b146dac00 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 12 2024-09-11T06:30:43.479 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing snap: 100% complete...done. 2024-09-11T06:30:43.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 10` 2024-09-11T06:30:43.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ snap_name=snap4 2024-09-11T06:30:43.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_snapshot cluster2 mirror test snap4 2024-09-11T06:30:43.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:30:43.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:30:43.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:30:43.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap=snap4 2024-09-11T06:30:43.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap4 2024-09-11T06:30:44.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing snap: 100% complete...done. 2024-09-11T06:30:44.503 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ snap_name=snap4-clean 2024-09-11T06:30:44.503 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_snapshot cluster2 mirror test snap4-clean 2024-09-11T06:30:44.503 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:30:44.503 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:30:44.503 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:30:44.503 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap=snap4-clean 2024-09-11T06:30:44.503 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap4-clean 2024-09-11T06:30:45.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing snap: 100% complete...done. 2024-09-11T06:30:45.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 10` 2024-09-11T06:30:45.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ snap_name=snap5 2024-09-11T06:30:45.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_snapshot cluster2 mirror test snap5 2024-09-11T06:30:45.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:30:45.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:30:45.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:30:45.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap=snap5 2024-09-11T06:30:45.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap5 2024-09-11T06:30:46.509 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing snap: 100% complete...done. 2024-09-11T06:30:46.547 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ snap_name=snap5-clean 2024-09-11T06:30:46.547 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_snapshot cluster2 mirror test snap5-clean 2024-09-11T06:30:46.547 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:30:46.547 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:30:46.547 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:30:46.547 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap=snap5-clean 2024-09-11T06:30:46.547 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap5-clean 2024-09-11T06:30:47.519 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing snap: 100% complete...done. 2024-09-11T06:30:47.549 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 10` 2024-09-11T06:30:47.549 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ snap_name=snap6 2024-09-11T06:30:47.550 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_snapshot cluster2 mirror test snap6 2024-09-11T06:30:47.550 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:30:47.550 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:30:47.550 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:30:47.550 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap=snap6 2024-09-11T06:30:47.550 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap6 2024-09-11T06:30:47.686 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:30:47.689+0000 7efd5cc66640 -1 bdev(0x560b146dac00 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 1 2024-09-11T06:30:47.688 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:30:47.689+0000 7efd6046d640 -1 bdev(0x560b146dac00 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 2 2024-09-11T06:30:48.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing snap: 100% complete...done. 2024-09-11T06:30:48.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ snap_name=snap6-clean 2024-09-11T06:30:48.543 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_snapshot cluster2 mirror test snap6-clean 2024-09-11T06:30:48.543 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:30:48.543 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:30:48.543 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:30:48.543 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap=snap6-clean 2024-09-11T06:30:48.543 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap6-clean 2024-09-11T06:30:49.515 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing snap: 100% complete...done. 2024-09-11T06:30:49.547 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 10` 2024-09-11T06:30:49.548 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ snap_name=snap7 2024-09-11T06:30:49.548 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_snapshot cluster2 mirror test snap7 2024-09-11T06:30:49.548 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:30:49.548 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:30:49.548 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:30:49.548 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap=snap7 2024-09-11T06:30:49.548 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap7 2024-09-11T06:30:50.536 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing snap: 100% complete...done. 2024-09-11T06:30:50.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ snap_name=snap7-clean 2024-09-11T06:30:50.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_snapshot cluster2 mirror test snap7-clean 2024-09-11T06:30:50.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:30:50.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:30:50.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:30:50.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap=snap7-clean 2024-09-11T06:30:50.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap7-clean 2024-09-11T06:30:51.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing snap: 100% complete...done. 2024-09-11T06:30:51.588 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 10` 2024-09-11T06:30:51.588 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ snap_name=snap8 2024-09-11T06:30:51.588 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_snapshot cluster2 mirror test snap8 2024-09-11T06:30:51.588 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:30:51.588 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:30:51.588 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:30:51.588 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap=snap8 2024-09-11T06:30:51.588 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap8 2024-09-11T06:30:52.472 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing snap: 100% complete...done. 2024-09-11T06:30:52.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ snap_name=snap8-clean 2024-09-11T06:30:52.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_snapshot cluster2 mirror test snap8-clean 2024-09-11T06:30:52.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:30:52.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:30:52.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:30:52.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap=snap8-clean 2024-09-11T06:30:52.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap8-clean 2024-09-11T06:30:53.543 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing snap: 100% complete...done. 2024-09-11T06:30:53.582 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 10` 2024-09-11T06:30:53.582 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ snap_name=snap9 2024-09-11T06:30:53.582 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_snapshot cluster2 mirror test snap9 2024-09-11T06:30:53.582 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:30:53.583 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:30:53.583 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:30:53.583 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap=snap9 2024-09-11T06:30:53.583 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap9 2024-09-11T06:30:54.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing snap: 100% complete...done. 2024-09-11T06:30:54.570 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ snap_name=snap9-clean 2024-09-11T06:30:54.570 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_snapshot cluster2 mirror test snap9-clean 2024-09-11T06:30:54.570 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:30:54.570 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:30:54.570 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:30:54.570 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap=snap9-clean 2024-09-11T06:30:54.570 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap9-clean 2024-09-11T06:30:54.665 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:30:54.665+0000 7efd5cc66640 -1 bdev(0x560b146dac00 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 1 2024-09-11T06:30:54.667 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:30:54.669+0000 7efd6046d640 -1 bdev(0x560b146dac00 /var/lib/ceph/osd/cluster2-2/block) aio_submit retries 6 2024-09-11T06:30:55.536 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing snap: 100% complete...done. 2024-09-11T06:30:55.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 10` 2024-09-11T06:30:55.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ snap_name=snap10 2024-09-11T06:30:55.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_snapshot cluster2 mirror test snap10 2024-09-11T06:30:55.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:30:55.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:30:55.572 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:30:55.572 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap=snap10 2024-09-11T06:30:55.572 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap10 2024-09-11T06:30:56.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing snap: 100% complete...done. 2024-09-11T06:30:56.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ snap_name=snap10-clean 2024-09-11T06:30:56.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_snapshot cluster2 mirror test snap10-clean 2024-09-11T06:30:56.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:30:56.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:30:56.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:30:56.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap=snap10-clean 2024-09-11T06:30:56.596 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap10-clean 2024-09-11T06:30:57.587 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing snap: 100% complete...done. 2024-09-11T06:30:57.608 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image_retry cluster2 mirror test 2024-09-11T06:30:57.608 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:30:57.608 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:30:57.609 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:30:57.609 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T06:30:57.609 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:30:57.609 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image cluster2 mirror test 2024-09-11T06:30:57.610 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:30:57.610 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:30:57.610 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:30:57.610 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 snap purge mirror/test 2024-09-11T06:30:57.660 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 rm mirror/test 2024-09-11T06:30:58.231 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-11T06:30:58.236 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:30:58.237 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_present cluster1 mirror test deleted 2024-09-11T06:30:58.237 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:30:58.237 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:30:58.237 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:30:58.237 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=deleted 2024-09-11T06:30:58.237 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:30:58.237 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:30:58.237 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:30:58.237 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror test 2024-09-11T06:30:58.237 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:30:58.237 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:30:58.237 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:30:58.238 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/test 2024-09-11T06:30:58.238 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:30:58.274 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image test: (2) No such file or directory 2024-09-11T06:30:58.278 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image_id= 2024-09-11T06:30:58.278 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T06:30:58.278 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.1 2024-09-11T06:30:58.379 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_present cluster1 mirror test deleted '' 2024-09-11T06:30:58.379 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:30:58.379 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:30:58.379 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=test 2024-09-11T06:30:58.379 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=deleted 2024-09-11T06:30:58.379 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:30:58.380 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=deleted 2024-09-11T06:30:58.380 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_image_id 2024-09-11T06:30:58.380 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror test 2024-09-11T06:30:58.380 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:30:58.381 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:30:58.381 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=test 2024-09-11T06:30:58.381 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/test 2024-09-11T06:30:58.381 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:30:58.422 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image test: (2) No such file or directory 2024-09-11T06:30:58.427 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_image_id= 2024-09-11T06:30:58.427 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:30:58.427 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test deleted = deleted 2024-09-11T06:30:58.427 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:30:58.427 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ testlog 'TEST: create many images' 2024-09-11T06:30:58.427 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ tee -a /tmp/tmp.Dxwd1ESqKa/rbd-mirror.test.log 2024-09-11T06:30:58.427 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ date '+%F %T' 2024-09-11T06:30:58.428 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ echo 2024-09-11 06:30:58 TEST: create many images 2024-09-11T06:30:58.429 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:2024-09-11 06:30:58 TEST: create many images 2024-09-11T06:30:58.429 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ snap_name=snap 2024-09-11T06:30:58.429 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 1 50 2024-09-11T06:30:58.430 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:30:58.430 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_1 2024-09-11T06:30:58.431 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_image_and_enable_mirror cluster2 mirror image_1 journal 128M 2024-09-11T06:30:58.431 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:30:58.431 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:30:58.431 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:30:58.431 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:30:58.431 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_1 2024-09-11T06:30:58.431 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:30:58.431 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local mode=journal 2024-09-11T06:30:58.431 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n journal ']' 2024-09-11T06:30:58.431 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:30:58.431 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_image cluster2 mirror image_1 128M 2024-09-11T06:30:58.431 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:30:58.431 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:30:58.431 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:30:58.432 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:30:58.432 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_1 2024-09-11T06:30:58.432 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:30:58.432 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local size=128 2024-09-11T06:30:58.432 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n 128M ']' 2024-09-11T06:30:58.432 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ size=128M 2024-09-11T06:30:58.432 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:30:58.432 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_1 2024-09-11T06:30:58.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T06:30:58.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:30:58.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' pool = image ']' 2024-09-11T06:30:58.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ case "${pool}" in 2024-09-11T06:30:58.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 1 2024-09-11T06:30:58.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n '' ']' 2024-09-11T06:30:58.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ stress_write_image cluster2 mirror image_1 2024-09-11T06:30:58.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:30:58.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:30:58.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_1 2024-09-11T06:30:58.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T06:30:58.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local duration=5 2024-09-11T06:30:58.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set +e 2024-09-11T06:30:58.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_1 --debug-rbd=20 --debug-journaler=20 2024-09-11T06:30:59.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:100 IOs, 46621696 bytes 2024-09-11T06:31:03.398 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:200 IOs, 88678400 bytes 2024-09-11T06:31:03.503 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ error_code=124 2024-09-11T06:31:03.503 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set -e 2024-09-11T06:31:03.504 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T06:31:03.504 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:31:03.504 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:31:03.504 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_2 2024-09-11T06:31:03.504 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_image_and_enable_mirror cluster2 mirror image_2 journal 128M 2024-09-11T06:31:03.504 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:31:03.504 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:31:03.504 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:31:03.504 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:31:03.504 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_2 2024-09-11T06:31:03.504 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:31:03.504 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local mode=journal 2024-09-11T06:31:03.504 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n journal ']' 2024-09-11T06:31:03.504 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:31:03.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_image cluster2 mirror image_2 128M 2024-09-11T06:31:03.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:31:03.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:31:03.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:31:03.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:31:03.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_2 2024-09-11T06:31:03.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:31:03.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local size=128 2024-09-11T06:31:03.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n 128M ']' 2024-09-11T06:31:03.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ size=128M 2024-09-11T06:31:03.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:31:03.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_2 2024-09-11T06:31:03.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T06:31:03.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:31:03.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' pool = image ']' 2024-09-11T06:31:03.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ case "${pool}" in 2024-09-11T06:31:03.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 1 2024-09-11T06:31:03.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n '' ']' 2024-09-11T06:31:03.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ stress_write_image cluster2 mirror image_2 2024-09-11T06:31:03.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:31:03.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:31:03.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_2 2024-09-11T06:31:03.806 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T06:31:03.807 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local duration=14 2024-09-11T06:31:03.807 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set +e 2024-09-11T06:31:03.807 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_2 --debug-rbd=20 --debug-journaler=20 2024-09-11T06:31:05.208 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:100 IOs, 28857344 bytes 2024-09-11T06:31:07.859 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:200 IOs, 61887488 bytes 2024-09-11T06:31:08.234 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:300 IOs, 66354176 bytes 2024-09-11T06:31:12.700 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:400 IOs, 144393216 bytes 2024-09-11T06:31:15.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:500 IOs, 198644736 bytes 2024-09-11T06:31:15.528 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:600 IOs, 217955328 bytes 2024-09-11T06:31:17.605 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:700 IOs, 241515520 bytes 2024-09-11T06:31:17.816 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ error_code=124 2024-09-11T06:31:17.816 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set -e 2024-09-11T06:31:17.816 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T06:31:17.816 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:31:17.816 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:31:17.817 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_3 2024-09-11T06:31:17.817 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_image_and_enable_mirror cluster2 mirror image_3 journal 128M 2024-09-11T06:31:17.817 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:31:17.817 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:31:17.817 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:31:17.817 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:31:17.817 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_3 2024-09-11T06:31:17.817 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:31:17.817 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local mode=journal 2024-09-11T06:31:17.817 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n journal ']' 2024-09-11T06:31:17.817 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:31:17.817 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_image cluster2 mirror image_3 128M 2024-09-11T06:31:17.817 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:31:17.818 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:31:17.818 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:31:17.818 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:31:17.818 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_3 2024-09-11T06:31:17.818 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:31:17.818 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local size=128 2024-09-11T06:31:17.818 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n 128M ']' 2024-09-11T06:31:17.818 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ size=128M 2024-09-11T06:31:17.818 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:31:17.818 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_3 2024-09-11T06:31:17.877 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T06:31:17.877 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:31:17.877 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' pool = image ']' 2024-09-11T06:31:17.877 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ case "${pool}" in 2024-09-11T06:31:17.877 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 1 2024-09-11T06:31:17.877 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n '' ']' 2024-09-11T06:31:17.878 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ stress_write_image cluster2 mirror image_3 2024-09-11T06:31:17.878 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:31:17.878 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:31:17.878 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_3 2024-09-11T06:31:17.878 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T06:31:17.879 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local duration=9 2024-09-11T06:31:17.879 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set +e 2024-09-11T06:31:17.879 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_3 --debug-rbd=20 --debug-journaler=20 2024-09-11T06:31:18.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:100 IOs, 15357952 bytes 2024-09-11T06:31:20.668 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:200 IOs, 35979264 bytes 2024-09-11T06:31:21.747 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:300 IOs, 49533952 bytes 2024-09-11T06:31:24.747 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:400 IOs, 121630720 bytes 2024-09-11T06:31:25.451 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:31:25.452+0000 7efd718c5640 -1 reset not still connected to 0x560b15963110 2024-09-11T06:31:25.451 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:31:25.452+0000 7efd718c5640 -1 reset not still connected to 0x560b15a70d00 2024-09-11T06:31:25.451 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:31:25.452+0000 7efd718c5640 -1 reset not still connected to 0x560b15a71ad0 2024-09-11T06:31:26.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ error_code=124 2024-09-11T06:31:26.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set -e 2024-09-11T06:31:26.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T06:31:26.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:31:26.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:31:26.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_4 2024-09-11T06:31:26.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_image_and_enable_mirror cluster2 mirror image_4 journal 128M 2024-09-11T06:31:26.889 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:31:26.889 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:31:26.889 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:31:26.889 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:31:26.889 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_4 2024-09-11T06:31:26.889 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:31:26.889 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local mode=journal 2024-09-11T06:31:26.889 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n journal ']' 2024-09-11T06:31:26.889 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:31:26.889 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_image cluster2 mirror image_4 128M 2024-09-11T06:31:26.889 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:31:26.889 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:31:26.889 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:31:26.890 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:31:26.890 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_4 2024-09-11T06:31:26.890 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:31:26.890 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local size=128 2024-09-11T06:31:26.890 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n 128M ']' 2024-09-11T06:31:26.890 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ size=128M 2024-09-11T06:31:26.890 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:31:26.890 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_4 2024-09-11T06:31:26.953 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T06:31:26.954 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:31:26.954 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' pool = image ']' 2024-09-11T06:31:26.954 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ case "${pool}" in 2024-09-11T06:31:26.954 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 1 2024-09-11T06:31:26.954 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n '' ']' 2024-09-11T06:31:26.954 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ stress_write_image cluster2 mirror image_4 2024-09-11T06:31:26.954 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:31:26.954 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:31:26.954 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_4 2024-09-11T06:31:26.954 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T06:31:26.955 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local duration=8 2024-09-11T06:31:26.955 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set +e 2024-09-11T06:31:26.955 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_4 --debug-rbd=20 --debug-journaler=20 2024-09-11T06:31:28.609 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:100 IOs, 30432256 bytes 2024-09-11T06:31:32.662 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:200 IOs, 90898432 bytes 2024-09-11T06:31:34.962 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ error_code=124 2024-09-11T06:31:34.962 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set -e 2024-09-11T06:31:34.962 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T06:31:34.963 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:31:34.963 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:31:34.963 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_5 2024-09-11T06:31:34.963 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_image_and_enable_mirror cluster2 mirror image_5 journal 128M 2024-09-11T06:31:34.963 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:31:34.963 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:31:34.963 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:31:34.963 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:31:34.963 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_5 2024-09-11T06:31:34.963 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:31:34.963 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local mode=journal 2024-09-11T06:31:34.963 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n journal ']' 2024-09-11T06:31:34.963 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:31:34.964 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_image cluster2 mirror image_5 128M 2024-09-11T06:31:34.964 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:31:34.964 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:31:34.964 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:31:34.964 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:31:34.964 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_5 2024-09-11T06:31:34.964 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:31:34.964 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local size=128 2024-09-11T06:31:34.964 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n 128M ']' 2024-09-11T06:31:34.964 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ size=128M 2024-09-11T06:31:34.964 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:31:34.964 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_5 2024-09-11T06:31:35.037 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T06:31:35.037 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:31:35.037 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' pool = image ']' 2024-09-11T06:31:35.037 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ case "${pool}" in 2024-09-11T06:31:35.037 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 1 2024-09-11T06:31:35.037 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n '' ']' 2024-09-11T06:31:35.038 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ stress_write_image cluster2 mirror image_5 2024-09-11T06:31:35.038 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:31:35.038 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:31:35.038 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_5 2024-09-11T06:31:35.038 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T06:31:35.038 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local duration=10 2024-09-11T06:31:35.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set +e 2024-09-11T06:31:35.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_5 --debug-rbd=20 --debug-journaler=20 2024-09-11T06:31:36.374 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:100 IOs, 14766080 bytes 2024-09-11T06:31:40.560 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:200 IOs, 60712960 bytes 2024-09-11T06:31:42.422 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:300 IOs, 91772928 bytes 2024-09-11T06:31:45.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ error_code=124 2024-09-11T06:31:45.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set -e 2024-09-11T06:31:45.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T06:31:45.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:31:45.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:31:45.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_6 2024-09-11T06:31:45.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_image_and_enable_mirror cluster2 mirror image_6 journal 128M 2024-09-11T06:31:45.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:31:45.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:31:45.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:31:45.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:31:45.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_6 2024-09-11T06:31:45.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:31:45.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local mode=journal 2024-09-11T06:31:45.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n journal ']' 2024-09-11T06:31:45.050 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:31:45.050 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_image cluster2 mirror image_6 128M 2024-09-11T06:31:45.050 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:31:45.050 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:31:45.050 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:31:45.050 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:31:45.050 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_6 2024-09-11T06:31:45.050 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:31:45.050 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local size=128 2024-09-11T06:31:45.050 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n 128M ']' 2024-09-11T06:31:45.050 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ size=128M 2024-09-11T06:31:45.050 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:31:45.051 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_6 2024-09-11T06:31:45.123 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T06:31:45.123 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:31:45.123 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' pool = image ']' 2024-09-11T06:31:45.123 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ case "${pool}" in 2024-09-11T06:31:45.123 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 1 2024-09-11T06:31:45.124 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n '' ']' 2024-09-11T06:31:45.124 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ stress_write_image cluster2 mirror image_6 2024-09-11T06:31:45.124 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:31:45.124 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:31:45.124 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_6 2024-09-11T06:31:45.124 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T06:31:45.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local duration=5 2024-09-11T06:31:45.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set +e 2024-09-11T06:31:45.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_6 --debug-rbd=20 --debug-journaler=20 2024-09-11T06:31:45.221 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:31:45.218+0000 7fb2ca654640 -1 reset not still connected to 0x55ed0d865e10 2024-09-11T06:31:45.222 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:31:45.218+0000 7fb2ca654640 -1 reset not still connected to 0x55ed0d8cd5f0 2024-09-11T06:31:45.222 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:31:45.218+0000 7fb2ca654640 -1 reset not still connected to 0x55ed0e550dd0 2024-09-11T06:31:45.223 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:31:45.218+0000 7fb2ca654640 -1 reset not still connected to 0x55ed12570820 2024-09-11T06:31:46.338 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:100 IOs, 17292288 bytes 2024-09-11T06:31:50.133 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ error_code=124 2024-09-11T06:31:50.133 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set -e 2024-09-11T06:31:50.133 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T06:31:50.133 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:31:50.133 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:31:50.133 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_7 2024-09-11T06:31:50.133 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_image_and_enable_mirror cluster2 mirror image_7 journal 128M 2024-09-11T06:31:50.133 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:31:50.133 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:31:50.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:31:50.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:31:50.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_7 2024-09-11T06:31:50.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:31:50.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local mode=journal 2024-09-11T06:31:50.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n journal ']' 2024-09-11T06:31:50.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:31:50.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_image cluster2 mirror image_7 128M 2024-09-11T06:31:50.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:31:50.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:31:50.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:31:50.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:31:50.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_7 2024-09-11T06:31:50.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:31:50.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local size=128 2024-09-11T06:31:50.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n 128M ']' 2024-09-11T06:31:50.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ size=128M 2024-09-11T06:31:50.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:31:50.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_7 2024-09-11T06:31:50.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T06:31:50.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:31:50.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' pool = image ']' 2024-09-11T06:31:50.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ case "${pool}" in 2024-09-11T06:31:50.221 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 1 2024-09-11T06:31:50.221 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n '' ']' 2024-09-11T06:31:50.221 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ stress_write_image cluster2 mirror image_7 2024-09-11T06:31:50.221 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:31:50.221 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:31:50.221 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_7 2024-09-11T06:31:50.222 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T06:31:50.225 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local duration=6 2024-09-11T06:31:50.225 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set +e 2024-09-11T06:31:50.225 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_7 --debug-rbd=20 --debug-journaler=20 2024-09-11T06:31:55.764 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:100 IOs, 99945472 bytes 2024-09-11T06:31:56.234 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ error_code=124 2024-09-11T06:31:56.235 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set -e 2024-09-11T06:31:56.235 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T06:31:56.235 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:31:56.235 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:31:56.235 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_8 2024-09-11T06:31:56.235 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_image_and_enable_mirror cluster2 mirror image_8 journal 128M 2024-09-11T06:31:56.235 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:31:56.235 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:31:56.235 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:31:56.235 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:31:56.235 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_8 2024-09-11T06:31:56.235 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:31:56.236 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local mode=journal 2024-09-11T06:31:56.236 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n journal ']' 2024-09-11T06:31:56.236 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:31:56.236 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_image cluster2 mirror image_8 128M 2024-09-11T06:31:56.236 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:31:56.236 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:31:56.236 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:31:56.236 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:31:56.236 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_8 2024-09-11T06:31:56.236 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:31:56.236 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local size=128 2024-09-11T06:31:56.236 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n 128M ']' 2024-09-11T06:31:56.236 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ size=128M 2024-09-11T06:31:56.237 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:31:56.237 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_8 2024-09-11T06:31:56.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T06:31:56.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:31:56.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' pool = image ']' 2024-09-11T06:31:56.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ case "${pool}" in 2024-09-11T06:31:56.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 1 2024-09-11T06:31:56.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n '' ']' 2024-09-11T06:31:56.317 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ stress_write_image cluster2 mirror image_8 2024-09-11T06:31:56.317 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:31:56.317 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:31:56.317 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_8 2024-09-11T06:31:56.317 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T06:31:56.318 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local duration=14 2024-09-11T06:31:56.318 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set +e 2024-09-11T06:31:56.318 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_8 --debug-rbd=20 --debug-journaler=20 2024-09-11T06:31:56.656 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:100 IOs, 3018752 bytes 2024-09-11T06:31:58.284 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:31:58.282+0000 7fb2ca654640 -1 reset not still connected to 0x55ed0d865e10 2024-09-11T06:31:58.286 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:31:58.282+0000 7fb2ca654640 -1 reset not still connected to 0x55ed0d8cd5f0 2024-09-11T06:31:58.287 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:31:58.282+0000 7fb2ca654640 -1 reset not still connected to 0x55ed0e550dd0 2024-09-11T06:31:58.287 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:31:58.282+0000 7fb2ca654640 -1 reset not still connected to 0x55ed12570820 2024-09-11T06:31:58.452 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:31:58.450+0000 7fb2ca654640 -1 reset not still connected to 0x55ed288cd110 2024-09-11T06:31:58.456 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:31:58.454+0000 7fb2ca654640 -1 reset not still connected to 0x55ed369ccc30 2024-09-11T06:32:00.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:200 IOs, 55251968 bytes 2024-09-11T06:32:07.021 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:32:07.018+0000 7fb2ca654640 -1 reset not still connected to 0x55ed0d8655f0 2024-09-11T06:32:07.023 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:32:07.018+0000 7fb2ca654640 -1 reset not still connected to 0x55ed0d8cdee0 2024-09-11T06:32:07.024 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:32:07.018+0000 7fb2ca654640 -1 reset not still connected to 0x55ed0e363ee0 2024-09-11T06:32:07.024 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:32:07.018+0000 7fb2ca654640 -1 reset not still connected to 0x55ed0e47ba00 2024-09-11T06:32:07.024 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:32:07.018+0000 7fb2ca654640 -1 reset not still connected to 0x55ed11f509c0 2024-09-11T06:32:07.025 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:32:07.022+0000 7fb2ca654640 -1 reset not still connected to 0x55ed169d91e0 2024-09-11T06:32:07.025 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:32:07.022+0000 7fb2ca654640 -1 reset not still connected to 0x55ed288cd110 2024-09-11T06:32:07.025 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:32:07.022+0000 7fb2ca654640 -1 reset not still connected to 0x55ed369ccc30 2024-09-11T06:32:08.052 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:300 IOs, 179052544 bytes 2024-09-11T06:32:09.607 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:32:09.602+0000 7fb2ca654640 -1 reset not still connected to 0x55ed169d91e0 2024-09-11T06:32:09.609 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:32:09.606+0000 7fb2ca654640 -1 reset not still connected to 0x55ed288cd110 2024-09-11T06:32:09.609 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:32:09.606+0000 7fb2ca654640 -1 reset not still connected to 0x55ed369ccc30 2024-09-11T06:32:10.163 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:400 IOs, 203592704 bytes 2024-09-11T06:32:10.326 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ error_code=124 2024-09-11T06:32:10.327 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set -e 2024-09-11T06:32:10.327 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T06:32:10.327 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:32:10.327 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:32:10.327 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_9 2024-09-11T06:32:10.327 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_image_and_enable_mirror cluster2 mirror image_9 journal 128M 2024-09-11T06:32:10.327 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:32:10.328 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:32:10.328 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:32:10.328 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:32:10.328 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_9 2024-09-11T06:32:10.328 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:32:10.328 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local mode=journal 2024-09-11T06:32:10.328 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n journal ']' 2024-09-11T06:32:10.328 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:32:10.328 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_image cluster2 mirror image_9 128M 2024-09-11T06:32:10.328 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:32:10.328 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:32:10.328 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:32:10.329 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:32:10.329 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_9 2024-09-11T06:32:10.329 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:32:10.329 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local size=128 2024-09-11T06:32:10.329 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n 128M ']' 2024-09-11T06:32:10.329 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ size=128M 2024-09-11T06:32:10.329 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:32:10.330 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_9 2024-09-11T06:32:10.406 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T06:32:10.406 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:32:10.406 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' pool = image ']' 2024-09-11T06:32:10.406 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ case "${pool}" in 2024-09-11T06:32:10.406 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 1 2024-09-11T06:32:10.406 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n '' ']' 2024-09-11T06:32:10.406 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ stress_write_image cluster2 mirror image_9 2024-09-11T06:32:10.406 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:32:10.407 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:32:10.407 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_9 2024-09-11T06:32:10.407 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T06:32:10.407 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local duration=8 2024-09-11T06:32:10.407 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set +e 2024-09-11T06:32:10.408 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_9 --debug-rbd=20 --debug-journaler=20 2024-09-11T06:32:14.869 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:100 IOs, 69149696 bytes 2024-09-11T06:32:16.911 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:200 IOs, 104697856 bytes 2024-09-11T06:32:18.416 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ error_code=124 2024-09-11T06:32:18.416 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set -e 2024-09-11T06:32:18.416 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T06:32:18.416 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:32:18.416 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:32:18.416 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_10 2024-09-11T06:32:18.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_image_and_enable_mirror cluster2 mirror image_10 journal 128M 2024-09-11T06:32:18.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:32:18.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:32:18.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:32:18.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:32:18.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_10 2024-09-11T06:32:18.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:32:18.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local mode=journal 2024-09-11T06:32:18.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n journal ']' 2024-09-11T06:32:18.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:32:18.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_image cluster2 mirror image_10 128M 2024-09-11T06:32:18.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:32:18.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:32:18.418 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:32:18.418 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:32:18.418 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_10 2024-09-11T06:32:18.418 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:32:18.418 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local size=128 2024-09-11T06:32:18.418 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n 128M ']' 2024-09-11T06:32:18.418 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ size=128M 2024-09-11T06:32:18.418 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:32:18.418 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_10 2024-09-11T06:32:18.521 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T06:32:18.521 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:32:18.521 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' pool = image ']' 2024-09-11T06:32:18.521 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ case "${pool}" in 2024-09-11T06:32:18.521 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 1 2024-09-11T06:32:18.521 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n '' ']' 2024-09-11T06:32:18.521 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ stress_write_image cluster2 mirror image_10 2024-09-11T06:32:18.521 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:32:18.522 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:32:18.522 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_10 2024-09-11T06:32:18.522 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T06:32:18.523 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local duration=10 2024-09-11T06:32:18.523 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set +e 2024-09-11T06:32:18.523 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_10 --debug-rbd=20 --debug-journaler=20 2024-09-11T06:32:20.827 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:100 IOs, 24440832 bytes 2024-09-11T06:32:23.938 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:32:23.933+0000 7fb45318f640 -1 reset not still connected to 0x561315ba5520 2024-09-11T06:32:23.938 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:32:23.933+0000 7fb45318f640 -1 reset not still connected to 0x56133339c410 2024-09-11T06:32:23.938 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:32:23.933+0000 7fb45318f640 -1 reset not still connected to 0x56134d81fd40 2024-09-11T06:32:27.879 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:200 IOs, 119738368 bytes 2024-09-11T06:32:28.464 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:300 IOs, 127349760 bytes 2024-09-11T06:32:28.530 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ error_code=124 2024-09-11T06:32:28.530 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set -e 2024-09-11T06:32:28.530 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T06:32:28.530 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:32:28.530 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:32:28.530 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_11 2024-09-11T06:32:28.530 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_image_and_enable_mirror cluster2 mirror image_11 journal 128M 2024-09-11T06:32:28.531 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:32:28.531 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:32:28.531 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:32:28.531 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:32:28.531 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_11 2024-09-11T06:32:28.531 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:32:28.531 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local mode=journal 2024-09-11T06:32:28.531 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n journal ']' 2024-09-11T06:32:28.531 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:32:28.531 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_image cluster2 mirror image_11 128M 2024-09-11T06:32:28.531 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:32:28.531 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:32:28.531 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:32:28.531 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:32:28.532 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_11 2024-09-11T06:32:28.532 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:32:28.532 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local size=128 2024-09-11T06:32:28.532 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n 128M ']' 2024-09-11T06:32:28.532 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ size=128M 2024-09-11T06:32:28.532 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:32:28.532 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_11 2024-09-11T06:32:28.599 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T06:32:28.599 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:32:28.599 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' pool = image ']' 2024-09-11T06:32:28.599 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ case "${pool}" in 2024-09-11T06:32:28.600 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 1 2024-09-11T06:32:28.600 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n '' ']' 2024-09-11T06:32:28.600 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ stress_write_image cluster2 mirror image_11 2024-09-11T06:32:28.600 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:32:28.600 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:32:28.600 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_11 2024-09-11T06:32:28.600 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T06:32:28.601 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local duration=8 2024-09-11T06:32:28.601 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set +e 2024-09-11T06:32:28.601 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_11 --debug-rbd=20 --debug-journaler=20 2024-09-11T06:32:32.429 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:100 IOs, 69820416 bytes 2024-09-11T06:32:35.147 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:200 IOs, 91750400 bytes 2024-09-11T06:32:36.611 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ error_code=124 2024-09-11T06:32:36.611 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set -e 2024-09-11T06:32:36.611 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T06:32:36.611 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:32:36.611 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:32:36.611 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_12 2024-09-11T06:32:36.611 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_image_and_enable_mirror cluster2 mirror image_12 journal 128M 2024-09-11T06:32:36.611 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:32:36.611 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:32:36.611 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:32:36.612 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:32:36.612 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_12 2024-09-11T06:32:36.612 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:32:36.612 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local mode=journal 2024-09-11T06:32:36.612 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n journal ']' 2024-09-11T06:32:36.612 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:32:36.612 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_image cluster2 mirror image_12 128M 2024-09-11T06:32:36.612 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:32:36.612 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:32:36.612 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:32:36.612 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:32:36.612 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_12 2024-09-11T06:32:36.613 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:32:36.613 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local size=128 2024-09-11T06:32:36.613 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n 128M ']' 2024-09-11T06:32:36.613 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ size=128M 2024-09-11T06:32:36.613 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:32:36.613 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_12 2024-09-11T06:32:36.722 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T06:32:36.722 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:32:36.722 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' pool = image ']' 2024-09-11T06:32:36.722 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ case "${pool}" in 2024-09-11T06:32:36.722 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 1 2024-09-11T06:32:36.722 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n '' ']' 2024-09-11T06:32:36.722 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ stress_write_image cluster2 mirror image_12 2024-09-11T06:32:36.723 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:32:36.723 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:32:36.723 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_12 2024-09-11T06:32:36.723 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T06:32:36.724 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local duration=11 2024-09-11T06:32:36.724 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set +e 2024-09-11T06:32:36.724 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_12 --debug-rbd=20 --debug-journaler=20 2024-09-11T06:32:39.708 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:100 IOs, 39910400 bytes 2024-09-11T06:32:41.769 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:200 IOs, 73336832 bytes 2024-09-11T06:32:47.732 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ error_code=124 2024-09-11T06:32:47.732 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set -e 2024-09-11T06:32:47.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T06:32:47.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:32:47.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:32:47.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_13 2024-09-11T06:32:47.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_image_and_enable_mirror cluster2 mirror image_13 journal 128M 2024-09-11T06:32:47.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:32:47.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:32:47.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:32:47.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:32:47.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_13 2024-09-11T06:32:47.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:32:47.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local mode=journal 2024-09-11T06:32:47.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n journal ']' 2024-09-11T06:32:47.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:32:47.734 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_image cluster2 mirror image_13 128M 2024-09-11T06:32:47.734 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:32:47.734 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:32:47.734 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:32:47.734 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:32:47.734 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_13 2024-09-11T06:32:47.734 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:32:47.734 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local size=128 2024-09-11T06:32:47.734 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n 128M ']' 2024-09-11T06:32:47.734 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ size=128M 2024-09-11T06:32:47.734 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:32:47.734 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_13 2024-09-11T06:32:47.800 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T06:32:47.801 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:32:47.801 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' pool = image ']' 2024-09-11T06:32:47.801 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ case "${pool}" in 2024-09-11T06:32:47.801 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 1 2024-09-11T06:32:47.801 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n '' ']' 2024-09-11T06:32:47.801 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ stress_write_image cluster2 mirror image_13 2024-09-11T06:32:47.801 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:32:47.801 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:32:47.801 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_13 2024-09-11T06:32:47.801 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T06:32:47.802 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local duration=5 2024-09-11T06:32:47.802 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set +e 2024-09-11T06:32:47.802 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_13 --debug-rbd=20 --debug-journaler=20 2024-09-11T06:32:48.080 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:32:48.077+0000 7fb45318f640 -1 reset not still connected to 0x561322165790 2024-09-11T06:32:48.080 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:32:48.077+0000 7fb45318f640 -1 reset not still connected to 0x561322bdf450 2024-09-11T06:32:48.080 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:32:48.077+0000 7fb45318f640 -1 reset not still connected to 0x561337b67e10 2024-09-11T06:32:48.080 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:32:48.077+0000 7fb45318f640 -1 reset not still connected to 0x56133cabc000 2024-09-11T06:32:51.589 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:100 IOs, 53769216 bytes 2024-09-11T06:32:52.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ error_code=124 2024-09-11T06:32:52.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set -e 2024-09-11T06:32:52.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T06:32:52.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:32:52.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:32:52.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_14 2024-09-11T06:32:52.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_image_and_enable_mirror cluster2 mirror image_14 journal 128M 2024-09-11T06:32:52.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:32:52.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:32:52.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:32:52.812 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:32:52.812 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_14 2024-09-11T06:32:52.812 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:32:52.812 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local mode=journal 2024-09-11T06:32:52.812 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n journal ']' 2024-09-11T06:32:52.812 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:32:52.812 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_image cluster2 mirror image_14 128M 2024-09-11T06:32:52.812 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:32:52.812 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:32:52.812 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:32:52.812 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:32:52.812 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_14 2024-09-11T06:32:52.812 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:32:52.812 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local size=128 2024-09-11T06:32:52.813 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n 128M ']' 2024-09-11T06:32:52.813 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ size=128M 2024-09-11T06:32:52.813 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:32:52.813 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_14 2024-09-11T06:32:52.901 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T06:32:52.902 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:32:52.902 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' pool = image ']' 2024-09-11T06:32:52.902 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ case "${pool}" in 2024-09-11T06:32:52.902 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 1 2024-09-11T06:32:52.902 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n '' ']' 2024-09-11T06:32:52.902 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ stress_write_image cluster2 mirror image_14 2024-09-11T06:32:52.902 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:32:52.902 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:32:52.902 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_14 2024-09-11T06:32:52.902 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T06:32:52.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local duration=5 2024-09-11T06:32:52.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set +e 2024-09-11T06:32:52.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_14 --debug-rbd=20 --debug-journaler=20 2024-09-11T06:32:55.080 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:32:55.077+0000 7fb2ca654640 -1 reset not still connected to 0x55ed0e363ee0 2024-09-11T06:32:55.081 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:32:55.077+0000 7fb2ca654640 -1 reset not still connected to 0x55ed0e47ba00 2024-09-11T06:32:55.081 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:32:55.077+0000 7fb2ca654640 -1 reset not still connected to 0x55ed11f509c0 2024-09-11T06:32:55.081 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:32:55.077+0000 7fb2ca654640 -1 reset not still connected to 0x55ed15c15ad0 2024-09-11T06:32:55.081 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:32:55.077+0000 7fb2ca654640 -1 reset not still connected to 0x55ed169d91e0 2024-09-11T06:32:55.084 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:32:55.077+0000 7fb2ca654640 -1 reset not still connected to 0x55ed288cd110 2024-09-11T06:32:55.084 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:32:55.077+0000 7fb2ca654640 -1 reset not still connected to 0x55ed369ccc30 2024-09-11T06:32:57.566 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:100 IOs, 38937600 bytes 2024-09-11T06:32:57.911 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ error_code=124 2024-09-11T06:32:57.911 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set -e 2024-09-11T06:32:57.911 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T06:32:57.911 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:32:57.911 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:32:57.911 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_15 2024-09-11T06:32:57.911 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_image_and_enable_mirror cluster2 mirror image_15 journal 128M 2024-09-11T06:32:57.911 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:32:57.911 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:32:57.912 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:32:57.912 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:32:57.912 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_15 2024-09-11T06:32:57.912 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:32:57.912 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local mode=journal 2024-09-11T06:32:57.912 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n journal ']' 2024-09-11T06:32:57.912 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:32:57.912 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_image cluster2 mirror image_15 128M 2024-09-11T06:32:57.912 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:32:57.912 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:32:57.912 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:32:57.912 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:32:57.912 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_15 2024-09-11T06:32:57.912 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:32:57.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local size=128 2024-09-11T06:32:57.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n 128M ']' 2024-09-11T06:32:57.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ size=128M 2024-09-11T06:32:57.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:32:57.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_15 2024-09-11T06:32:58.005 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T06:32:58.005 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:32:58.005 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' pool = image ']' 2024-09-11T06:32:58.005 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ case "${pool}" in 2024-09-11T06:32:58.005 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 1 2024-09-11T06:32:58.005 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n '' ']' 2024-09-11T06:32:58.005 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ stress_write_image cluster2 mirror image_15 2024-09-11T06:32:58.005 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:32:58.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:32:58.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_15 2024-09-11T06:32:58.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T06:32:58.007 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local duration=13 2024-09-11T06:32:58.007 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set +e 2024-09-11T06:32:58.007 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_15 --debug-rbd=20 --debug-journaler=20 2024-09-11T06:33:00.632 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:33:00.629+0000 7fb2ca654640 -1 reset not still connected to 0x55ed0d8655f0 2024-09-11T06:33:00.636 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:33:00.629+0000 7fb2ca654640 -1 reset not still connected to 0x55ed0d8cdee0 2024-09-11T06:33:00.636 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:33:00.629+0000 7fb2ca654640 -1 reset not still connected to 0x55ed0e363ee0 2024-09-11T06:33:00.636 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:33:00.633+0000 7fb2ca654640 -1 reset not still connected to 0x55ed0e47ba00 2024-09-11T06:33:00.637 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:33:00.633+0000 7fb2ca654640 -1 reset not still connected to 0x55ed11f509c0 2024-09-11T06:33:00.637 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:33:00.633+0000 7fb2ca654640 -1 reset not still connected to 0x55ed15c15ad0 2024-09-11T06:33:00.637 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:33:00.633+0000 7fb2ca654640 -1 reset not still connected to 0x55ed15c36d00 2024-09-11T06:33:00.637 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:33:00.633+0000 7fb2ca654640 -1 reset not still connected to 0x55ed169d91e0 2024-09-11T06:33:00.637 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:33:00.633+0000 7fb2ca654640 -1 reset not still connected to 0x55ed288cd110 2024-09-11T06:33:00.637 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:33:00.633+0000 7fb2ca654640 -1 reset not still connected to 0x55ed369ccc30 2024-09-11T06:33:01.005 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:33:01.001+0000 7fb2ca654640 -1 reset not still connected to 0x55ed0d8cdee0 2024-09-11T06:33:01.005 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:33:01.001+0000 7fb2ca654640 -1 reset not still connected to 0x55ed0e363ee0 2024-09-11T06:33:01.009 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:33:01.005+0000 7fb2ca654640 -1 reset not still connected to 0x55ed0e47ba00 2024-09-11T06:33:01.010 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:33:01.005+0000 7fb2ca654640 -1 reset not still connected to 0x55ed11f509c0 2024-09-11T06:33:01.010 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:33:01.005+0000 7fb2ca654640 -1 reset not still connected to 0x55ed15c36d00 2024-09-11T06:33:01.014 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:33:01.009+0000 7fb2ca654640 -1 reset not still connected to 0x55ed169d91e0 2024-09-11T06:33:01.014 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:33:01.009+0000 7fb2ca654640 -1 reset not still connected to 0x55ed288cd110 2024-09-11T06:33:01.014 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:33:01.009+0000 7fb2ca654640 -1 reset not still connected to 0x55ed369ccc30 2024-09-11T06:33:04.736 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:100 IOs, 102672384 bytes 2024-09-11T06:33:09.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:200 IOs, 155464704 bytes 2024-09-11T06:33:09.858 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:33:09.853+0000 7fb2ca654640 -1 reset not still connected to 0x55ed288cd110 2024-09-11T06:33:09.858 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:33:09.853+0000 7fb2ca654640 -1 reset not still connected to 0x55ed369ccc30 2024-09-11T06:33:09.913 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:33:09.909+0000 7fb2ca654640 -1 reset not still connected to 0x55ed0d8655f0 2024-09-11T06:33:09.919 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:33:09.913+0000 7fb2ca654640 -1 reset not still connected to 0x55ed0d8cdee0 2024-09-11T06:33:09.919 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:33:09.913+0000 7fb2ca654640 -1 reset not still connected to 0x55ed0e363ee0 2024-09-11T06:33:09.919 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:33:09.913+0000 7fb2ca654640 -1 reset not still connected to 0x55ed0e47ba00 2024-09-11T06:33:09.919 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:33:09.913+0000 7fb2ca654640 -1 reset not still connected to 0x55ed11f509c0 2024-09-11T06:33:09.919 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:33:09.913+0000 7fb2ca654640 -1 reset not still connected to 0x55ed15c15ad0 2024-09-11T06:33:09.919 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:33:09.913+0000 7fb2ca654640 -1 reset not still connected to 0x55ed15c36d00 2024-09-11T06:33:09.920 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:33:09.917+0000 7fb2ca654640 -1 reset not still connected to 0x55ed169d91e0 2024-09-11T06:33:09.920 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:33:09.917+0000 7fb2ca654640 -1 reset not still connected to 0x55ed288cd110 2024-09-11T06:33:09.920 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:33:09.917+0000 7fb2ca654640 -1 reset not still connected to 0x55ed369ccc30 2024-09-11T06:33:11.016 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ error_code=124 2024-09-11T06:33:11.016 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set -e 2024-09-11T06:33:11.016 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T06:33:11.016 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:33:11.016 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:33:11.016 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_16 2024-09-11T06:33:11.016 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_image_and_enable_mirror cluster2 mirror image_16 journal 128M 2024-09-11T06:33:11.016 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:33:11.016 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:33:11.017 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:33:11.017 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:33:11.017 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_16 2024-09-11T06:33:11.017 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:33:11.017 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local mode=journal 2024-09-11T06:33:11.017 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n journal ']' 2024-09-11T06:33:11.017 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:33:11.017 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_image cluster2 mirror image_16 128M 2024-09-11T06:33:11.017 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:33:11.017 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:33:11.017 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:33:11.017 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:33:11.017 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_16 2024-09-11T06:33:11.017 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:33:11.018 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local size=128 2024-09-11T06:33:11.018 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n 128M ']' 2024-09-11T06:33:11.018 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ size=128M 2024-09-11T06:33:11.018 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:33:11.018 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_16 2024-09-11T06:33:11.095 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T06:33:11.095 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:33:11.095 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' pool = image ']' 2024-09-11T06:33:11.095 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ case "${pool}" in 2024-09-11T06:33:11.095 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 1 2024-09-11T06:33:11.096 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n '' ']' 2024-09-11T06:33:11.096 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ stress_write_image cluster2 mirror image_16 2024-09-11T06:33:11.096 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:33:11.096 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:33:11.096 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_16 2024-09-11T06:33:11.096 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T06:33:11.097 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local duration=6 2024-09-11T06:33:11.097 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set +e 2024-09-11T06:33:11.097 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_16 --debug-rbd=20 --debug-journaler=20 2024-09-11T06:33:11.655 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:100 IOs, 3469312 bytes 2024-09-11T06:33:13.009 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:200 IOs, 15423488 bytes 2024-09-11T06:33:15.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:300 IOs, 43821056 bytes 2024-09-11T06:33:17.105 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ error_code=124 2024-09-11T06:33:17.105 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set -e 2024-09-11T06:33:17.105 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T06:33:17.105 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:33:17.105 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:33:17.105 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_17 2024-09-11T06:33:17.106 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_image_and_enable_mirror cluster2 mirror image_17 journal 128M 2024-09-11T06:33:17.106 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:33:17.106 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:33:17.106 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:33:17.106 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:33:17.106 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_17 2024-09-11T06:33:17.106 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:33:17.106 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local mode=journal 2024-09-11T06:33:17.106 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n journal ']' 2024-09-11T06:33:17.106 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:33:17.106 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_image cluster2 mirror image_17 128M 2024-09-11T06:33:17.106 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:33:17.106 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:33:17.106 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:33:17.106 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:33:17.107 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_17 2024-09-11T06:33:17.107 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:33:17.107 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local size=128 2024-09-11T06:33:17.107 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n 128M ']' 2024-09-11T06:33:17.107 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ size=128M 2024-09-11T06:33:17.107 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:33:17.107 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_17 2024-09-11T06:33:17.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T06:33:17.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:33:17.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' pool = image ']' 2024-09-11T06:33:17.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ case "${pool}" in 2024-09-11T06:33:17.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 1 2024-09-11T06:33:17.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n '' ']' 2024-09-11T06:33:17.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ stress_write_image cluster2 mirror image_17 2024-09-11T06:33:17.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:33:17.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:33:17.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_17 2024-09-11T06:33:17.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T06:33:17.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local duration=6 2024-09-11T06:33:17.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set +e 2024-09-11T06:33:17.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_17 --debug-rbd=20 --debug-journaler=20 2024-09-11T06:33:18.232 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:33:18.225+0000 7fb45318f640 -1 reset not still connected to 0x561322bdf450 2024-09-11T06:33:18.232 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:33:18.225+0000 7fb45318f640 -1 reset not still connected to 0x561337b67e10 2024-09-11T06:33:18.232 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:33:18.225+0000 7fb45318f640 -1 reset not still connected to 0x56133cabc000 2024-09-11T06:33:18.232 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:33:18.225+0000 7fb45318f640 -1 reset not still connected to 0x561342690270 2024-09-11T06:33:18.359 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:33:18.353+0000 7fb45318f640 -1 reset not still connected to 0x561322931860 2024-09-11T06:33:18.360 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:33:18.357+0000 7fb45318f640 -1 reset not still connected to 0x561322ded930 2024-09-11T06:33:18.360 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:33:18.357+0000 7fb45318f640 -1 reset not still connected to 0x56134d81fd40 2024-09-11T06:33:18.360 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:33:18.357+0000 7fb45318f640 -1 reset not still connected to 0x561355040c30 2024-09-11T06:33:18.391 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:100 IOs, 11502592 bytes 2024-09-11T06:33:18.761 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:33:18.761+0000 7efd718c5640 -1 reset not still connected to 0x560b14dc4000 2024-09-11T06:33:18.761 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:33:18.761+0000 7efd718c5640 -1 reset not still connected to 0x560b15a70a90 2024-09-11T06:33:18.761 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:33:18.761+0000 7efd718c5640 -1 reset not still connected to 0x560b1899a410 2024-09-11T06:33:18.761 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:33:18.761+0000 7efd718c5640 -1 reset not still connected to 0x560b29059520 2024-09-11T06:33:20.805 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:33:20.801+0000 7fb2ca654640 -1 reset not still connected to 0x55ed15c36d00 2024-09-11T06:33:20.807 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:33:20.801+0000 7fb2ca654640 -1 reset not still connected to 0x55ed169d91e0 2024-09-11T06:33:20.808 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:33:20.805+0000 7fb2ca654640 -1 reset not still connected to 0x55ed288cd110 2024-09-11T06:33:20.808 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:33:20.805+0000 7fb2ca654640 -1 reset not still connected to 0x55ed369ccc30 2024-09-11T06:33:21.340 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:200 IOs, 46370816 bytes 2024-09-11T06:33:23.202 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ error_code=124 2024-09-11T06:33:23.202 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set -e 2024-09-11T06:33:23.203 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T06:33:23.203 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:33:23.203 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:33:23.203 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_18 2024-09-11T06:33:23.203 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_image_and_enable_mirror cluster2 mirror image_18 journal 128M 2024-09-11T06:33:23.203 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:33:23.203 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:33:23.203 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:33:23.203 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:33:23.203 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_18 2024-09-11T06:33:23.203 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:33:23.203 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local mode=journal 2024-09-11T06:33:23.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n journal ']' 2024-09-11T06:33:23.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:33:23.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_image cluster2 mirror image_18 128M 2024-09-11T06:33:23.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:33:23.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:33:23.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:33:23.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:33:23.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_18 2024-09-11T06:33:23.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:33:23.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local size=128 2024-09-11T06:33:23.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n 128M ']' 2024-09-11T06:33:23.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ size=128M 2024-09-11T06:33:23.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:33:23.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_18 2024-09-11T06:33:23.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T06:33:23.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:33:23.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' pool = image ']' 2024-09-11T06:33:23.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ case "${pool}" in 2024-09-11T06:33:23.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 1 2024-09-11T06:33:23.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n '' ']' 2024-09-11T06:33:23.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ stress_write_image cluster2 mirror image_18 2024-09-11T06:33:23.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:33:23.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:33:23.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_18 2024-09-11T06:33:23.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T06:33:23.305 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local duration=7 2024-09-11T06:33:23.305 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set +e 2024-09-11T06:33:23.306 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_18 --debug-rbd=20 --debug-journaler=20 2024-09-11T06:33:28.922 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:100 IOs, 72396800 bytes 2024-09-11T06:33:30.314 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ error_code=124 2024-09-11T06:33:30.314 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set -e 2024-09-11T06:33:30.314 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T06:33:30.315 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:33:30.315 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:33:30.315 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_19 2024-09-11T06:33:30.315 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_image_and_enable_mirror cluster2 mirror image_19 journal 128M 2024-09-11T06:33:30.315 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:33:30.315 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:33:30.315 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:33:30.315 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:33:30.315 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_19 2024-09-11T06:33:30.315 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:33:30.315 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local mode=journal 2024-09-11T06:33:30.315 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n journal ']' 2024-09-11T06:33:30.315 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:33:30.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_image cluster2 mirror image_19 128M 2024-09-11T06:33:30.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:33:30.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:33:30.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:33:30.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:33:30.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_19 2024-09-11T06:33:30.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:33:30.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local size=128 2024-09-11T06:33:30.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n 128M ']' 2024-09-11T06:33:30.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ size=128M 2024-09-11T06:33:30.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:33:30.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_19 2024-09-11T06:33:30.427 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T06:33:30.427 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:33:30.427 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' pool = image ']' 2024-09-11T06:33:30.427 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ case "${pool}" in 2024-09-11T06:33:30.427 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 1 2024-09-11T06:33:30.427 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n '' ']' 2024-09-11T06:33:30.427 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ stress_write_image cluster2 mirror image_19 2024-09-11T06:33:30.427 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:33:30.428 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:33:30.428 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_19 2024-09-11T06:33:30.428 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T06:33:30.429 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local duration=6 2024-09-11T06:33:30.429 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set +e 2024-09-11T06:33:30.429 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_19 --debug-rbd=20 --debug-journaler=20 2024-09-11T06:33:31.122 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:33:31.120+0000 7efd718c5640 -1 reset not still connected to 0x560b179fc1a0 2024-09-11T06:33:31.123 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:33:31.120+0000 7efd718c5640 -1 reset not still connected to 0x560b1899a410 2024-09-11T06:33:31.123 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:33:31.120+0000 7efd718c5640 -1 reset not still connected to 0x560b29059520 2024-09-11T06:33:31.781 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:33:31.780+0000 7efd718c5640 -1 reset not still connected to 0x560b179fc1a0 2024-09-11T06:33:31.781 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:33:31.780+0000 7efd718c5640 -1 reset not still connected to 0x560b1899a410 2024-09-11T06:33:31.782 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:33:31.780+0000 7efd718c5640 -1 reset not still connected to 0x560b29059520 2024-09-11T06:33:31.782 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:33:31.780+0000 7efd718c5640 -1 reset not still connected to 0x560b37798410 2024-09-11T06:33:33.184 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:33:33.180+0000 7fc41084f640 -1 reset not still connected to 0x55eb3893ac30 2024-09-11T06:33:33.184 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:33:33.180+0000 7fc41084f640 -1 reset not still connected to 0x55eb4f1329c0 2024-09-11T06:33:33.184 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:33:33.180+0000 7fc41084f640 -1 reset not still connected to 0x55eb53d56f70 2024-09-11T06:33:34.549 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:100 IOs, 41867264 bytes 2024-09-11T06:33:36.470 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ error_code=124 2024-09-11T06:33:36.470 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set -e 2024-09-11T06:33:36.470 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T06:33:36.470 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:33:36.470 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:33:36.470 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_20 2024-09-11T06:33:36.470 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_image_and_enable_mirror cluster2 mirror image_20 journal 128M 2024-09-11T06:33:36.470 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:33:36.470 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:33:36.470 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:33:36.470 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:33:36.471 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_20 2024-09-11T06:33:36.471 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:33:36.471 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local mode=journal 2024-09-11T06:33:36.471 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n journal ']' 2024-09-11T06:33:36.471 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:33:36.471 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_image cluster2 mirror image_20 128M 2024-09-11T06:33:36.471 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:33:36.471 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:33:36.471 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:33:36.471 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:33:36.471 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_20 2024-09-11T06:33:36.471 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:33:36.471 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local size=128 2024-09-11T06:33:36.471 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n 128M ']' 2024-09-11T06:33:36.472 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ size=128M 2024-09-11T06:33:36.472 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:33:36.472 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_20 2024-09-11T06:33:36.584 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T06:33:36.584 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:33:36.584 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' pool = image ']' 2024-09-11T06:33:36.584 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ case "${pool}" in 2024-09-11T06:33:36.584 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 1 2024-09-11T06:33:36.584 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n '' ']' 2024-09-11T06:33:36.584 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ stress_write_image cluster2 mirror image_20 2024-09-11T06:33:36.584 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:33:36.584 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:33:36.584 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_20 2024-09-11T06:33:36.584 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T06:33:36.585 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local duration=14 2024-09-11T06:33:36.585 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set +e 2024-09-11T06:33:36.585 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_20 --debug-rbd=20 --debug-journaler=20 2024-09-11T06:33:37.023 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:100 IOs, 2211840 bytes 2024-09-11T06:33:39.632 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:200 IOs, 38782976 bytes 2024-09-11T06:33:43.700 INFO:tasks.ceph.cluster2.osd.0.smithi112.stderr:2024-09-11T06:33:43.700+0000 7f3fa0017640 -1 reset not still connected to 0x563d381c04e0 2024-09-11T06:33:44.473 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:300 IOs, 94322688 bytes 2024-09-11T06:33:46.763 INFO:tasks.ceph.cluster2.osd.0.smithi112.stderr:2024-09-11T06:33:46.764+0000 7f3fa0017640 -1 reset not still connected to 0x563d2f3f8c30 2024-09-11T06:33:46.763 INFO:tasks.ceph.cluster2.osd.0.smithi112.stderr:2024-09-11T06:33:46.764+0000 7f3fa0017640 -1 reset not still connected to 0x563d381c04e0 2024-09-11T06:33:50.274 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:400 IOs, 181650432 bytes 2024-09-11T06:33:50.593 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ error_code=124 2024-09-11T06:33:50.593 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set -e 2024-09-11T06:33:50.593 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T06:33:50.594 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:33:50.594 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:33:50.594 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_21 2024-09-11T06:33:50.594 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_image_and_enable_mirror cluster2 mirror image_21 journal 128M 2024-09-11T06:33:50.594 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:33:50.594 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:33:50.594 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:33:50.594 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:33:50.594 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_21 2024-09-11T06:33:50.594 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:33:50.594 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local mode=journal 2024-09-11T06:33:50.594 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n journal ']' 2024-09-11T06:33:50.594 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:33:50.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_image cluster2 mirror image_21 128M 2024-09-11T06:33:50.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:33:50.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:33:50.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:33:50.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:33:50.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_21 2024-09-11T06:33:50.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:33:50.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local size=128 2024-09-11T06:33:50.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n 128M ']' 2024-09-11T06:33:50.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ size=128M 2024-09-11T06:33:50.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:33:50.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_21 2024-09-11T06:33:50.694 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T06:33:50.694 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:33:50.694 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' pool = image ']' 2024-09-11T06:33:50.694 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ case "${pool}" in 2024-09-11T06:33:50.695 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 1 2024-09-11T06:33:50.695 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n '' ']' 2024-09-11T06:33:50.695 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ stress_write_image cluster2 mirror image_21 2024-09-11T06:33:50.695 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:33:50.695 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:33:50.695 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_21 2024-09-11T06:33:50.695 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T06:33:50.696 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local duration=14 2024-09-11T06:33:50.696 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set +e 2024-09-11T06:33:50.696 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_21 --debug-rbd=20 --debug-journaler=20 2024-09-11T06:33:54.731 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:100 IOs, 48574464 bytes 2024-09-11T06:33:59.332 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:200 IOs, 111318016 bytes 2024-09-11T06:34:00.136 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:34:00.139+0000 7efd718c5640 -1 reset not still connected to 0x560b14da5d40 2024-09-11T06:34:00.136 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:34:00.139+0000 7efd718c5640 -1 reset not still connected to 0x560b14e4e820 2024-09-11T06:34:00.136 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:34:00.139+0000 7efd718c5640 -1 reset not still connected to 0x560b159635f0 2024-09-11T06:34:00.136 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:34:00.139+0000 7efd718c5640 -1 reset not still connected to 0x560b15a708f0 2024-09-11T06:34:00.136 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:34:00.139+0000 7efd718c5640 -1 reset not still connected to 0x560b15a70dd0 2024-09-11T06:34:00.136 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:34:00.139+0000 7efd718c5640 -1 reset not still connected to 0x560b15a71520 2024-09-11T06:34:01.320 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:300 IOs, 132582400 bytes 2024-09-11T06:34:02.796 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:34:02.792+0000 7fc41084f640 -1 reset not still connected to 0x55eb1afbe4e0 2024-09-11T06:34:02.796 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:34:02.792+0000 7fc41084f640 -1 reset not still connected to 0x55eb2613e680 2024-09-11T06:34:02.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:400 IOs, 147673088 bytes 2024-09-11T06:34:03.237 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:34:03.236+0000 7fb2ca654640 -1 reset not still connected to 0x55ed0d759d40 2024-09-11T06:34:03.237 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:34:03.236+0000 7fb2ca654640 -1 reset not still connected to 0x55ed0d862410 2024-09-11T06:34:03.237 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:34:03.236+0000 7fb2ca654640 -1 reset not still connected to 0x55ed0d863040 2024-09-11T06:34:03.237 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:34:03.236+0000 7fb2ca654640 -1 reset not still connected to 0x55ed0d865e10 2024-09-11T06:34:03.241 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:34:03.240+0000 7fb2ca654640 -1 reset not still connected to 0x55ed0d8cd5f0 2024-09-11T06:34:03.241 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:34:03.240+0000 7fb2ca654640 -1 reset not still connected to 0x55ed0e550dd0 2024-09-11T06:34:03.241 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:34:03.240+0000 7fb2ca654640 -1 reset not still connected to 0x55ed12570820 2024-09-11T06:34:03.241 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:34:03.240+0000 7fb2ca654640 -1 reset not still connected to 0x55ed149fe8f0 2024-09-11T06:34:03.241 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:34:03.240+0000 7fb2ca654640 -1 reset not still connected to 0x55ed161c2000 2024-09-11T06:34:03.241 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:34:03.240+0000 7fb2ca654640 -1 reset not still connected to 0x55ed194eaea0 2024-09-11T06:34:03.242 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:34:03.240+0000 7fb2ca654640 -1 reset not still connected to 0x55ed1bd1add0 2024-09-11T06:34:04.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ error_code=124 2024-09-11T06:34:04.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set -e 2024-09-11T06:34:04.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T06:34:04.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:34:04.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:34:04.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_22 2024-09-11T06:34:04.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_image_and_enable_mirror cluster2 mirror image_22 journal 128M 2024-09-11T06:34:04.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:34:04.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:34:04.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:34:04.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:34:04.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_22 2024-09-11T06:34:04.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:34:04.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local mode=journal 2024-09-11T06:34:04.706 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n journal ']' 2024-09-11T06:34:04.706 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:34:04.706 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_image cluster2 mirror image_22 128M 2024-09-11T06:34:04.706 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:34:04.706 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:34:04.706 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:34:04.706 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:34:04.706 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_22 2024-09-11T06:34:04.706 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:34:04.706 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local size=128 2024-09-11T06:34:04.706 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n 128M ']' 2024-09-11T06:34:04.706 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ size=128M 2024-09-11T06:34:04.706 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:34:04.706 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_22 2024-09-11T06:34:04.810 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T06:34:04.810 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:34:04.810 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' pool = image ']' 2024-09-11T06:34:04.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ case "${pool}" in 2024-09-11T06:34:04.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 1 2024-09-11T06:34:04.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n '' ']' 2024-09-11T06:34:04.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ stress_write_image cluster2 mirror image_22 2024-09-11T06:34:04.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:34:04.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:34:04.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_22 2024-09-11T06:34:04.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T06:34:04.812 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local duration=6 2024-09-11T06:34:04.812 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set +e 2024-09-11T06:34:04.812 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_22 --debug-rbd=20 --debug-journaler=20 2024-09-11T06:34:08.175 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:100 IOs, 32136192 bytes 2024-09-11T06:34:10.820 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ error_code=124 2024-09-11T06:34:10.820 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set -e 2024-09-11T06:34:10.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T06:34:10.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:34:10.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:34:10.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_23 2024-09-11T06:34:10.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_image_and_enable_mirror cluster2 mirror image_23 journal 128M 2024-09-11T06:34:10.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:34:10.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:34:10.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:34:10.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:34:10.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_23 2024-09-11T06:34:10.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:34:10.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local mode=journal 2024-09-11T06:34:10.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n journal ']' 2024-09-11T06:34:10.822 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:34:10.822 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_image cluster2 mirror image_23 128M 2024-09-11T06:34:10.822 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:34:10.822 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:34:10.822 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:34:10.822 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:34:10.822 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_23 2024-09-11T06:34:10.822 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:34:10.822 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local size=128 2024-09-11T06:34:10.822 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n 128M ']' 2024-09-11T06:34:10.822 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ size=128M 2024-09-11T06:34:10.822 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:34:10.822 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_23 2024-09-11T06:34:11.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T06:34:11.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:34:11.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' pool = image ']' 2024-09-11T06:34:11.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ case "${pool}" in 2024-09-11T06:34:11.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 1 2024-09-11T06:34:11.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n '' ']' 2024-09-11T06:34:11.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ stress_write_image cluster2 mirror image_23 2024-09-11T06:34:11.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:34:11.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:34:11.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_23 2024-09-11T06:34:11.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T06:34:11.002 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local duration=9 2024-09-11T06:34:11.002 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set +e 2024-09-11T06:34:11.002 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_23 --debug-rbd=20 --debug-journaler=20 2024-09-11T06:34:14.509 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:100 IOs, 43201536 bytes 2024-09-11T06:34:18.270 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:200 IOs, 110252032 bytes 2024-09-11T06:34:20.010 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ error_code=124 2024-09-11T06:34:20.010 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set -e 2024-09-11T06:34:20.010 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T06:34:20.010 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:34:20.010 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:34:20.011 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_24 2024-09-11T06:34:20.011 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_image_and_enable_mirror cluster2 mirror image_24 journal 128M 2024-09-11T06:34:20.011 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:34:20.011 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:34:20.011 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:34:20.011 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:34:20.011 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_24 2024-09-11T06:34:20.011 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:34:20.011 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local mode=journal 2024-09-11T06:34:20.011 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n journal ']' 2024-09-11T06:34:20.011 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:34:20.011 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_image cluster2 mirror image_24 128M 2024-09-11T06:34:20.011 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:34:20.012 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:34:20.012 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:34:20.012 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:34:20.012 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_24 2024-09-11T06:34:20.012 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:34:20.012 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local size=128 2024-09-11T06:34:20.012 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n 128M ']' 2024-09-11T06:34:20.012 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ size=128M 2024-09-11T06:34:20.012 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:34:20.012 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_24 2024-09-11T06:34:20.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T06:34:20.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:34:20.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' pool = image ']' 2024-09-11T06:34:20.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ case "${pool}" in 2024-09-11T06:34:20.128 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 1 2024-09-11T06:34:20.128 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n '' ']' 2024-09-11T06:34:20.128 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ stress_write_image cluster2 mirror image_24 2024-09-11T06:34:20.128 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:34:20.128 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:34:20.128 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_24 2024-09-11T06:34:20.128 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T06:34:20.130 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local duration=9 2024-09-11T06:34:20.130 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set +e 2024-09-11T06:34:20.130 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_24 --debug-rbd=20 --debug-journaler=20 2024-09-11T06:34:20.512 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:34:20.508+0000 7fb2ca654640 -1 reset not still connected to 0x55ed0e363ee0 2024-09-11T06:34:20.516 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:34:20.516+0000 7fb2ca654640 -1 reset not still connected to 0x55ed0e47ba00 2024-09-11T06:34:20.516 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:34:20.516+0000 7fb2ca654640 -1 reset not still connected to 0x55ed11f509c0 2024-09-11T06:34:20.516 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:34:20.516+0000 7fb2ca654640 -1 reset not still connected to 0x55ed1b91f450 2024-09-11T06:34:20.537 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:100 IOs, 617472 bytes 2024-09-11T06:34:23.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:200 IOs, 33646592 bytes 2024-09-11T06:34:26.282 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:300 IOs, 61849600 bytes 2024-09-11T06:34:29.139 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ error_code=124 2024-09-11T06:34:29.139 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set -e 2024-09-11T06:34:29.139 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T06:34:29.139 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:34:29.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:34:29.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_25 2024-09-11T06:34:29.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_image_and_enable_mirror cluster2 mirror image_25 journal 128M 2024-09-11T06:34:29.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:34:29.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:34:29.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:34:29.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:34:29.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_25 2024-09-11T06:34:29.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:34:29.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local mode=journal 2024-09-11T06:34:29.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n journal ']' 2024-09-11T06:34:29.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:34:29.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_image cluster2 mirror image_25 128M 2024-09-11T06:34:29.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:34:29.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:34:29.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:34:29.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:34:29.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_25 2024-09-11T06:34:29.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:34:29.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local size=128 2024-09-11T06:34:29.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n 128M ']' 2024-09-11T06:34:29.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ size=128M 2024-09-11T06:34:29.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:34:29.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_25 2024-09-11T06:34:29.236 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T06:34:29.236 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:34:29.236 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' pool = image ']' 2024-09-11T06:34:29.237 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ case "${pool}" in 2024-09-11T06:34:29.237 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 1 2024-09-11T06:34:29.237 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n '' ']' 2024-09-11T06:34:29.237 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ stress_write_image cluster2 mirror image_25 2024-09-11T06:34:29.237 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:34:29.237 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:34:29.237 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_25 2024-09-11T06:34:29.237 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T06:34:29.238 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local duration=9 2024-09-11T06:34:29.238 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set +e 2024-09-11T06:34:29.238 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_25 --debug-rbd=20 --debug-journaler=20 2024-09-11T06:34:32.069 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:100 IOs, 37190656 bytes 2024-09-11T06:34:32.267 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:34:32.260+0000 7fc41084f640 -1 reset not still connected to 0x55eb15565d40 2024-09-11T06:34:32.267 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:34:32.260+0000 7fc41084f640 -1 reset not still connected to 0x55eb16561450 2024-09-11T06:34:32.267 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:34:32.260+0000 7fc41084f640 -1 reset not still connected to 0x55eb2613e680 2024-09-11T06:34:34.019 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:200 IOs, 58211328 bytes 2024-09-11T06:34:38.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ error_code=124 2024-09-11T06:34:38.254 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set -e 2024-09-11T06:34:38.254 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T06:34:38.254 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:34:38.254 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:34:38.254 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_26 2024-09-11T06:34:38.254 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_image_and_enable_mirror cluster2 mirror image_26 journal 128M 2024-09-11T06:34:38.254 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:34:38.254 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:34:38.254 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:34:38.254 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:34:38.254 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_26 2024-09-11T06:34:38.254 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:34:38.254 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local mode=journal 2024-09-11T06:34:38.254 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n journal ']' 2024-09-11T06:34:38.255 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:34:38.255 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_image cluster2 mirror image_26 128M 2024-09-11T06:34:38.255 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:34:38.255 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:34:38.255 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:34:38.255 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:34:38.255 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_26 2024-09-11T06:34:38.255 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:34:38.255 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local size=128 2024-09-11T06:34:38.255 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n 128M ']' 2024-09-11T06:34:38.255 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ size=128M 2024-09-11T06:34:38.255 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:34:38.255 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_26 2024-09-11T06:34:38.374 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T06:34:38.374 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:34:38.374 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' pool = image ']' 2024-09-11T06:34:38.374 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ case "${pool}" in 2024-09-11T06:34:38.374 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 1 2024-09-11T06:34:38.374 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n '' ']' 2024-09-11T06:34:38.374 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ stress_write_image cluster2 mirror image_26 2024-09-11T06:34:38.374 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:34:38.374 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:34:38.375 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_26 2024-09-11T06:34:38.375 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T06:34:38.376 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local duration=5 2024-09-11T06:34:38.376 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set +e 2024-09-11T06:34:38.376 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_26 --debug-rbd=20 --debug-journaler=20 2024-09-11T06:34:40.600 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:34:40.602+0000 7efd718c5640 -1 reset not still connected to 0x560b14dc4000 2024-09-11T06:34:40.600 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:34:40.602+0000 7efd718c5640 -1 reset not still connected to 0x560b15a70a90 2024-09-11T06:34:40.601 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:34:40.602+0000 7efd718c5640 -1 reset not still connected to 0x560b1899a410 2024-09-11T06:34:41.187 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:34:41.183+0000 7fb45318f640 -1 reset not still connected to 0x561337b67e10 2024-09-11T06:34:41.193 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:34:41.187+0000 7fb45318f640 -1 reset not still connected to 0x56133cabc000 2024-09-11T06:34:41.193 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:34:41.187+0000 7fb45318f640 -1 reset not still connected to 0x561342690270 2024-09-11T06:34:41.276 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:34:41.278+0000 7efd718c5640 -1 reset not still connected to 0x560b35329ad0 2024-09-11T06:34:41.723 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:100 IOs, 38876160 bytes 2024-09-11T06:34:41.888 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:34:41.890+0000 7efd718c5640 -1 reset not still connected to 0x560b14e4e9c0 2024-09-11T06:34:41.888 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:34:41.890+0000 7efd718c5640 -1 reset not still connected to 0x560b170c4270 2024-09-11T06:34:41.888 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:34:41.890+0000 7efd718c5640 -1 reset not still connected to 0x560b1c92f1e0 2024-09-11T06:34:41.889 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:34:41.890+0000 7efd718c5640 -1 reset not still connected to 0x560b35329ad0 2024-09-11T06:34:42.271 INFO:tasks.ceph.cluster2.osd.0.smithi112.stderr:2024-09-11T06:34:42.270+0000 7f3fa0017640 -1 reset not still connected to 0x563d332f71e0 2024-09-11T06:34:42.271 INFO:tasks.ceph.cluster2.osd.0.smithi112.stderr:2024-09-11T06:34:42.270+0000 7f3fa0017640 -1 reset not still connected to 0x563d4619c8f0 2024-09-11T06:34:43.386 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ error_code=124 2024-09-11T06:34:43.386 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set -e 2024-09-11T06:34:43.386 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T06:34:43.386 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:34:43.386 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:34:43.386 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_27 2024-09-11T06:34:43.386 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_image_and_enable_mirror cluster2 mirror image_27 journal 128M 2024-09-11T06:34:43.386 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:34:43.386 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:34:43.386 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:34:43.386 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:34:43.387 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_27 2024-09-11T06:34:43.387 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:34:43.387 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local mode=journal 2024-09-11T06:34:43.387 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n journal ']' 2024-09-11T06:34:43.387 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:34:43.387 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_image cluster2 mirror image_27 128M 2024-09-11T06:34:43.387 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:34:43.387 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:34:43.387 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:34:43.387 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:34:43.387 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_27 2024-09-11T06:34:43.387 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:34:43.387 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local size=128 2024-09-11T06:34:43.388 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n 128M ']' 2024-09-11T06:34:43.388 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ size=128M 2024-09-11T06:34:43.388 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:34:43.388 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_27 2024-09-11T06:34:43.529 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T06:34:43.529 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:34:43.529 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' pool = image ']' 2024-09-11T06:34:43.529 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ case "${pool}" in 2024-09-11T06:34:43.529 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 1 2024-09-11T06:34:43.530 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n '' ']' 2024-09-11T06:34:43.530 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ stress_write_image cluster2 mirror image_27 2024-09-11T06:34:43.530 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:34:43.530 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:34:43.530 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_27 2024-09-11T06:34:43.530 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T06:34:43.531 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local duration=14 2024-09-11T06:34:43.531 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set +e 2024-09-11T06:34:43.531 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_27 --debug-rbd=20 --debug-journaler=20 2024-09-11T06:34:46.494 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:34:46.494+0000 7efd718c5640 -1 reset not still connected to 0x560b37798410 2024-09-11T06:34:46.495 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:34:46.494+0000 7efd718c5640 -1 reset not still connected to 0x560b3a7fa4e0 2024-09-11T06:34:50.357 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:100 IOs, 64600064 bytes 2024-09-11T06:34:53.061 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:34:53.055+0000 7fb45318f640 -1 reset not still connected to 0x561322bdf450 2024-09-11T06:34:53.062 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:34:53.059+0000 7fb45318f640 -1 reset not still connected to 0x561337b67e10 2024-09-11T06:34:53.062 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:34:53.059+0000 7fb45318f640 -1 reset not still connected to 0x56133cabc000 2024-09-11T06:34:53.067 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:34:53.059+0000 7fb45318f640 -1 reset not still connected to 0x561342690270 2024-09-11T06:34:53.067 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:34:53.059+0000 7fb45318f640 -1 reset not still connected to 0x5613480c0a90 2024-09-11T06:34:53.067 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:34:53.059+0000 7fb45318f640 -1 reset not still connected to 0x5613545c3790 2024-09-11T06:34:54.143 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:34:54.139+0000 7fb45318f640 -1 reset not still connected to 0x561355040c30 2024-09-11T06:34:56.100 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:200 IOs, 138770432 bytes 2024-09-11T06:34:56.958 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:300 IOs, 149751808 bytes 2024-09-11T06:34:57.541 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ error_code=124 2024-09-11T06:34:57.541 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set -e 2024-09-11T06:34:57.541 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T06:34:57.541 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:34:57.541 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:34:57.541 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_28 2024-09-11T06:34:57.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_image_and_enable_mirror cluster2 mirror image_28 journal 128M 2024-09-11T06:34:57.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:34:57.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:34:57.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:34:57.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:34:57.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_28 2024-09-11T06:34:57.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:34:57.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local mode=journal 2024-09-11T06:34:57.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n journal ']' 2024-09-11T06:34:57.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:34:57.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_image cluster2 mirror image_28 128M 2024-09-11T06:34:57.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:34:57.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:34:57.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:34:57.543 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:34:57.543 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_28 2024-09-11T06:34:57.543 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:34:57.543 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local size=128 2024-09-11T06:34:57.543 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n 128M ']' 2024-09-11T06:34:57.543 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ size=128M 2024-09-11T06:34:57.543 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:34:57.543 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_28 2024-09-11T06:34:57.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T06:34:57.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:34:57.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' pool = image ']' 2024-09-11T06:34:57.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ case "${pool}" in 2024-09-11T06:34:57.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 1 2024-09-11T06:34:57.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n '' ']' 2024-09-11T06:34:57.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ stress_write_image cluster2 mirror image_28 2024-09-11T06:34:57.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:34:57.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:34:57.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_28 2024-09-11T06:34:57.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T06:34:57.655 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local duration=7 2024-09-11T06:34:57.655 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set +e 2024-09-11T06:34:57.655 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_28 --debug-rbd=20 --debug-journaler=20 2024-09-11T06:34:58.408 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:100 IOs, 3487744 bytes 2024-09-11T06:35:02.804 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:200 IOs, 57954304 bytes 2024-09-11T06:35:04.227 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:300 IOs, 67579904 bytes 2024-09-11T06:35:04.665 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ error_code=124 2024-09-11T06:35:04.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set -e 2024-09-11T06:35:04.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T06:35:04.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:35:04.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:35:04.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_29 2024-09-11T06:35:04.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_image_and_enable_mirror cluster2 mirror image_29 journal 128M 2024-09-11T06:35:04.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:35:04.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:35:04.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:35:04.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:35:04.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_29 2024-09-11T06:35:04.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:35:04.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local mode=journal 2024-09-11T06:35:04.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n journal ']' 2024-09-11T06:35:04.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:35:04.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_image cluster2 mirror image_29 128M 2024-09-11T06:35:04.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:35:04.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:35:04.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:35:04.668 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:35:04.668 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_29 2024-09-11T06:35:04.668 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:35:04.668 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local size=128 2024-09-11T06:35:04.668 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n 128M ']' 2024-09-11T06:35:04.668 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ size=128M 2024-09-11T06:35:04.668 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:35:04.668 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_29 2024-09-11T06:35:04.829 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T06:35:04.829 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:35:04.829 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' pool = image ']' 2024-09-11T06:35:04.829 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ case "${pool}" in 2024-09-11T06:35:04.829 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 1 2024-09-11T06:35:04.829 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n '' ']' 2024-09-11T06:35:04.830 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ stress_write_image cluster2 mirror image_29 2024-09-11T06:35:04.830 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:35:04.830 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:35:04.830 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_29 2024-09-11T06:35:04.830 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T06:35:04.831 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local duration=9 2024-09-11T06:35:04.831 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set +e 2024-09-11T06:35:04.831 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_29 --debug-rbd=20 --debug-journaler=20 2024-09-11T06:35:07.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:100 IOs, 37684224 bytes 2024-09-11T06:35:09.473 INFO:tasks.ceph.cluster2.osd.0.smithi112.stderr:2024-09-11T06:35:09.474+0000 7f3fa0017640 -1 reset not still connected to 0x563d2e638dd0 2024-09-11T06:35:09.473 INFO:tasks.ceph.cluster2.osd.0.smithi112.stderr:2024-09-11T06:35:09.474+0000 7f3fa0017640 -1 reset not still connected to 0x563d346a61a0 2024-09-11T06:35:09.474 INFO:tasks.ceph.cluster2.osd.0.smithi112.stderr:2024-09-11T06:35:09.474+0000 7f3fa0017640 -1 reset not still connected to 0x563d35fe6750 2024-09-11T06:35:09.474 INFO:tasks.ceph.cluster2.osd.0.smithi112.stderr:2024-09-11T06:35:09.474+0000 7f3fa0017640 -1 reset not still connected to 0x563d3b4eb6c0 2024-09-11T06:35:09.474 INFO:tasks.ceph.cluster2.osd.0.smithi112.stderr:2024-09-11T06:35:09.474+0000 7f3fa0017640 -1 reset not still connected to 0x563d46a3dee0 2024-09-11T06:35:09.477 INFO:tasks.ceph.cluster2.osd.0.smithi112.stderr:2024-09-11T06:35:09.478+0000 7f3fa0017640 -1 reset not still connected to 0x563d2e638dd0 2024-09-11T06:35:09.842 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:35:09.839+0000 7fb2ca654640 -1 reset not still connected to 0x55ed0e363ee0 2024-09-11T06:35:09.846 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:35:09.843+0000 7fb2ca654640 -1 reset not still connected to 0x55ed0e47ba00 2024-09-11T06:35:09.846 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:35:09.843+0000 7fb2ca654640 -1 reset not still connected to 0x55ed11f509c0 2024-09-11T06:35:09.847 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:35:09.843+0000 7fb2ca654640 -1 reset not still connected to 0x55ed15c15ad0 2024-09-11T06:35:09.853 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:35:09.847+0000 7fb2ca654640 -1 reset not still connected to 0x55ed15c36d00 2024-09-11T06:35:09.853 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:35:09.847+0000 7fb2ca654640 -1 reset not still connected to 0x55ed169d91e0 2024-09-11T06:35:09.853 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:35:09.847+0000 7fb2ca654640 -1 reset not still connected to 0x55ed1b91f450 2024-09-11T06:35:09.853 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:35:09.847+0000 7fb2ca654640 -1 reset not still connected to 0x55ed288cd110 2024-09-11T06:35:09.856 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:35:09.851+0000 7fb2ca654640 -1 reset not still connected to 0x55ed32b6bba0 2024-09-11T06:35:09.856 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:35:09.851+0000 7fb2ca654640 -1 reset not still connected to 0x55ed369ccc30 2024-09-11T06:35:09.856 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:35:09.851+0000 7fb2ca654640 -1 reset not still connected to 0x55ed3a812820 2024-09-11T06:35:09.856 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:35:09.851+0000 7fb2ca654640 -1 reset not still connected to 0x55ed43611450 2024-09-11T06:35:12.194 INFO:tasks.ceph.cluster2.osd.1.smithi112.stderr:2024-09-11T06:35:12.194+0000 7f2cecaac640 -1 reset not still connected to 0x55daaf92cdd0 2024-09-11T06:35:13.838 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ error_code=124 2024-09-11T06:35:13.838 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set -e 2024-09-11T06:35:13.838 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T06:35:13.838 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:35:13.838 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:35:13.838 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_30 2024-09-11T06:35:13.839 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_image_and_enable_mirror cluster2 mirror image_30 journal 128M 2024-09-11T06:35:13.839 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:35:13.839 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:35:13.839 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:35:13.839 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:35:13.839 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_30 2024-09-11T06:35:13.839 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:35:13.839 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local mode=journal 2024-09-11T06:35:13.839 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n journal ']' 2024-09-11T06:35:13.839 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:35:13.839 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_image cluster2 mirror image_30 128M 2024-09-11T06:35:13.839 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:35:13.839 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:35:13.840 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:35:13.840 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:35:13.840 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_30 2024-09-11T06:35:13.840 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:35:13.840 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local size=128 2024-09-11T06:35:13.840 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n 128M ']' 2024-09-11T06:35:13.840 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ size=128M 2024-09-11T06:35:13.840 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:35:13.840 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_30 2024-09-11T06:35:13.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T06:35:13.985 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:35:13.985 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' pool = image ']' 2024-09-11T06:35:13.985 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ case "${pool}" in 2024-09-11T06:35:13.985 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 1 2024-09-11T06:35:13.985 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n '' ']' 2024-09-11T06:35:13.985 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ stress_write_image cluster2 mirror image_30 2024-09-11T06:35:13.985 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:35:13.985 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:35:13.985 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_30 2024-09-11T06:35:13.985 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T06:35:13.989 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local duration=10 2024-09-11T06:35:13.989 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set +e 2024-09-11T06:35:13.989 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_30 --debug-rbd=20 --debug-journaler=20 2024-09-11T06:35:15.540 INFO:tasks.ceph.cluster2.osd.0.smithi112.stderr:2024-09-11T06:35:15.541+0000 7f3fa0017640 -1 reset not still connected to 0x563d3b4eb6c0 2024-09-11T06:35:15.541 INFO:tasks.ceph.cluster2.osd.0.smithi112.stderr:2024-09-11T06:35:15.541+0000 7f3fa0017640 -1 reset not still connected to 0x563d46a3dee0 2024-09-11T06:35:15.599 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:35:15.597+0000 7efd718c5640 -1 reset not still connected to 0x560b3a5f0750 2024-09-11T06:35:15.894 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:35:15.893+0000 7efd718c5640 -1 reset not still connected to 0x560b1d3f3380 2024-09-11T06:35:15.894 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:35:15.893+0000 7efd718c5640 -1 reset not still connected to 0x560b1d9aa4e0 2024-09-11T06:35:15.900 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:35:15.901+0000 7efd718c5640 -1 reset not still connected to 0x560b29059520 2024-09-11T06:35:15.900 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:35:15.901+0000 7efd718c5640 -1 reset not still connected to 0x560b2a0f7ad0 2024-09-11T06:35:16.725 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:35:16.723+0000 7fb2ca654640 -1 reset not still connected to 0x55ed15c36d00 2024-09-11T06:35:16.725 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:35:16.723+0000 7fb2ca654640 -1 reset not still connected to 0x55ed169d91e0 2024-09-11T06:35:16.725 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:35:16.723+0000 7fb2ca654640 -1 reset not still connected to 0x55ed1b91f450 2024-09-11T06:35:16.726 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:35:16.723+0000 7fb2ca654640 -1 reset not still connected to 0x55ed1c2d45b0 2024-09-11T06:35:16.726 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:35:16.723+0000 7fb2ca654640 -1 reset not still connected to 0x55ed288cd110 2024-09-11T06:35:16.727 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:35:16.723+0000 7fb2ca654640 -1 reset not still connected to 0x55ed32b6bba0 2024-09-11T06:35:16.728 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:35:16.723+0000 7fb2ca654640 -1 reset not still connected to 0x55ed369ccc30 2024-09-11T06:35:16.728 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:35:16.723+0000 7fb2ca654640 -1 reset not still connected to 0x55ed3a812820 2024-09-11T06:35:16.728 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:35:16.723+0000 7fb2ca654640 -1 reset not still connected to 0x55ed43611450 2024-09-11T06:35:20.078 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:100 IOs, 63806464 bytes 2024-09-11T06:35:21.023 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:200 IOs, 73978880 bytes 2024-09-11T06:35:24.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ error_code=124 2024-09-11T06:35:24.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set -e 2024-09-11T06:35:24.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T06:35:24.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:35:24.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:35:24.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_31 2024-09-11T06:35:24.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_image_and_enable_mirror cluster2 mirror image_31 journal 128M 2024-09-11T06:35:24.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:35:24.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:35:24.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:35:24.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:35:24.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_31 2024-09-11T06:35:24.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:35:24.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local mode=journal 2024-09-11T06:35:24.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n journal ']' 2024-09-11T06:35:24.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:35:24.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_image cluster2 mirror image_31 128M 2024-09-11T06:35:24.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:35:24.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:35:24.002 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:35:24.002 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:35:24.002 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_31 2024-09-11T06:35:24.002 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:35:24.002 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local size=128 2024-09-11T06:35:24.002 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n 128M ']' 2024-09-11T06:35:24.002 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ size=128M 2024-09-11T06:35:24.002 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:35:24.002 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_31 2024-09-11T06:35:24.113 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T06:35:24.114 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:35:24.114 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' pool = image ']' 2024-09-11T06:35:24.114 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ case "${pool}" in 2024-09-11T06:35:24.114 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 1 2024-09-11T06:35:24.114 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n '' ']' 2024-09-11T06:35:24.114 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ stress_write_image cluster2 mirror image_31 2024-09-11T06:35:24.114 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:35:24.114 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:35:24.114 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_31 2024-09-11T06:35:24.114 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T06:35:24.115 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local duration=14 2024-09-11T06:35:24.115 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set +e 2024-09-11T06:35:24.116 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_31 --debug-rbd=20 --debug-journaler=20 2024-09-11T06:35:28.076 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:100 IOs, 50019328 bytes 2024-09-11T06:35:29.608 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:200 IOs, 64276480 bytes 2024-09-11T06:35:35.937 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:300 IOs, 122632192 bytes 2024-09-11T06:35:38.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ error_code=124 2024-09-11T06:35:38.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set -e 2024-09-11T06:35:38.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T06:35:38.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:35:38.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:35:38.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_32 2024-09-11T06:35:38.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_image_and_enable_mirror cluster2 mirror image_32 journal 128M 2024-09-11T06:35:38.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:35:38.128 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:35:38.128 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:35:38.128 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:35:38.128 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_32 2024-09-11T06:35:38.128 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:35:38.128 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local mode=journal 2024-09-11T06:35:38.128 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n journal ']' 2024-09-11T06:35:38.128 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:35:38.128 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_image cluster2 mirror image_32 128M 2024-09-11T06:35:38.128 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:35:38.128 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:35:38.128 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:35:38.128 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:35:38.129 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_32 2024-09-11T06:35:38.129 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:35:38.129 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local size=128 2024-09-11T06:35:38.129 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n 128M ']' 2024-09-11T06:35:38.129 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ size=128M 2024-09-11T06:35:38.129 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:35:38.129 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_32 2024-09-11T06:35:38.252 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T06:35:38.252 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:35:38.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' pool = image ']' 2024-09-11T06:35:38.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ case "${pool}" in 2024-09-11T06:35:38.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 1 2024-09-11T06:35:38.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n '' ']' 2024-09-11T06:35:38.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ stress_write_image cluster2 mirror image_32 2024-09-11T06:35:38.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:35:38.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:35:38.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_32 2024-09-11T06:35:38.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T06:35:38.255 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local duration=6 2024-09-11T06:35:38.255 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set +e 2024-09-11T06:35:38.255 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_32 --debug-rbd=20 --debug-journaler=20 2024-09-11T06:35:39.056 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:100 IOs, 25732096 bytes 2024-09-11T06:35:41.962 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:200 IOs, 29699072 bytes 2024-09-11T06:35:42.997 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:300 IOs, 37301248 bytes 2024-09-11T06:35:44.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ error_code=124 2024-09-11T06:35:44.264 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set -e 2024-09-11T06:35:44.264 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T06:35:44.264 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:35:44.264 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:35:44.264 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_33 2024-09-11T06:35:44.264 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_image_and_enable_mirror cluster2 mirror image_33 journal 128M 2024-09-11T06:35:44.264 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:35:44.264 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:35:44.264 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:35:44.264 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:35:44.264 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_33 2024-09-11T06:35:44.264 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:35:44.264 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local mode=journal 2024-09-11T06:35:44.265 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n journal ']' 2024-09-11T06:35:44.265 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:35:44.265 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_image cluster2 mirror image_33 128M 2024-09-11T06:35:44.265 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:35:44.265 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:35:44.265 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:35:44.265 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:35:44.265 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_33 2024-09-11T06:35:44.265 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:35:44.265 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local size=128 2024-09-11T06:35:44.265 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n 128M ']' 2024-09-11T06:35:44.265 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ size=128M 2024-09-11T06:35:44.265 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:35:44.265 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_33 2024-09-11T06:35:44.389 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T06:35:44.390 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:35:44.390 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' pool = image ']' 2024-09-11T06:35:44.390 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ case "${pool}" in 2024-09-11T06:35:44.390 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 1 2024-09-11T06:35:44.390 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n '' ']' 2024-09-11T06:35:44.390 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ stress_write_image cluster2 mirror image_33 2024-09-11T06:35:44.390 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:35:44.390 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:35:44.390 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_33 2024-09-11T06:35:44.390 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T06:35:44.392 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local duration=6 2024-09-11T06:35:44.392 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set +e 2024-09-11T06:35:44.392 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_33 --debug-rbd=20 --debug-journaler=20 2024-09-11T06:35:44.833 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:35:44.833+0000 7efd718c5640 -1 reset not still connected to 0x560b15a70dd0 2024-09-11T06:35:47.290 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:35:47.287+0000 7fb45318f640 -1 reset not still connected to 0x561322bdf450 2024-09-11T06:35:47.290 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:35:47.287+0000 7fb45318f640 -1 reset not still connected to 0x561328b5c680 2024-09-11T06:35:47.291 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:35:47.287+0000 7fb45318f640 -1 reset not still connected to 0x561337b67e10 2024-09-11T06:35:47.291 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:35:47.287+0000 7fb45318f640 -1 reset not still connected to 0x56133cabc000 2024-09-11T06:35:47.294 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:35:47.291+0000 7fb45318f640 -1 reset not still connected to 0x561342690270 2024-09-11T06:35:47.294 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:35:47.291+0000 7fb45318f640 -1 reset not still connected to 0x5613480c0a90 2024-09-11T06:35:47.294 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:35:47.291+0000 7fb45318f640 -1 reset not still connected to 0x5613480c1040 2024-09-11T06:35:47.294 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:35:47.291+0000 7fb45318f640 -1 reset not still connected to 0x5613545c3790 2024-09-11T06:35:47.872 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:35:47.869+0000 7efd718c5640 -1 reset not still connected to 0x560b15a729c0 2024-09-11T06:35:47.880 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:35:47.873+0000 7efd718c5640 -1 reset not still connected to 0x560b1d3f3380 2024-09-11T06:35:47.882 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:35:47.877+0000 7efd718c5640 -1 reset not still connected to 0x560b29059520 2024-09-11T06:35:47.883 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:35:47.881+0000 7efd718c5640 -1 reset not still connected to 0x560b2fc524e0 2024-09-11T06:35:47.883 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:35:47.881+0000 7efd718c5640 -1 reset not still connected to 0x560b37798410 2024-09-11T06:35:47.891 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:35:47.893+0000 7efd718c5640 -1 reset not still connected to 0x560b3a7fa4e0 2024-09-11T06:35:48.348 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:35:48.345+0000 7efd718c5640 -1 reset not still connected to 0x560b15963930 2024-09-11T06:35:48.360 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:35:48.357+0000 7efd718c5640 -1 reset not still connected to 0x560b15a729c0 2024-09-11T06:35:48.360 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:35:48.357+0000 7efd718c5640 -1 reset not still connected to 0x560b17cd6f70 2024-09-11T06:35:48.360 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:35:48.357+0000 7efd718c5640 -1 reset not still connected to 0x560b186a8d00 2024-09-11T06:35:48.360 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:35:48.357+0000 7efd718c5640 -1 reset not still connected to 0x560b1899a410 2024-09-11T06:35:48.360 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:35:48.357+0000 7efd718c5640 -1 reset not still connected to 0x560b1d3f3380 2024-09-11T06:35:48.360 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:35:48.357+0000 7efd718c5640 -1 reset not still connected to 0x560b27668b60 2024-09-11T06:35:48.360 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:35:48.357+0000 7efd718c5640 -1 reset not still connected to 0x560b29059520 2024-09-11T06:35:48.361 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:35:48.357+0000 7efd718c5640 -1 reset not still connected to 0x560b2ea5d2b0 2024-09-11T06:35:48.361 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:35:48.357+0000 7efd718c5640 -1 reset not still connected to 0x560b2fc524e0 2024-09-11T06:35:48.361 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:35:48.357+0000 7efd718c5640 -1 reset not still connected to 0x560b37798410 2024-09-11T06:35:48.361 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:35:48.357+0000 7efd718c5640 -1 reset not still connected to 0x560b3a7fa4e0 2024-09-11T06:35:50.068 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:100 IOs, 43926528 bytes 2024-09-11T06:35:50.336 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:35:50.331+0000 7fb2ca654640 -1 reset not still connected to 0x55ed0d758270 2024-09-11T06:35:50.336 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:35:50.331+0000 7fb2ca654640 -1 reset not still connected to 0x55ed0d759d40 2024-09-11T06:35:50.336 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:35:50.331+0000 7fb2ca654640 -1 reset not still connected to 0x55ed0d863040 2024-09-11T06:35:50.336 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:35:50.331+0000 7fb2ca654640 -1 reset not still connected to 0x55ed0d865e10 2024-09-11T06:35:50.337 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:35:50.335+0000 7fb2ca654640 -1 reset not still connected to 0x55ed0d8cd5f0 2024-09-11T06:35:50.337 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:35:50.335+0000 7fb2ca654640 -1 reset not still connected to 0x55ed149fe8f0 2024-09-11T06:35:50.337 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:35:50.335+0000 7fb2ca654640 -1 reset not still connected to 0x55ed161c2000 2024-09-11T06:35:50.339 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:35:50.335+0000 7fb2ca654640 -1 reset not still connected to 0x55ed194eaea0 2024-09-11T06:35:50.339 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:35:50.335+0000 7fb2ca654640 -1 reset not still connected to 0x55ed2a1e0ea0 2024-09-11T06:35:50.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ error_code=124 2024-09-11T06:35:50.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set -e 2024-09-11T06:35:50.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T06:35:50.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:35:50.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:35:50.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_34 2024-09-11T06:35:50.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_image_and_enable_mirror cluster2 mirror image_34 journal 128M 2024-09-11T06:35:50.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:35:50.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:35:50.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:35:50.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:35:50.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_34 2024-09-11T06:35:50.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:35:50.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local mode=journal 2024-09-11T06:35:50.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n journal ']' 2024-09-11T06:35:50.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:35:50.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_image cluster2 mirror image_34 128M 2024-09-11T06:35:50.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:35:50.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:35:50.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:35:50.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:35:50.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_34 2024-09-11T06:35:50.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:35:50.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local size=128 2024-09-11T06:35:50.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n 128M ']' 2024-09-11T06:35:50.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ size=128M 2024-09-11T06:35:50.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:35:50.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_34 2024-09-11T06:35:50.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T06:35:50.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:35:50.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' pool = image ']' 2024-09-11T06:35:50.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ case "${pool}" in 2024-09-11T06:35:50.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 1 2024-09-11T06:35:50.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n '' ']' 2024-09-11T06:35:50.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ stress_write_image cluster2 mirror image_34 2024-09-11T06:35:50.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:35:50.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:35:50.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_34 2024-09-11T06:35:50.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T06:35:50.570 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local duration=9 2024-09-11T06:35:50.570 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set +e 2024-09-11T06:35:50.570 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_34 --debug-rbd=20 --debug-journaler=20 2024-09-11T06:35:50.670 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:35:50.667+0000 7fb45318f640 -1 reset not still connected to 0x561315ba4270 2024-09-11T06:35:50.673 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:35:50.667+0000 7fb45318f640 -1 reset not still connected to 0x561315ba5520 2024-09-11T06:35:50.673 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:35:50.667+0000 7fb45318f640 -1 reset not still connected to 0x56131a1ffe10 2024-09-11T06:35:50.674 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:35:50.671+0000 7fb45318f640 -1 reset not still connected to 0x561322931860 2024-09-11T06:35:50.674 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:35:50.671+0000 7fb45318f640 -1 reset not still connected to 0x561322ded930 2024-09-11T06:35:50.674 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:35:50.671+0000 7fb45318f640 -1 reset not still connected to 0x561330ed0410 2024-09-11T06:35:50.677 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:35:50.675+0000 7fb45318f640 -1 reset not still connected to 0x56133339c410 2024-09-11T06:35:50.677 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:35:50.675+0000 7fb45318f640 -1 reset not still connected to 0x56133c1909c0 2024-09-11T06:35:50.679 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:35:50.675+0000 7fb45318f640 -1 reset not still connected to 0x5613480e41a0 2024-09-11T06:35:50.679 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:35:50.675+0000 7fb45318f640 -1 reset not still connected to 0x56134d76fe10 2024-09-11T06:35:50.679 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:35:50.675+0000 7fb45318f640 -1 reset not still connected to 0x56134d81fd40 2024-09-11T06:35:50.679 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:35:50.675+0000 7fb45318f640 -1 reset not still connected to 0x561354d57040 2024-09-11T06:35:50.679 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:35:50.675+0000 7fb45318f640 -1 reset not still connected to 0x561354d77110 2024-09-11T06:35:50.679 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:35:50.675+0000 7fb45318f640 -1 reset not still connected to 0x561355040c30 2024-09-11T06:35:50.679 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:35:50.675+0000 7fb45318f640 -1 reset not still connected to 0x5613551fd110 2024-09-11T06:35:50.687 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:35:50.685+0000 7efd718c5640 -1 reset not still connected to 0x560b35329ad0 2024-09-11T06:35:50.694 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:35:50.693+0000 7efd718c5640 -1 reset not still connected to 0x560b3a5f0750 2024-09-11T06:35:50.698 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:35:50.697+0000 7efd718c5640 -1 reset not still connected to 0x560b3b89c0d0 2024-09-11T06:35:50.698 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:35:50.697+0000 7efd718c5640 -1 reset not still connected to 0x560b3e445040 2024-09-11T06:35:51.050 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:35:51.047+0000 7fb45318f640 -1 reset not still connected to 0x561315ba5520 2024-09-11T06:35:51.051 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:35:51.047+0000 7fb45318f640 -1 reset not still connected to 0x56131a1ffe10 2024-09-11T06:35:51.052 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:35:51.047+0000 7fb45318f640 -1 reset not still connected to 0x561322931860 2024-09-11T06:35:51.055 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:35:51.051+0000 7fb45318f640 -1 reset not still connected to 0x561322ded930 2024-09-11T06:35:51.055 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:35:51.051+0000 7fb45318f640 -1 reset not still connected to 0x561330ed0410 2024-09-11T06:35:51.055 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:35:51.051+0000 7fb45318f640 -1 reset not still connected to 0x56133339c410 2024-09-11T06:35:51.056 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:35:51.051+0000 7fb45318f640 -1 reset not still connected to 0x56133c1909c0 2024-09-11T06:35:51.059 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:35:51.055+0000 7fb45318f640 -1 reset not still connected to 0x5613480e41a0 2024-09-11T06:35:51.059 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:35:51.055+0000 7fb45318f640 -1 reset not still connected to 0x56134d76fe10 2024-09-11T06:35:51.059 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:35:51.055+0000 7fb45318f640 -1 reset not still connected to 0x56134d81fd40 2024-09-11T06:35:51.059 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:35:51.055+0000 7fb45318f640 -1 reset not still connected to 0x561354d57040 2024-09-11T06:35:51.059 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:35:51.055+0000 7fb45318f640 -1 reset not still connected to 0x561354d77110 2024-09-11T06:35:51.059 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:35:51.055+0000 7fb45318f640 -1 reset not still connected to 0x561355040c30 2024-09-11T06:35:51.059 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:35:51.055+0000 7fb45318f640 -1 reset not still connected to 0x5613551fd110 2024-09-11T06:35:51.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:100 IOs, 586752 bytes 2024-09-11T06:35:52.550 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:35:52.543+0000 7fb45318f640 -1 reset not still connected to 0x561337b67e10 2024-09-11T06:35:52.550 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:35:52.543+0000 7fb45318f640 -1 reset not still connected to 0x56133cabc000 2024-09-11T06:35:52.551 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:35:52.547+0000 7fb45318f640 -1 reset not still connected to 0x561342690270 2024-09-11T06:35:52.551 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:35:52.547+0000 7fb45318f640 -1 reset not still connected to 0x5613480c0a90 2024-09-11T06:35:52.551 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:35:52.547+0000 7fb45318f640 -1 reset not still connected to 0x5613480c1040 2024-09-11T06:35:52.551 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:35:52.547+0000 7fb45318f640 -1 reset not still connected to 0x5613545c3790 2024-09-11T06:35:54.807 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:200 IOs, 45867008 bytes 2024-09-11T06:35:56.716 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:300 IOs, 74738688 bytes 2024-09-11T06:35:59.578 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ error_code=124 2024-09-11T06:35:59.578 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set -e 2024-09-11T06:35:59.578 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T06:35:59.578 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:35:59.578 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:35:59.578 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_35 2024-09-11T06:35:59.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_image_and_enable_mirror cluster2 mirror image_35 journal 128M 2024-09-11T06:35:59.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:35:59.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:35:59.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:35:59.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:35:59.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_35 2024-09-11T06:35:59.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:35:59.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local mode=journal 2024-09-11T06:35:59.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n journal ']' 2024-09-11T06:35:59.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:35:59.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_image cluster2 mirror image_35 128M 2024-09-11T06:35:59.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:35:59.580 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:35:59.580 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:35:59.580 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:35:59.580 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_35 2024-09-11T06:35:59.580 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:35:59.580 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local size=128 2024-09-11T06:35:59.580 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n 128M ']' 2024-09-11T06:35:59.580 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ size=128M 2024-09-11T06:35:59.580 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:35:59.580 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_35 2024-09-11T06:35:59.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T06:35:59.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:35:59.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' pool = image ']' 2024-09-11T06:35:59.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ case "${pool}" in 2024-09-11T06:35:59.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 1 2024-09-11T06:35:59.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n '' ']' 2024-09-11T06:35:59.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ stress_write_image cluster2 mirror image_35 2024-09-11T06:35:59.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:35:59.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:35:59.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_35 2024-09-11T06:35:59.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T06:35:59.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local duration=13 2024-09-11T06:35:59.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set +e 2024-09-11T06:35:59.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_35 --debug-rbd=20 --debug-journaler=20 2024-09-11T06:36:00.385 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:100 IOs, 2889728 bytes 2024-09-11T06:36:00.724 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:200 IOs, 3494912 bytes 2024-09-11T06:36:01.208 INFO:tasks.ceph.cluster2.osd.1.smithi112.stderr:2024-09-11T06:36:01.204+0000 7f2cecaac640 -1 reset not still connected to 0x55da8f389930 2024-09-11T06:36:01.208 INFO:tasks.ceph.cluster2.osd.1.smithi112.stderr:2024-09-11T06:36:01.204+0000 7f2cecaac640 -1 reset not still connected to 0x55da941c60d0 2024-09-11T06:36:01.208 INFO:tasks.ceph.cluster2.osd.1.smithi112.stderr:2024-09-11T06:36:01.204+0000 7f2cecaac640 -1 reset not still connected to 0x55daac3e89c0 2024-09-11T06:36:01.208 INFO:tasks.ceph.cluster2.osd.1.smithi112.stderr:2024-09-11T06:36:01.204+0000 7f2cecaac640 -1 reset not still connected to 0x55daaf92cdd0 2024-09-11T06:36:02.544 INFO:tasks.ceph.cluster2.osd.1.smithi112.stderr:2024-09-11T06:36:02.544+0000 7f2cecaac640 -1 reset not still connected to 0x55da8f389930 2024-09-11T06:36:02.545 INFO:tasks.ceph.cluster2.osd.1.smithi112.stderr:2024-09-11T06:36:02.544+0000 7f2cecaac640 -1 reset not still connected to 0x55da941c60d0 2024-09-11T06:36:02.545 INFO:tasks.ceph.cluster2.osd.1.smithi112.stderr:2024-09-11T06:36:02.544+0000 7f2cecaac640 -1 reset not still connected to 0x55daac3e89c0 2024-09-11T06:36:02.545 INFO:tasks.ceph.cluster2.osd.1.smithi112.stderr:2024-09-11T06:36:02.544+0000 7f2cecaac640 -1 reset not still connected to 0x55daaf92cdd0 2024-09-11T06:36:05.315 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:300 IOs, 55212032 bytes 2024-09-11T06:36:05.906 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:05.902+0000 7fb45318f640 -1 reset not still connected to 0x561315884d00 2024-09-11T06:36:05.906 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:05.902+0000 7fb45318f640 -1 reset not still connected to 0x561315ba4270 2024-09-11T06:36:05.907 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:05.902+0000 7fb45318f640 -1 reset not still connected to 0x561315ba5520 2024-09-11T06:36:05.908 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:05.902+0000 7fb45318f640 -1 reset not still connected to 0x56131a1ffe10 2024-09-11T06:36:05.910 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:05.902+0000 7fb45318f640 -1 reset not still connected to 0x561322931860 2024-09-11T06:36:05.913 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:05.906+0000 7fb45318f640 -1 reset not still connected to 0x561322ded930 2024-09-11T06:36:05.913 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:05.906+0000 7fb45318f640 -1 reset not still connected to 0x561330ed0410 2024-09-11T06:36:05.913 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:05.906+0000 7fb45318f640 -1 reset not still connected to 0x56133339c410 2024-09-11T06:36:05.913 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:05.906+0000 7fb45318f640 -1 reset not still connected to 0x5613480872b0 2024-09-11T06:36:05.913 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:05.906+0000 7fb45318f640 -1 reset not still connected to 0x5613480e41a0 2024-09-11T06:36:05.913 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:05.906+0000 7fb45318f640 -1 reset not still connected to 0x56134d76fe10 2024-09-11T06:36:05.913 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:05.906+0000 7fb45318f640 -1 reset not still connected to 0x56134d81fd40 2024-09-11T06:36:05.913 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:05.906+0000 7fb45318f640 -1 reset not still connected to 0x561354d57040 2024-09-11T06:36:05.914 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:05.906+0000 7fb45318f640 -1 reset not still connected to 0x561354d77110 2024-09-11T06:36:05.914 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:05.906+0000 7fb45318f640 -1 reset not still connected to 0x561355040c30 2024-09-11T06:36:05.914 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:05.906+0000 7fb45318f640 -1 reset not still connected to 0x5613551fd110 2024-09-11T06:36:07.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:400 IOs, 76568576 bytes 2024-09-11T06:36:07.954 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:36:07.952+0000 7efd718c5640 -1 reset not still connected to 0x560b15963930 2024-09-11T06:36:07.958 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:36:07.956+0000 7efd718c5640 -1 reset not still connected to 0x560b15a729c0 2024-09-11T06:36:07.958 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:36:07.956+0000 7efd718c5640 -1 reset not still connected to 0x560b186a8d00 2024-09-11T06:36:07.958 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:36:07.956+0000 7efd718c5640 -1 reset not still connected to 0x560b1899a410 2024-09-11T06:36:07.958 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:36:07.956+0000 7efd718c5640 -1 reset not still connected to 0x560b1d3f3380 2024-09-11T06:36:07.958 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:36:07.956+0000 7efd718c5640 -1 reset not still connected to 0x560b29059520 2024-09-11T06:36:07.958 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:36:07.956+0000 7efd718c5640 -1 reset not still connected to 0x560b2ea5d2b0 2024-09-11T06:36:07.958 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:36:07.956+0000 7efd718c5640 -1 reset not still connected to 0x560b2fc524e0 2024-09-11T06:36:07.958 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:36:07.956+0000 7efd718c5640 -1 reset not still connected to 0x560b37798410 2024-09-11T06:36:07.962 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:36:07.960+0000 7efd718c5640 -1 reset not still connected to 0x560b3a7fa4e0 2024-09-11T06:36:08.340 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:36:08.334+0000 7fc41084f640 -1 reset not still connected to 0x55eb354a21a0 2024-09-11T06:36:08.345 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:36:08.338+0000 7fc41084f640 -1 reset not still connected to 0x55eb3893ac30 2024-09-11T06:36:08.345 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:36:08.338+0000 7fc41084f640 -1 reset not still connected to 0x55eb53d56f70 2024-09-11T06:36:09.473 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:500 IOs, 93668352 bytes 2024-09-11T06:36:10.595 INFO:tasks.ceph.cluster2.osd.0.smithi112.stderr:2024-09-11T06:36:10.596+0000 7f3fa0017640 -1 reset not still connected to 0x563d35fe6750 2024-09-11T06:36:10.596 INFO:tasks.ceph.cluster2.osd.0.smithi112.stderr:2024-09-11T06:36:10.596+0000 7f3fa0017640 -1 reset not still connected to 0x563d3838dba0 2024-09-11T06:36:10.596 INFO:tasks.ceph.cluster2.osd.0.smithi112.stderr:2024-09-11T06:36:10.596+0000 7f3fa0017640 -1 reset not still connected to 0x563d46a3dee0 2024-09-11T06:36:10.932 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:36:10.926+0000 7fc41084f640 -1 reset not still connected to 0x55eb354a21a0 2024-09-11T06:36:10.941 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:600 IOs, 106317824 bytes 2024-09-11T06:36:11.714 INFO:tasks.ceph.cluster2.osd.0.smithi112.stderr:2024-09-11T06:36:11.712+0000 7f3fa0017640 -1 reset not still connected to 0x563d43515c70 2024-09-11T06:36:11.718 INFO:tasks.ceph.cluster2.osd.0.smithi112.stderr:2024-09-11T06:36:11.716+0000 7f3fa0017640 -1 reset not still connected to 0x563d4619c8f0 2024-09-11T06:36:11.868 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:700 IOs, 115912704 bytes 2024-09-11T06:36:12.727 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ error_code=124 2024-09-11T06:36:12.727 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set -e 2024-09-11T06:36:12.727 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T06:36:12.727 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:36:12.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:36:12.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_36 2024-09-11T06:36:12.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_image_and_enable_mirror cluster2 mirror image_36 journal 128M 2024-09-11T06:36:12.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:36:12.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:36:12.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:36:12.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:36:12.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_36 2024-09-11T06:36:12.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:36:12.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local mode=journal 2024-09-11T06:36:12.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n journal ']' 2024-09-11T06:36:12.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:36:12.729 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_image cluster2 mirror image_36 128M 2024-09-11T06:36:12.729 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:36:12.729 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:36:12.729 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:36:12.729 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:36:12.729 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_36 2024-09-11T06:36:12.729 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:36:12.729 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local size=128 2024-09-11T06:36:12.729 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n 128M ']' 2024-09-11T06:36:12.729 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ size=128M 2024-09-11T06:36:12.729 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:36:12.729 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_36 2024-09-11T06:36:13.248 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T06:36:13.248 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:36:13.248 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' pool = image ']' 2024-09-11T06:36:13.248 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ case "${pool}" in 2024-09-11T06:36:13.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 1 2024-09-11T06:36:13.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n '' ']' 2024-09-11T06:36:13.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ stress_write_image cluster2 mirror image_36 2024-09-11T06:36:13.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:36:13.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:36:13.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_36 2024-09-11T06:36:13.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T06:36:13.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local duration=13 2024-09-11T06:36:13.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set +e 2024-09-11T06:36:13.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_36 --debug-rbd=20 --debug-journaler=20 2024-09-11T06:36:14.933 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:14.930+0000 7fb45318f640 -1 reset not still connected to 0x561322bdf450 2024-09-11T06:36:14.934 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:14.930+0000 7fb45318f640 -1 reset not still connected to 0x5613273cfd40 2024-09-11T06:36:14.938 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:14.930+0000 7fb45318f640 -1 reset not still connected to 0x561328b5c680 2024-09-11T06:36:14.938 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:14.930+0000 7fb45318f640 -1 reset not still connected to 0x561337a04410 2024-09-11T06:36:14.938 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:14.930+0000 7fb45318f640 -1 reset not still connected to 0x561337b67e10 2024-09-11T06:36:14.938 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:14.930+0000 7fb45318f640 -1 reset not still connected to 0x56133cabc000 2024-09-11T06:36:14.938 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:14.930+0000 7fb45318f640 -1 reset not still connected to 0x561342690270 2024-09-11T06:36:14.938 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:14.930+0000 7fb45318f640 -1 reset not still connected to 0x5613480c0a90 2024-09-11T06:36:14.938 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:14.930+0000 7fb45318f640 -1 reset not still connected to 0x5613480c1040 2024-09-11T06:36:14.938 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:14.930+0000 7fb45318f640 -1 reset not still connected to 0x5613545c3790 2024-09-11T06:36:16.122 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:16.114+0000 7fb45318f640 -1 reset not still connected to 0x561322165790 2024-09-11T06:36:16.124 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:16.118+0000 7fb45318f640 -1 reset not still connected to 0x5613225fd930 2024-09-11T06:36:16.125 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:16.118+0000 7fb45318f640 -1 reset not still connected to 0x561322bdf450 2024-09-11T06:36:16.127 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:16.122+0000 7fb45318f640 -1 reset not still connected to 0x561328b5c680 2024-09-11T06:36:16.127 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:16.122+0000 7fb45318f640 -1 reset not still connected to 0x561328b5c8f0 2024-09-11T06:36:16.128 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:16.122+0000 7fb45318f640 -1 reset not still connected to 0x561337a04410 2024-09-11T06:36:16.128 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:16.122+0000 7fb45318f640 -1 reset not still connected to 0x561337b67e10 2024-09-11T06:36:16.128 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:16.122+0000 7fb45318f640 -1 reset not still connected to 0x56133cabc000 2024-09-11T06:36:16.129 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:16.122+0000 7fb45318f640 -1 reset not still connected to 0x561342690270 2024-09-11T06:36:16.129 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:16.122+0000 7fb45318f640 -1 reset not still connected to 0x5613480c0a90 2024-09-11T06:36:16.129 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:16.122+0000 7fb45318f640 -1 reset not still connected to 0x5613480c1040 2024-09-11T06:36:16.129 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:16.122+0000 7fb45318f640 -1 reset not still connected to 0x5613545c3790 2024-09-11T06:36:16.257 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:100 IOs, 20039680 bytes 2024-09-11T06:36:16.301 INFO:tasks.ceph.cluster2.osd.1.smithi112.stderr:2024-09-11T06:36:16.304+0000 7f2cecaac640 -1 reset not still connected to 0x55da8f389930 2024-09-11T06:36:16.301 INFO:tasks.ceph.cluster2.osd.1.smithi112.stderr:2024-09-11T06:36:16.304+0000 7f2cecaac640 -1 reset not still connected to 0x55da93619110 2024-09-11T06:36:16.301 INFO:tasks.ceph.cluster2.osd.1.smithi112.stderr:2024-09-11T06:36:16.304+0000 7f2cecaac640 -1 reset not still connected to 0x55da941c60d0 2024-09-11T06:36:16.301 INFO:tasks.ceph.cluster2.osd.1.smithi112.stderr:2024-09-11T06:36:16.304+0000 7f2cecaac640 -1 reset not still connected to 0x55daa159cf70 2024-09-11T06:36:16.304 INFO:tasks.ceph.cluster2.osd.1.smithi112.stderr:2024-09-11T06:36:16.308+0000 7f2cecaac640 -1 reset not still connected to 0x55daaf92cdd0 2024-09-11T06:36:16.856 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:16.850+0000 7fb45318f640 -1 reset not still connected to 0x561322165790 2024-09-11T06:36:16.856 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:16.850+0000 7fb45318f640 -1 reset not still connected to 0x5613224a2d00 2024-09-11T06:36:16.857 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:16.850+0000 7fb45318f640 -1 reset not still connected to 0x561322bdf450 2024-09-11T06:36:16.857 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:16.850+0000 7fb45318f640 -1 reset not still connected to 0x5613273cfd40 2024-09-11T06:36:16.857 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:16.850+0000 7fb45318f640 -1 reset not still connected to 0x561328b5c680 2024-09-11T06:36:16.857 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:16.850+0000 7fb45318f640 -1 reset not still connected to 0x561337a04410 2024-09-11T06:36:16.857 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:16.850+0000 7fb45318f640 -1 reset not still connected to 0x561337b67e10 2024-09-11T06:36:16.857 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:16.850+0000 7fb45318f640 -1 reset not still connected to 0x56133cabc000 2024-09-11T06:36:16.869 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:16.862+0000 7fb45318f640 -1 reset not still connected to 0x561342690270 2024-09-11T06:36:16.869 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:16.862+0000 7fb45318f640 -1 reset not still connected to 0x5613480c0a90 2024-09-11T06:36:16.869 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:16.862+0000 7fb45318f640 -1 reset not still connected to 0x5613480c1040 2024-09-11T06:36:16.869 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:16.862+0000 7fb45318f640 -1 reset not still connected to 0x5613545c3790 2024-09-11T06:36:16.869 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:16.862+0000 7fb45318f640 -1 reset not still connected to 0x5613191b3a00 2024-09-11T06:36:16.869 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:16.862+0000 7fb45318f640 -1 reset not still connected to 0x561322165790 2024-09-11T06:36:16.870 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:16.862+0000 7fb45318f640 -1 reset not still connected to 0x5613224a2d00 2024-09-11T06:36:16.870 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:16.866+0000 7fb45318f640 -1 reset not still connected to 0x561322bdf450 2024-09-11T06:36:16.870 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:16.866+0000 7fb45318f640 -1 reset not still connected to 0x5613273cfd40 2024-09-11T06:36:16.871 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:16.866+0000 7fb45318f640 -1 reset not still connected to 0x561328b5c680 2024-09-11T06:36:16.871 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:16.866+0000 7fb45318f640 -1 reset not still connected to 0x561337a04410 2024-09-11T06:36:16.871 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:16.866+0000 7fb45318f640 -1 reset not still connected to 0x561337b67e10 2024-09-11T06:36:16.871 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:16.866+0000 7fb45318f640 -1 reset not still connected to 0x56133cabc000 2024-09-11T06:36:16.871 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:16.866+0000 7fb45318f640 -1 reset not still connected to 0x561342690270 2024-09-11T06:36:16.871 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:16.866+0000 7fb45318f640 -1 reset not still connected to 0x5613480c0a90 2024-09-11T06:36:16.871 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:16.866+0000 7fb45318f640 -1 reset not still connected to 0x5613480c1040 2024-09-11T06:36:17.897 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:36:17.894+0000 7fb2ca654640 -1 reset not still connected to 0x55ed44536340 2024-09-11T06:36:17.898 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:36:17.894+0000 7fb2ca654640 -1 reset not still connected to 0x55ed457b9790 2024-09-11T06:36:17.996 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:17.990+0000 7fb45318f640 -1 reset not still connected to 0x561322165790 2024-09-11T06:36:17.996 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:17.990+0000 7fb45318f640 -1 reset not still connected to 0x5613225fd930 2024-09-11T06:36:17.997 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:17.990+0000 7fb45318f640 -1 reset not still connected to 0x561322bdf450 2024-09-11T06:36:17.997 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:17.994+0000 7fb45318f640 -1 reset not still connected to 0x5613273cfd40 2024-09-11T06:36:18.000 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:17.994+0000 7fb45318f640 -1 reset not still connected to 0x561328b5c680 2024-09-11T06:36:18.000 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:17.994+0000 7fb45318f640 -1 reset not still connected to 0x561328b5c8f0 2024-09-11T06:36:18.000 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:17.994+0000 7fb45318f640 -1 reset not still connected to 0x561337a04410 2024-09-11T06:36:18.000 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:17.994+0000 7fb45318f640 -1 reset not still connected to 0x561337b67e10 2024-09-11T06:36:18.000 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:17.994+0000 7fb45318f640 -1 reset not still connected to 0x56133cabc000 2024-09-11T06:36:18.004 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:17.998+0000 7fb45318f640 -1 reset not still connected to 0x561342690270 2024-09-11T06:36:18.004 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:17.998+0000 7fb45318f640 -1 reset not still connected to 0x5613480c0a90 2024-09-11T06:36:18.004 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:17.998+0000 7fb45318f640 -1 reset not still connected to 0x5613480c1040 2024-09-11T06:36:18.008 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:18.002+0000 7fb45318f640 -1 reset not still connected to 0x5613545c3790 2024-09-11T06:36:21.894 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:21.882+0000 7fb45318f640 -1 reset not still connected to 0x561355040c30 2024-09-11T06:36:22.151 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:22.142+0000 7fb45318f640 -1 reset not still connected to 0x56134d81fd40 2024-09-11T06:36:22.151 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:22.146+0000 7fb45318f640 -1 reset not still connected to 0x561354d77110 2024-09-11T06:36:22.151 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:22.146+0000 7fb45318f640 -1 reset not still connected to 0x561355040c30 2024-09-11T06:36:22.323 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:22.318+0000 7fb45318f640 -1 reset not still connected to 0x561318ae0a90 2024-09-11T06:36:22.323 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:22.318+0000 7fb45318f640 -1 reset not still connected to 0x5613191b3a00 2024-09-11T06:36:22.323 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:22.318+0000 7fb45318f640 -1 reset not still connected to 0x561322165790 2024-09-11T06:36:22.323 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:22.318+0000 7fb45318f640 -1 reset not still connected to 0x561322bdf450 2024-09-11T06:36:22.324 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:22.318+0000 7fb45318f640 -1 reset not still connected to 0x561337b67e10 2024-09-11T06:36:22.324 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:22.318+0000 7fb45318f640 -1 reset not still connected to 0x56133cabc000 2024-09-11T06:36:22.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:200 IOs, 102076416 bytes 2024-09-11T06:36:22.948 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:22.942+0000 7fb45318f640 -1 reset not still connected to 0x56134d81fd40 2024-09-11T06:36:22.948 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:22.942+0000 7fb45318f640 -1 reset not still connected to 0x561354d77110 2024-09-11T06:36:22.948 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:22.942+0000 7fb45318f640 -1 reset not still connected to 0x561355040c30 2024-09-11T06:36:23.569 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:23.562+0000 7fb45318f640 -1 reset not still connected to 0x561322165790 2024-09-11T06:36:23.570 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:23.562+0000 7fb45318f640 -1 reset not still connected to 0x5613224a2d00 2024-09-11T06:36:23.570 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:23.562+0000 7fb45318f640 -1 reset not still connected to 0x561322bdf450 2024-09-11T06:36:23.570 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:23.562+0000 7fb45318f640 -1 reset not still connected to 0x561337b67e10 2024-09-11T06:36:23.570 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:23.562+0000 7fb45318f640 -1 reset not still connected to 0x56133cabc000 2024-09-11T06:36:23.570 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:23.566+0000 7fb45318f640 -1 reset not still connected to 0x561342690270 2024-09-11T06:36:23.968 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:300 IOs, 113662976 bytes 2024-09-11T06:36:27.067 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ error_code=124 2024-09-11T06:36:27.068 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set -e 2024-09-11T06:36:27.068 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T06:36:27.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:36:27.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:36:27.071 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_37 2024-09-11T06:36:27.071 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_image_and_enable_mirror cluster2 mirror image_37 journal 128M 2024-09-11T06:36:27.071 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:36:27.071 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:36:27.072 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:36:27.072 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:36:27.072 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_37 2024-09-11T06:36:27.072 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:36:27.072 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local mode=journal 2024-09-11T06:36:27.072 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n journal ']' 2024-09-11T06:36:27.073 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:36:27.073 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_image cluster2 mirror image_37 128M 2024-09-11T06:36:27.073 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:36:27.073 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:36:27.073 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:36:27.073 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:36:27.074 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_37 2024-09-11T06:36:27.074 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:36:27.074 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local size=128 2024-09-11T06:36:27.074 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n 128M ']' 2024-09-11T06:36:27.074 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ size=128M 2024-09-11T06:36:27.074 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:36:27.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_37 2024-09-11T06:36:27.358 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T06:36:27.358 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:36:27.358 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' pool = image ']' 2024-09-11T06:36:27.358 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ case "${pool}" in 2024-09-11T06:36:27.358 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 1 2024-09-11T06:36:27.358 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n '' ']' 2024-09-11T06:36:27.358 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ stress_write_image cluster2 mirror image_37 2024-09-11T06:36:27.358 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:36:27.359 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:36:27.359 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_37 2024-09-11T06:36:27.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T06:36:27.411 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local duration=8 2024-09-11T06:36:27.411 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set +e 2024-09-11T06:36:27.411 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_37 --debug-rbd=20 --debug-journaler=20 2024-09-11T06:36:27.924 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:36:27.918+0000 7fb2ca654640 -1 reset not still connected to 0x55ed161c2000 2024-09-11T06:36:32.311 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:100 IOs, 45580288 bytes 2024-09-11T06:36:33.371 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:36:33.366+0000 7fc41084f640 -1 reset not still connected to 0x55eb2458e680 2024-09-11T06:36:33.371 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:36:33.366+0000 7fc41084f640 -1 reset not still connected to 0x55eb24ad96c0 2024-09-11T06:36:33.371 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:36:33.366+0000 7fc41084f640 -1 reset not still connected to 0x55eb2da77ba0 2024-09-11T06:36:33.371 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:36:33.366+0000 7fc41084f640 -1 reset not still connected to 0x55eb35243040 2024-09-11T06:36:33.371 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:36:33.366+0000 7fc41084f640 -1 reset not still connected to 0x55eb354a21a0 2024-09-11T06:36:33.371 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:36:33.366+0000 7fc41084f640 -1 reset not still connected to 0x55eb3893ac30 2024-09-11T06:36:33.372 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:36:33.366+0000 7fc41084f640 -1 reset not still connected to 0x55eb4f1329c0 2024-09-11T06:36:33.372 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:36:33.366+0000 7fc41084f640 -1 reset not still connected to 0x55eb53d56f70 2024-09-11T06:36:34.104 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:36:34.098+0000 7fc41084f640 -1 reset not still connected to 0x55eb24ad96c0 2024-09-11T06:36:34.104 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:36:34.098+0000 7fc41084f640 -1 reset not still connected to 0x55eb2da77ba0 2024-09-11T06:36:34.108 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:36:34.106+0000 7fc41084f640 -1 reset not still connected to 0x55eb35243040 2024-09-11T06:36:34.109 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:36:34.106+0000 7fc41084f640 -1 reset not still connected to 0x55eb354a21a0 2024-09-11T06:36:34.111 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:36:34.106+0000 7fc41084f640 -1 reset not still connected to 0x55eb3893ac30 2024-09-11T06:36:34.111 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:36:34.106+0000 7fc41084f640 -1 reset not still connected to 0x55eb4f1329c0 2024-09-11T06:36:34.111 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:36:34.106+0000 7fc41084f640 -1 reset not still connected to 0x55eb53d56f70 2024-09-11T06:36:34.143 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:34.134+0000 7fb45318f640 -1 reset not still connected to 0x561315ba4270 2024-09-11T06:36:34.143 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:34.134+0000 7fb45318f640 -1 reset not still connected to 0x561315ba5520 2024-09-11T06:36:34.143 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:34.138+0000 7fb45318f640 -1 reset not still connected to 0x56131a1ffe10 2024-09-11T06:36:34.143 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:34.138+0000 7fb45318f640 -1 reset not still connected to 0x561322931860 2024-09-11T06:36:34.146 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:34.142+0000 7fb45318f640 -1 reset not still connected to 0x561322ded930 2024-09-11T06:36:34.146 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:34.142+0000 7fb45318f640 -1 reset not still connected to 0x561330ed0410 2024-09-11T06:36:34.147 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:34.142+0000 7fb45318f640 -1 reset not still connected to 0x56133339c410 2024-09-11T06:36:34.147 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:34.142+0000 7fb45318f640 -1 reset not still connected to 0x56133c1909c0 2024-09-11T06:36:34.148 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:34.142+0000 7fb45318f640 -1 reset not still connected to 0x5613480e41a0 2024-09-11T06:36:34.148 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:34.142+0000 7fb45318f640 -1 reset not still connected to 0x56134d76fe10 2024-09-11T06:36:34.148 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:34.142+0000 7fb45318f640 -1 reset not still connected to 0x56134d81fd40 2024-09-11T06:36:34.148 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:34.142+0000 7fb45318f640 -1 reset not still connected to 0x561354d57040 2024-09-11T06:36:34.148 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:34.142+0000 7fb45318f640 -1 reset not still connected to 0x561354d77110 2024-09-11T06:36:34.148 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:34.142+0000 7fb45318f640 -1 reset not still connected to 0x561355040c30 2024-09-11T06:36:34.148 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:34.142+0000 7fb45318f640 -1 reset not still connected to 0x5613551fd110 2024-09-11T06:36:34.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:200 IOs, 70653952 bytes 2024-09-11T06:36:35.441 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ error_code=124 2024-09-11T06:36:35.441 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set -e 2024-09-11T06:36:35.442 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T06:36:35.442 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:36:35.442 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:36:35.442 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_38 2024-09-11T06:36:35.442 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_image_and_enable_mirror cluster2 mirror image_38 journal 128M 2024-09-11T06:36:35.442 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:36:35.442 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:36:35.442 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:36:35.442 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:36:35.442 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_38 2024-09-11T06:36:35.442 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:36:35.442 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local mode=journal 2024-09-11T06:36:35.442 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n journal ']' 2024-09-11T06:36:35.443 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:36:35.443 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_image cluster2 mirror image_38 128M 2024-09-11T06:36:35.443 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:36:35.443 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:36:35.443 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:36:35.443 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:36:35.443 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_38 2024-09-11T06:36:35.443 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:36:35.443 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local size=128 2024-09-11T06:36:35.443 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n 128M ']' 2024-09-11T06:36:35.443 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ size=128M 2024-09-11T06:36:35.443 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:36:35.443 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_38 2024-09-11T06:36:35.599 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T06:36:35.599 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:36:35.599 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' pool = image ']' 2024-09-11T06:36:35.599 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ case "${pool}" in 2024-09-11T06:36:35.600 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 1 2024-09-11T06:36:35.600 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n '' ']' 2024-09-11T06:36:35.600 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ stress_write_image cluster2 mirror image_38 2024-09-11T06:36:35.600 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:36:35.600 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:36:35.600 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_38 2024-09-11T06:36:35.600 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T06:36:35.604 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local duration=7 2024-09-11T06:36:35.604 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set +e 2024-09-11T06:36:35.605 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_38 --debug-rbd=20 --debug-journaler=20 2024-09-11T06:36:37.509 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:100 IOs, 18937856 bytes 2024-09-11T06:36:37.738 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:36:37.740+0000 7efd718c5640 -1 reset not still connected to 0x560b3a5f0750 2024-09-11T06:36:37.743 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:36:37.744+0000 7efd718c5640 -1 reset not still connected to 0x560b3b89c0d0 2024-09-11T06:36:37.755 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:36:37.760+0000 7efd718c5640 -1 reset not still connected to 0x560b41410000 2024-09-11T06:36:39.144 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:200 IOs, 37218304 bytes 2024-09-11T06:36:41.324 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:300 IOs, 66121728 bytes 2024-09-11T06:36:42.066 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:42.062+0000 7fb45318f640 -1 reset not still connected to 0x56131a1ffe10 2024-09-11T06:36:42.067 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:42.062+0000 7fb45318f640 -1 reset not still connected to 0x561322931860 2024-09-11T06:36:42.067 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:42.062+0000 7fb45318f640 -1 reset not still connected to 0x561322ded930 2024-09-11T06:36:42.067 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:42.062+0000 7fb45318f640 -1 reset not still connected to 0x561330ed0410 2024-09-11T06:36:42.067 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:42.062+0000 7fb45318f640 -1 reset not still connected to 0x56133c1909c0 2024-09-11T06:36:42.067 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:42.062+0000 7fb45318f640 -1 reset not still connected to 0x5613480e41a0 2024-09-11T06:36:42.070 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:42.062+0000 7fb45318f640 -1 reset not still connected to 0x56134d76fe10 2024-09-11T06:36:42.071 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:42.062+0000 7fb45318f640 -1 reset not still connected to 0x56134d81fd40 2024-09-11T06:36:42.071 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:42.062+0000 7fb45318f640 -1 reset not still connected to 0x561354d77110 2024-09-11T06:36:42.071 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:42.062+0000 7fb45318f640 -1 reset not still connected to 0x561355040c30 2024-09-11T06:36:42.614 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ error_code=124 2024-09-11T06:36:42.614 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set -e 2024-09-11T06:36:42.614 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T06:36:42.614 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:36:42.614 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:36:42.614 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_39 2024-09-11T06:36:42.614 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_image_and_enable_mirror cluster2 mirror image_39 journal 128M 2024-09-11T06:36:42.614 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:36:42.614 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:36:42.614 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:36:42.615 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:36:42.615 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_39 2024-09-11T06:36:42.615 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:36:42.615 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local mode=journal 2024-09-11T06:36:42.615 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n journal ']' 2024-09-11T06:36:42.615 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:36:42.615 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_image cluster2 mirror image_39 128M 2024-09-11T06:36:42.615 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:36:42.615 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:36:42.615 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:36:42.615 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:36:42.615 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_39 2024-09-11T06:36:42.615 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:36:42.615 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local size=128 2024-09-11T06:36:42.615 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n 128M ']' 2024-09-11T06:36:42.616 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ size=128M 2024-09-11T06:36:42.616 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:36:42.616 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_39 2024-09-11T06:36:42.804 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T06:36:42.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:36:42.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' pool = image ']' 2024-09-11T06:36:42.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ case "${pool}" in 2024-09-11T06:36:42.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 1 2024-09-11T06:36:42.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n '' ']' 2024-09-11T06:36:42.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ stress_write_image cluster2 mirror image_39 2024-09-11T06:36:42.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:36:42.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:36:42.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_39 2024-09-11T06:36:42.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T06:36:42.806 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local duration=14 2024-09-11T06:36:42.807 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set +e 2024-09-11T06:36:42.807 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_39 --debug-rbd=20 --debug-journaler=20 2024-09-11T06:36:46.217 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:36:46.202+0000 7fb2ca654640 -1 reset not still connected to 0x55ed0e363ee0 2024-09-11T06:36:46.218 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:36:46.202+0000 7fb2ca654640 -1 reset not still connected to 0x55ed0e47ba00 2024-09-11T06:36:46.218 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:36:46.202+0000 7fb2ca654640 -1 reset not still connected to 0x55ed11f509c0 2024-09-11T06:36:46.220 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:36:46.210+0000 7fb2ca654640 -1 reset not still connected to 0x55ed15c36d00 2024-09-11T06:36:46.220 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:36:46.214+0000 7fb2ca654640 -1 reset not still connected to 0x55ed169d91e0 2024-09-11T06:36:46.221 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:36:46.214+0000 7fb2ca654640 -1 reset not still connected to 0x55ed1b91f450 2024-09-11T06:36:46.221 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:36:46.214+0000 7fb2ca654640 -1 reset not still connected to 0x55ed288cd110 2024-09-11T06:36:46.221 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:36:46.214+0000 7fb2ca654640 -1 reset not still connected to 0x55ed32b6bba0 2024-09-11T06:36:46.222 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:36:46.214+0000 7fb2ca654640 -1 reset not still connected to 0x55ed369ccc30 2024-09-11T06:36:46.222 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:36:46.214+0000 7fb2ca654640 -1 reset not still connected to 0x55ed3a812820 2024-09-11T06:36:46.224 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:36:46.218+0000 7fb2ca654640 -1 reset not still connected to 0x55ed3ecf6b60 2024-09-11T06:36:46.224 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:36:46.218+0000 7fb2ca654640 -1 reset not still connected to 0x55ed43611450 2024-09-11T06:36:46.225 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:36:46.218+0000 7fb2ca654640 -1 reset not still connected to 0x55ed436c8750 2024-09-11T06:36:46.225 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:36:46.218+0000 7fb2ca654640 -1 reset not still connected to 0x55ed44536340 2024-09-11T06:36:46.225 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:36:46.218+0000 7fb2ca654640 -1 reset not still connected to 0x55ed4b336c30 2024-09-11T06:36:47.015 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:47.006+0000 7fb45318f640 -1 reset not still connected to 0x561315ba4270 2024-09-11T06:36:47.016 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:47.010+0000 7fb45318f640 -1 reset not still connected to 0x561315ba5520 2024-09-11T06:36:47.020 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:47.014+0000 7fb45318f640 -1 reset not still connected to 0x56131a1ffe10 2024-09-11T06:36:47.020 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:47.014+0000 7fb45318f640 -1 reset not still connected to 0x561322931860 2024-09-11T06:36:47.021 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:47.014+0000 7fb45318f640 -1 reset not still connected to 0x561322ded930 2024-09-11T06:36:47.021 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:47.014+0000 7fb45318f640 -1 reset not still connected to 0x561330ed0410 2024-09-11T06:36:47.021 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:47.014+0000 7fb45318f640 -1 reset not still connected to 0x56133339c410 2024-09-11T06:36:47.021 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:47.014+0000 7fb45318f640 -1 reset not still connected to 0x56133c1909c0 2024-09-11T06:36:47.021 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:47.014+0000 7fb45318f640 -1 reset not still connected to 0x5613480e41a0 2024-09-11T06:36:47.022 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:47.014+0000 7fb45318f640 -1 reset not still connected to 0x56134d76fe10 2024-09-11T06:36:47.022 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:47.014+0000 7fb45318f640 -1 reset not still connected to 0x56134d81fd40 2024-09-11T06:36:47.022 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:47.014+0000 7fb45318f640 -1 reset not still connected to 0x561354d77110 2024-09-11T06:36:47.022 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:47.014+0000 7fb45318f640 -1 reset not still connected to 0x561355040c30 2024-09-11T06:36:48.115 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:100 IOs, 69756928 bytes 2024-09-11T06:36:48.492 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:48.482+0000 7fb45318f640 -1 reset not still connected to 0x561355040c30 2024-09-11T06:36:48.668 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:36:48.662+0000 7fb2ca654640 -1 reset not still connected to 0x55ed0d758270 2024-09-11T06:36:48.669 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:36:48.662+0000 7fb2ca654640 -1 reset not still connected to 0x55ed0d759d40 2024-09-11T06:36:48.677 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:36:48.670+0000 7fb2ca654640 -1 reset not still connected to 0x55ed0d862410 2024-09-11T06:36:48.677 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:36:48.670+0000 7fb2ca654640 -1 reset not still connected to 0x55ed0d863040 2024-09-11T06:36:48.677 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:36:48.670+0000 7fb2ca654640 -1 reset not still connected to 0x55ed0d865e10 2024-09-11T06:36:48.677 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:36:48.670+0000 7fb2ca654640 -1 reset not still connected to 0x55ed0d8cd5f0 2024-09-11T06:36:48.677 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:36:48.670+0000 7fb2ca654640 -1 reset not still connected to 0x55ed12570820 2024-09-11T06:36:48.677 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:36:48.670+0000 7fb2ca654640 -1 reset not still connected to 0x55ed149fe8f0 2024-09-11T06:36:48.677 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:36:48.670+0000 7fb2ca654640 -1 reset not still connected to 0x55ed161c2000 2024-09-11T06:36:48.677 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:36:48.670+0000 7fb2ca654640 -1 reset not still connected to 0x55ed194eaea0 2024-09-11T06:36:48.677 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:36:48.670+0000 7fb2ca654640 -1 reset not still connected to 0x55ed1bd1add0 2024-09-11T06:36:48.677 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:36:48.670+0000 7fb2ca654640 -1 reset not still connected to 0x55ed2a1e0ea0 2024-09-11T06:36:48.677 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:36:48.670+0000 7fb2ca654640 -1 reset not still connected to 0x55ed2b00a000 2024-09-11T06:36:48.678 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:36:48.670+0000 7fb2ca654640 -1 reset not still connected to 0x55ed38d48270 2024-09-11T06:36:48.678 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:36:48.670+0000 7fb2ca654640 -1 reset not still connected to 0x55ed3d399d40 2024-09-11T06:36:48.678 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:36:48.670+0000 7fb2ca654640 -1 reset not still connected to 0x55ed43824c30 2024-09-11T06:36:49.407 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:36:49.402+0000 7fb2ca654640 -1 reset not still connected to 0x55ed0d8655f0 2024-09-11T06:36:49.407 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:36:49.402+0000 7fb2ca654640 -1 reset not still connected to 0x55ed0d8cdee0 2024-09-11T06:36:49.408 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:36:49.402+0000 7fb2ca654640 -1 reset not still connected to 0x55ed0e363ee0 2024-09-11T06:36:49.408 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:36:49.402+0000 7fb2ca654640 -1 reset not still connected to 0x55ed0e47ba00 2024-09-11T06:36:49.408 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:36:49.402+0000 7fb2ca654640 -1 reset not still connected to 0x55ed11f509c0 2024-09-11T06:36:49.408 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:36:49.402+0000 7fb2ca654640 -1 reset not still connected to 0x55ed128edba0 2024-09-11T06:36:49.408 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:36:49.402+0000 7fb2ca654640 -1 reset not still connected to 0x55ed15c36d00 2024-09-11T06:36:49.409 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:36:49.406+0000 7fb2ca654640 -1 reset not still connected to 0x55ed169d91e0 2024-09-11T06:36:49.409 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:36:49.406+0000 7fb2ca654640 -1 reset not still connected to 0x55ed1b91f450 2024-09-11T06:36:49.409 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:36:49.406+0000 7fb2ca654640 -1 reset not still connected to 0x55ed1c2d45b0 2024-09-11T06:36:49.409 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:36:49.406+0000 7fb2ca654640 -1 reset not still connected to 0x55ed288cd110 2024-09-11T06:36:49.409 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:36:49.406+0000 7fb2ca654640 -1 reset not still connected to 0x55ed2b074410 2024-09-11T06:36:49.409 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:36:49.406+0000 7fb2ca654640 -1 reset not still connected to 0x55ed32b6bba0 2024-09-11T06:36:49.409 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:36:49.406+0000 7fb2ca654640 -1 reset not still connected to 0x55ed369ccc30 2024-09-11T06:36:49.409 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:36:49.406+0000 7fb2ca654640 -1 reset not still connected to 0x55ed3a812820 2024-09-11T06:36:49.412 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:36:49.406+0000 7fb2ca654640 -1 reset not still connected to 0x55ed43611450 2024-09-11T06:36:49.413 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:36:49.410+0000 7fb2ca654640 -1 reset not still connected to 0x55ed436c8750 2024-09-11T06:36:49.414 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:36:49.410+0000 7fb2ca654640 -1 reset not still connected to 0x55ed44536340 2024-09-11T06:36:49.414 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:36:49.410+0000 7fb2ca654640 -1 reset not still connected to 0x55ed4b336c30 2024-09-11T06:36:51.242 INFO:tasks.ceph.cluster2.osd.0.smithi112.stderr:2024-09-11T06:36:51.243+0000 7f3fa0017640 -1 reset not still connected to 0x563d394c8ea0 2024-09-11T06:36:51.242 INFO:tasks.ceph.cluster2.osd.0.smithi112.stderr:2024-09-11T06:36:51.243+0000 7f3fa0017640 -1 reset not still connected to 0x563d43515c70 2024-09-11T06:36:51.242 INFO:tasks.ceph.cluster2.osd.0.smithi112.stderr:2024-09-11T06:36:51.243+0000 7f3fa0017640 -1 reset not still connected to 0x563d4619c8f0 2024-09-11T06:36:54.729 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:54.726+0000 7fb45318f640 -1 reset not still connected to 0x561337a04410 2024-09-11T06:36:54.729 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:54.726+0000 7fb45318f640 -1 reset not still connected to 0x561337b67e10 2024-09-11T06:36:54.729 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:54.726+0000 7fb45318f640 -1 reset not still connected to 0x56133cabc000 2024-09-11T06:36:54.729 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:54.726+0000 7fb45318f640 -1 reset not still connected to 0x561342690270 2024-09-11T06:36:54.731 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:54.726+0000 7fb45318f640 -1 reset not still connected to 0x5613480c0a90 2024-09-11T06:36:54.731 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:54.726+0000 7fb45318f640 -1 reset not still connected to 0x5613480c1040 2024-09-11T06:36:54.731 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:54.726+0000 7fb45318f640 -1 reset not still connected to 0x5613545c3790 2024-09-11T06:36:54.967 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:54.962+0000 7fb45318f640 -1 reset not still connected to 0x561315ba5520 2024-09-11T06:36:54.973 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:54.966+0000 7fb45318f640 -1 reset not still connected to 0x56131a1ffe10 2024-09-11T06:36:54.973 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:54.966+0000 7fb45318f640 -1 reset not still connected to 0x561322931860 2024-09-11T06:36:54.973 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:54.966+0000 7fb45318f640 -1 reset not still connected to 0x561322ded930 2024-09-11T06:36:54.973 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:54.966+0000 7fb45318f640 -1 reset not still connected to 0x561330ed0410 2024-09-11T06:36:54.973 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:54.966+0000 7fb45318f640 -1 reset not still connected to 0x56133339c410 2024-09-11T06:36:54.973 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:54.966+0000 7fb45318f640 -1 reset not still connected to 0x56133c1909c0 2024-09-11T06:36:54.973 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:54.966+0000 7fb45318f640 -1 reset not still connected to 0x5613480e41a0 2024-09-11T06:36:54.974 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:54.966+0000 7fb45318f640 -1 reset not still connected to 0x56134d76fe10 2024-09-11T06:36:54.974 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:54.966+0000 7fb45318f640 -1 reset not still connected to 0x56134d81fd40 2024-09-11T06:36:54.974 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:54.966+0000 7fb45318f640 -1 reset not still connected to 0x561354d77110 2024-09-11T06:36:54.974 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:54.966+0000 7fb45318f640 -1 reset not still connected to 0x561355040c30 2024-09-11T06:36:54.974 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:36:54.970+0000 7fb45318f640 -1 reset not still connected to 0x5613551fd110 2024-09-11T06:36:55.067 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:200 IOs, 129283072 bytes 2024-09-11T06:36:56.675 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:300 IOs, 154408960 bytes 2024-09-11T06:36:56.816 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ error_code=124 2024-09-11T06:36:56.817 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set -e 2024-09-11T06:36:56.817 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T06:36:56.817 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:36:56.817 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:36:56.817 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_40 2024-09-11T06:36:56.817 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_image_and_enable_mirror cluster2 mirror image_40 journal 128M 2024-09-11T06:36:56.817 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:36:56.817 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:36:56.817 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:36:56.817 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:36:56.818 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_40 2024-09-11T06:36:56.818 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:36:56.818 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local mode=journal 2024-09-11T06:36:56.818 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n journal ']' 2024-09-11T06:36:56.818 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:36:56.818 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_image cluster2 mirror image_40 128M 2024-09-11T06:36:56.818 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:36:56.818 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:36:56.818 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:36:56.818 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:36:56.818 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_40 2024-09-11T06:36:56.818 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:36:56.818 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local size=128 2024-09-11T06:36:56.819 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n 128M ']' 2024-09-11T06:36:56.819 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ size=128M 2024-09-11T06:36:56.819 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:36:56.819 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_40 2024-09-11T06:36:57.026 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:36:57.022+0000 7fc41084f640 -1 reset not still connected to 0x55eb2613e680 2024-09-11T06:36:57.029 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:36:57.022+0000 7fc41084f640 -1 reset not still connected to 0x55eb2da77ba0 2024-09-11T06:36:57.029 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:36:57.022+0000 7fc41084f640 -1 reset not still connected to 0x55eb35243040 2024-09-11T06:36:57.029 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:36:57.022+0000 7fc41084f640 -1 reset not still connected to 0x55eb354a21a0 2024-09-11T06:36:57.031 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:36:57.026+0000 7fc41084f640 -1 reset not still connected to 0x55eb3893ac30 2024-09-11T06:36:57.031 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:36:57.026+0000 7fc41084f640 -1 reset not still connected to 0x55eb4480b930 2024-09-11T06:36:57.031 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:36:57.026+0000 7fc41084f640 -1 reset not still connected to 0x55eb4f1329c0 2024-09-11T06:36:57.031 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:36:57.026+0000 7fc41084f640 -1 reset not still connected to 0x55eb53d56f70 2024-09-11T06:36:57.162 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T06:36:57.162 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:36:57.162 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' pool = image ']' 2024-09-11T06:36:57.162 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ case "${pool}" in 2024-09-11T06:36:57.162 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 1 2024-09-11T06:36:57.162 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n '' ']' 2024-09-11T06:36:57.162 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ stress_write_image cluster2 mirror image_40 2024-09-11T06:36:57.162 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:36:57.162 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:36:57.162 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_40 2024-09-11T06:36:57.162 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T06:36:57.163 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local duration=7 2024-09-11T06:36:57.164 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set +e 2024-09-11T06:36:57.164 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_40 --debug-rbd=20 --debug-journaler=20 2024-09-11T06:37:00.092 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:00.086+0000 7fb2ca654640 -1 reset not still connected to 0x55ed0d8655f0 2024-09-11T06:37:00.092 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:00.086+0000 7fb2ca654640 -1 reset not still connected to 0x55ed0d8cdee0 2024-09-11T06:37:00.092 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:00.086+0000 7fb2ca654640 -1 reset not still connected to 0x55ed0e363ee0 2024-09-11T06:37:00.095 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:00.090+0000 7fb2ca654640 -1 reset not still connected to 0x55ed0e47ba00 2024-09-11T06:37:00.095 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:00.090+0000 7fb2ca654640 -1 reset not still connected to 0x55ed0fd8cf70 2024-09-11T06:37:00.095 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:00.090+0000 7fb2ca654640 -1 reset not still connected to 0x55ed11f509c0 2024-09-11T06:37:00.095 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:00.090+0000 7fb2ca654640 -1 reset not still connected to 0x55ed128edba0 2024-09-11T06:37:00.095 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:00.090+0000 7fb2ca654640 -1 reset not still connected to 0x55ed15c15ad0 2024-09-11T06:37:00.095 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:00.090+0000 7fb2ca654640 -1 reset not still connected to 0x55ed15c36d00 2024-09-11T06:37:00.095 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:00.090+0000 7fb2ca654640 -1 reset not still connected to 0x55ed169d91e0 2024-09-11T06:37:00.095 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:00.090+0000 7fb2ca654640 -1 reset not still connected to 0x55ed1b91f450 2024-09-11T06:37:00.105 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:00.098+0000 7fb2ca654640 -1 reset not still connected to 0x55ed1c2d45b0 2024-09-11T06:37:00.105 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:00.098+0000 7fb2ca654640 -1 reset not still connected to 0x55ed288cd110 2024-09-11T06:37:00.105 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:00.098+0000 7fb2ca654640 -1 reset not still connected to 0x55ed2b074410 2024-09-11T06:37:00.105 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:00.098+0000 7fb2ca654640 -1 reset not still connected to 0x55ed32b6bba0 2024-09-11T06:37:00.106 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:00.098+0000 7fb2ca654640 -1 reset not still connected to 0x55ed369ccc30 2024-09-11T06:37:00.106 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:00.098+0000 7fb2ca654640 -1 reset not still connected to 0x55ed3a812820 2024-09-11T06:37:00.106 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:00.098+0000 7fb2ca654640 -1 reset not still connected to 0x55ed3ecf6b60 2024-09-11T06:37:00.106 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:00.098+0000 7fb2ca654640 -1 reset not still connected to 0x55ed3ecf76c0 2024-09-11T06:37:00.106 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:00.098+0000 7fb2ca654640 -1 reset not still connected to 0x55ed43611450 2024-09-11T06:37:00.106 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:00.098+0000 7fb2ca654640 -1 reset not still connected to 0x55ed436c8750 2024-09-11T06:37:00.106 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:00.098+0000 7fb2ca654640 -1 reset not still connected to 0x55ed44536340 2024-09-11T06:37:00.106 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:00.098+0000 7fb2ca654640 -1 reset not still connected to 0x55ed4b336c30 2024-09-11T06:37:02.462 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:02.458+0000 7fb2ca654640 -1 reset not still connected to 0x55ed0e363ee0 2024-09-11T06:37:02.465 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:02.462+0000 7fb2ca654640 -1 reset not still connected to 0x55ed0e47ba00 2024-09-11T06:37:02.465 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:02.462+0000 7fb2ca654640 -1 reset not still connected to 0x55ed1b91f450 2024-09-11T06:37:02.465 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:02.462+0000 7fb2ca654640 -1 reset not still connected to 0x55ed4b336c30 2024-09-11T06:37:02.634 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:100 IOs, 57717760 bytes 2024-09-11T06:37:04.162 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:37:04.158+0000 7fb45318f640 -1 reset not still connected to 0x561355040c30 2024-09-11T06:37:04.177 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ error_code=124 2024-09-11T06:37:04.177 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set -e 2024-09-11T06:37:04.177 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T06:37:04.177 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:37:04.177 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:37:04.177 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_41 2024-09-11T06:37:04.178 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_image_and_enable_mirror cluster2 mirror image_41 journal 128M 2024-09-11T06:37:04.178 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:37:04.178 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:37:04.178 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:37:04.178 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:37:04.178 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_41 2024-09-11T06:37:04.178 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:37:04.178 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local mode=journal 2024-09-11T06:37:04.178 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n journal ']' 2024-09-11T06:37:04.178 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:37:04.178 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_image cluster2 mirror image_41 128M 2024-09-11T06:37:04.178 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:37:04.178 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:37:04.178 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:37:04.179 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:37:04.179 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_41 2024-09-11T06:37:04.179 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:37:04.179 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local size=128 2024-09-11T06:37:04.179 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n 128M ']' 2024-09-11T06:37:04.179 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ size=128M 2024-09-11T06:37:04.179 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:37:04.179 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_41 2024-09-11T06:37:04.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T06:37:04.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:37:04.343 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' pool = image ']' 2024-09-11T06:37:04.343 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ case "${pool}" in 2024-09-11T06:37:04.343 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 1 2024-09-11T06:37:04.343 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n '' ']' 2024-09-11T06:37:04.343 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ stress_write_image cluster2 mirror image_41 2024-09-11T06:37:04.343 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:37:04.343 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:37:04.343 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_41 2024-09-11T06:37:04.343 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T06:37:04.344 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local duration=7 2024-09-11T06:37:04.344 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set +e 2024-09-11T06:37:04.344 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_41 --debug-rbd=20 --debug-journaler=20 2024-09-11T06:37:04.452 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:37:04.455+0000 7efd718c5640 -1 reset not still connected to 0x560b28ee1ee0 2024-09-11T06:37:04.453 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:37:04.455+0000 7efd718c5640 -1 reset not still connected to 0x560b29059520 2024-09-11T06:37:04.462 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:37:04.467+0000 7efd718c5640 -1 reset not still connected to 0x560b2fc524e0 2024-09-11T06:37:04.465 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:37:04.467+0000 7efd718c5640 -1 reset not still connected to 0x560b37798410 2024-09-11T06:37:04.476 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:37:04.479+0000 7efd718c5640 -1 reset not still connected to 0x560b3c64e9c0 2024-09-11T06:37:08.030 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:37:08.022+0000 7fb45318f640 -1 reset not still connected to 0x5613480c0a90 2024-09-11T06:37:08.031 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:37:08.022+0000 7fb45318f640 -1 reset not still connected to 0x5613480c1040 2024-09-11T06:37:08.031 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:37:08.022+0000 7fb45318f640 -1 reset not still connected to 0x5613551aac30 2024-09-11T06:37:09.234 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:09.230+0000 7fb2ca654640 -1 reset not still connected to 0x55ed2a1e0ea0 2024-09-11T06:37:09.237 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:09.234+0000 7fb2ca654640 -1 reset not still connected to 0x55ed3d399d40 2024-09-11T06:37:09.242 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:09.238+0000 7fb2ca654640 -1 reset not still connected to 0x55ed0d758270 2024-09-11T06:37:09.243 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:09.238+0000 7fb2ca654640 -1 reset not still connected to 0x55ed0d759d40 2024-09-11T06:37:09.250 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:09.242+0000 7fb2ca654640 -1 reset not still connected to 0x55ed0d862410 2024-09-11T06:37:09.251 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:09.242+0000 7fb2ca654640 -1 reset not still connected to 0x55ed0d863040 2024-09-11T06:37:09.251 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:09.242+0000 7fb2ca654640 -1 reset not still connected to 0x55ed0d865e10 2024-09-11T06:37:09.251 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:09.242+0000 7fb2ca654640 -1 reset not still connected to 0x55ed0d8cd5f0 2024-09-11T06:37:09.251 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:09.242+0000 7fb2ca654640 -1 reset not still connected to 0x55ed0e550dd0 2024-09-11T06:37:09.251 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:09.242+0000 7fb2ca654640 -1 reset not still connected to 0x55ed12570820 2024-09-11T06:37:09.251 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:09.242+0000 7fb2ca654640 -1 reset not still connected to 0x55ed149fe8f0 2024-09-11T06:37:09.251 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:09.242+0000 7fb2ca654640 -1 reset not still connected to 0x55ed161c2000 2024-09-11T06:37:09.251 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:09.242+0000 7fb2ca654640 -1 reset not still connected to 0x55ed194eaea0 2024-09-11T06:37:09.251 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:09.242+0000 7fb2ca654640 -1 reset not still connected to 0x55ed2a1e0ea0 2024-09-11T06:37:09.251 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:09.242+0000 7fb2ca654640 -1 reset not still connected to 0x55ed3d399d40 2024-09-11T06:37:09.872 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:37:09.875+0000 7efd718c5640 -1 reset not still connected to 0x560b14da5d40 2024-09-11T06:37:09.873 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:37:09.875+0000 7efd718c5640 -1 reset not still connected to 0x560b159635f0 2024-09-11T06:37:09.873 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:37:09.875+0000 7efd718c5640 -1 reset not still connected to 0x560b15a70dd0 2024-09-11T06:37:09.873 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:37:09.875+0000 7efd718c5640 -1 reset not still connected to 0x560b15a71520 2024-09-11T06:37:09.970 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:09.966+0000 7fb2ca654640 -1 reset not still connected to 0x55ed2a1e0ea0 2024-09-11T06:37:09.971 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:09.966+0000 7fb2ca654640 -1 reset not still connected to 0x55ed3d399d40 2024-09-11T06:37:10.196 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:100 IOs, 86881280 bytes 2024-09-11T06:37:10.888 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:37:10.882+0000 7fc41084f640 -1 reset not still connected to 0x55eb3add8270 2024-09-11T06:37:10.889 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:37:10.882+0000 7fc41084f640 -1 reset not still connected to 0x55eb4480b930 2024-09-11T06:37:10.889 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:37:10.886+0000 7fc41084f640 -1 reset not still connected to 0x55eb4f1329c0 2024-09-11T06:37:10.889 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:37:10.886+0000 7fc41084f640 -1 reset not still connected to 0x55eb53d56f70 2024-09-11T06:37:11.354 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ error_code=124 2024-09-11T06:37:11.354 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set -e 2024-09-11T06:37:11.354 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T06:37:11.354 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:37:11.354 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:37:11.354 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_42 2024-09-11T06:37:11.354 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_image_and_enable_mirror cluster2 mirror image_42 journal 128M 2024-09-11T06:37:11.354 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:37:11.354 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:37:11.354 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:37:11.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:37:11.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_42 2024-09-11T06:37:11.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:37:11.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local mode=journal 2024-09-11T06:37:11.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n journal ']' 2024-09-11T06:37:11.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:37:11.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_image cluster2 mirror image_42 128M 2024-09-11T06:37:11.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:37:11.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:37:11.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:37:11.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:37:11.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_42 2024-09-11T06:37:11.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:37:11.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local size=128 2024-09-11T06:37:11.356 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n 128M ']' 2024-09-11T06:37:11.356 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ size=128M 2024-09-11T06:37:11.356 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:37:11.356 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_42 2024-09-11T06:37:11.909 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T06:37:11.909 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:37:11.909 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' pool = image ']' 2024-09-11T06:37:11.909 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ case "${pool}" in 2024-09-11T06:37:11.909 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 1 2024-09-11T06:37:11.909 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n '' ']' 2024-09-11T06:37:11.909 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ stress_write_image cluster2 mirror image_42 2024-09-11T06:37:11.909 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:37:11.909 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:37:11.909 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_42 2024-09-11T06:37:11.910 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T06:37:11.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local duration=14 2024-09-11T06:37:11.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set +e 2024-09-11T06:37:11.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_42 --debug-rbd=20 --debug-journaler=20 2024-09-11T06:37:12.179 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:37:12.183+0000 7efd718c5640 -1 reset not still connected to 0x560b1899a410 2024-09-11T06:37:12.180 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:37:12.183+0000 7efd718c5640 -1 reset not still connected to 0x560b28ee1ee0 2024-09-11T06:37:12.180 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:37:12.183+0000 7efd718c5640 -1 reset not still connected to 0x560b29059520 2024-09-11T06:37:12.180 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:37:12.183+0000 7efd718c5640 -1 reset not still connected to 0x560b2f4745b0 2024-09-11T06:37:12.180 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:37:12.183+0000 7efd718c5640 -1 reset not still connected to 0x560b2fc524e0 2024-09-11T06:37:12.180 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:37:12.183+0000 7efd718c5640 -1 reset not still connected to 0x560b37798410 2024-09-11T06:37:12.180 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:37:12.183+0000 7efd718c5640 -1 reset not still connected to 0x560b3c64e9c0 2024-09-11T06:37:12.866 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:12.858+0000 7fb2ca654640 -1 reset not still connected to 0x55ed1b91f450 2024-09-11T06:37:12.869 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:12.866+0000 7fb2ca654640 -1 reset not still connected to 0x55ed288cd110 2024-09-11T06:37:12.875 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:12.866+0000 7fb2ca654640 -1 reset not still connected to 0x55ed32b6bba0 2024-09-11T06:37:12.875 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:12.866+0000 7fb2ca654640 -1 reset not still connected to 0x55ed369ccc30 2024-09-11T06:37:12.875 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:12.866+0000 7fb2ca654640 -1 reset not still connected to 0x55ed3a812820 2024-09-11T06:37:12.875 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:12.866+0000 7fb2ca654640 -1 reset not still connected to 0x55ed43611450 2024-09-11T06:37:12.875 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:12.866+0000 7fb2ca654640 -1 reset not still connected to 0x55ed44536340 2024-09-11T06:37:12.875 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:12.866+0000 7fb2ca654640 -1 reset not still connected to 0x55ed4b336c30 2024-09-11T06:37:12.875 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:12.866+0000 7fb2ca654640 -1 reset not still connected to 0x55ed4b37ca90 2024-09-11T06:37:15.342 INFO:tasks.ceph.cluster2.osd.0.smithi112.stderr:2024-09-11T06:37:15.343+0000 7f3fa0017640 -1 reset not still connected to 0x563d35fe6750 2024-09-11T06:37:15.342 INFO:tasks.ceph.cluster2.osd.0.smithi112.stderr:2024-09-11T06:37:15.343+0000 7f3fa0017640 -1 reset not still connected to 0x563d4633c1a0 2024-09-11T06:37:15.683 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:100 IOs, 64653312 bytes 2024-09-11T06:37:15.794 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:37:15.786+0000 7fb45318f640 -1 reset not still connected to 0x561337b67e10 2024-09-11T06:37:15.794 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:37:15.786+0000 7fb45318f640 -1 reset not still connected to 0x56133cabc000 2024-09-11T06:37:15.794 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:37:15.786+0000 7fb45318f640 -1 reset not still connected to 0x561342690270 2024-09-11T06:37:15.794 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:37:15.786+0000 7fb45318f640 -1 reset not still connected to 0x5613480c0a90 2024-09-11T06:37:15.794 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:37:15.790+0000 7fb45318f640 -1 reset not still connected to 0x5613480c1040 2024-09-11T06:37:15.794 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:37:15.790+0000 7fb45318f640 -1 reset not still connected to 0x5613551aac30 2024-09-11T06:37:16.623 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:37:16.623+0000 7efd718c5640 -1 reset not still connected to 0x560b1899a410 2024-09-11T06:37:16.623 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:37:16.623+0000 7efd718c5640 -1 reset not still connected to 0x560b28ee1ee0 2024-09-11T06:37:16.623 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:37:16.623+0000 7efd718c5640 -1 reset not still connected to 0x560b29059520 2024-09-11T06:37:16.623 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:37:16.623+0000 7efd718c5640 -1 reset not still connected to 0x560b2f4745b0 2024-09-11T06:37:16.623 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:37:16.623+0000 7efd718c5640 -1 reset not still connected to 0x560b37798410 2024-09-11T06:37:16.624 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:37:16.623+0000 7efd718c5640 -1 reset not still connected to 0x560b3c64e9c0 2024-09-11T06:37:17.391 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:37:17.385+0000 7fb45318f640 -1 reset not still connected to 0x561315ba4270 2024-09-11T06:37:17.391 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:37:17.385+0000 7fb45318f640 -1 reset not still connected to 0x561315ba5520 2024-09-11T06:37:17.397 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:37:17.389+0000 7fb45318f640 -1 reset not still connected to 0x56131a1ffe10 2024-09-11T06:37:17.397 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:37:17.389+0000 7fb45318f640 -1 reset not still connected to 0x561322931860 2024-09-11T06:37:17.402 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:37:17.389+0000 7fb45318f640 -1 reset not still connected to 0x561322ded930 2024-09-11T06:37:17.402 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:37:17.389+0000 7fb45318f640 -1 reset not still connected to 0x561330ed0410 2024-09-11T06:37:17.402 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:37:17.389+0000 7fb45318f640 -1 reset not still connected to 0x56133339c410 2024-09-11T06:37:17.402 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:37:17.389+0000 7fb45318f640 -1 reset not still connected to 0x56133c1909c0 2024-09-11T06:37:17.403 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:37:17.389+0000 7fb45318f640 -1 reset not still connected to 0x5613480e41a0 2024-09-11T06:37:17.403 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:37:17.389+0000 7fb45318f640 -1 reset not still connected to 0x56134d76fe10 2024-09-11T06:37:17.403 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:37:17.389+0000 7fb45318f640 -1 reset not still connected to 0x56134d81fd40 2024-09-11T06:37:17.403 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:37:17.389+0000 7fb45318f640 -1 reset not still connected to 0x561354d77110 2024-09-11T06:37:17.403 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:37:17.389+0000 7fb45318f640 -1 reset not still connected to 0x561355040c30 2024-09-11T06:37:17.403 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:37:17.393+0000 7fb45318f640 -1 reset not still connected to 0x5613551fd110 2024-09-11T06:37:21.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:200 IOs, 115342336 bytes 2024-09-11T06:37:21.539 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:21.533+0000 7fb2ca654640 -1 reset not still connected to 0x55ed0d759d40 2024-09-11T06:37:21.539 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:21.533+0000 7fb2ca654640 -1 reset not still connected to 0x55ed0d862410 2024-09-11T06:37:21.540 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:21.533+0000 7fb2ca654640 -1 reset not still connected to 0x55ed0d863040 2024-09-11T06:37:21.540 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:21.533+0000 7fb2ca654640 -1 reset not still connected to 0x55ed0d865e10 2024-09-11T06:37:21.540 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:21.533+0000 7fb2ca654640 -1 reset not still connected to 0x55ed0e550dd0 2024-09-11T06:37:21.543 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:21.537+0000 7fb2ca654640 -1 reset not still connected to 0x55ed12570820 2024-09-11T06:37:21.543 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:21.537+0000 7fb2ca654640 -1 reset not still connected to 0x55ed149fe8f0 2024-09-11T06:37:21.543 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:21.537+0000 7fb2ca654640 -1 reset not still connected to 0x55ed161c2000 2024-09-11T06:37:21.543 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:21.537+0000 7fb2ca654640 -1 reset not still connected to 0x55ed194eaea0 2024-09-11T06:37:21.543 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:21.537+0000 7fb2ca654640 -1 reset not still connected to 0x55ed1bd1add0 2024-09-11T06:37:21.543 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:21.537+0000 7fb2ca654640 -1 reset not still connected to 0x55ed2a1e0ea0 2024-09-11T06:37:21.543 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:21.537+0000 7fb2ca654640 -1 reset not still connected to 0x55ed2b00a000 2024-09-11T06:37:21.543 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:21.537+0000 7fb2ca654640 -1 reset not still connected to 0x55ed38d48270 2024-09-11T06:37:21.544 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:21.537+0000 7fb2ca654640 -1 reset not still connected to 0x55ed3d399d40 2024-09-11T06:37:21.544 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:21.537+0000 7fb2ca654640 -1 reset not still connected to 0x55ed43824c30 2024-09-11T06:37:22.870 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:37:22.865+0000 7fb45318f640 -1 reset not still connected to 0x5613191b3a00 2024-09-11T06:37:22.871 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:37:22.865+0000 7fb45318f640 -1 reset not still connected to 0x561322165790 2024-09-11T06:37:22.871 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:37:22.865+0000 7fb45318f640 -1 reset not still connected to 0x5613224a2d00 2024-09-11T06:37:22.874 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:37:22.865+0000 7fb45318f640 -1 reset not still connected to 0x5613225fd930 2024-09-11T06:37:22.874 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:37:22.865+0000 7fb45318f640 -1 reset not still connected to 0x561322bdf450 2024-09-11T06:37:22.874 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:37:22.865+0000 7fb45318f640 -1 reset not still connected to 0x5613273cfd40 2024-09-11T06:37:22.874 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:37:22.865+0000 7fb45318f640 -1 reset not still connected to 0x561328b5c680 2024-09-11T06:37:22.874 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:37:22.865+0000 7fb45318f640 -1 reset not still connected to 0x561328b5c8f0 2024-09-11T06:37:22.874 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:37:22.865+0000 7fb45318f640 -1 reset not still connected to 0x56132e90e340 2024-09-11T06:37:22.874 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:37:22.865+0000 7fb45318f640 -1 reset not still connected to 0x561337a04410 2024-09-11T06:37:22.874 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:37:22.865+0000 7fb45318f640 -1 reset not still connected to 0x561337b67e10 2024-09-11T06:37:22.874 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:37:22.865+0000 7fb45318f640 -1 reset not still connected to 0x56133cabc000 2024-09-11T06:37:22.874 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:37:22.865+0000 7fb45318f640 -1 reset not still connected to 0x561342690270 2024-09-11T06:37:22.874 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:37:22.865+0000 7fb45318f640 -1 reset not still connected to 0x5613480c0a90 2024-09-11T06:37:22.874 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:37:22.865+0000 7fb45318f640 -1 reset not still connected to 0x5613480c1040 2024-09-11T06:37:22.874 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:37:22.865+0000 7fb45318f640 -1 reset not still connected to 0x5613545c3790 2024-09-11T06:37:22.875 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:37:22.865+0000 7fb45318f640 -1 reset not still connected to 0x5613551aac30 2024-09-11T06:37:23.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:300 IOs, 149297152 bytes 2024-09-11T06:37:24.394 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:24.381+0000 7fb2ca654640 -1 reset not still connected to 0x55ed0e47ba00 2024-09-11T06:37:24.397 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:24.389+0000 7fb2ca654640 -1 reset not still connected to 0x55ed15c15ad0 2024-09-11T06:37:24.405 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:24.397+0000 7fb2ca654640 -1 reset not still connected to 0x55ed15c36d00 2024-09-11T06:37:24.405 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:24.397+0000 7fb2ca654640 -1 reset not still connected to 0x55ed169d91e0 2024-09-11T06:37:24.405 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:24.397+0000 7fb2ca654640 -1 reset not still connected to 0x55ed1b91f450 2024-09-11T06:37:24.406 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:24.397+0000 7fb2ca654640 -1 reset not still connected to 0x55ed288cd110 2024-09-11T06:37:24.406 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:24.397+0000 7fb2ca654640 -1 reset not still connected to 0x55ed32b6bba0 2024-09-11T06:37:24.406 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:24.397+0000 7fb2ca654640 -1 reset not still connected to 0x55ed369ccc30 2024-09-11T06:37:24.406 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:24.401+0000 7fb2ca654640 -1 reset not still connected to 0x55ed3a812820 2024-09-11T06:37:24.406 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:24.401+0000 7fb2ca654640 -1 reset not still connected to 0x55ed3ecf6b60 2024-09-11T06:37:24.406 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:24.401+0000 7fb2ca654640 -1 reset not still connected to 0x55ed3ecf76c0 2024-09-11T06:37:24.406 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:24.401+0000 7fb2ca654640 -1 reset not still connected to 0x55ed43611450 2024-09-11T06:37:24.406 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:24.401+0000 7fb2ca654640 -1 reset not still connected to 0x55ed436c8750 2024-09-11T06:37:24.406 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:24.401+0000 7fb2ca654640 -1 reset not still connected to 0x55ed44536340 2024-09-11T06:37:24.406 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:24.401+0000 7fb2ca654640 -1 reset not still connected to 0x55ed4b336c30 2024-09-11T06:37:24.407 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:24.401+0000 7fb2ca654640 -1 reset not still connected to 0x55ed4b37ca90 2024-09-11T06:37:24.959 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:400 IOs, 158354432 bytes 2024-09-11T06:37:25.600 INFO:tasks.ceph.cluster2.osd.0.smithi112.stderr:2024-09-11T06:37:25.603+0000 7f3f913c4640 -1 bdev(0x563d17e7cc00 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 2 2024-09-11T06:37:25.608 INFO:tasks.ceph.cluster2.osd.0.smithi112.stderr:2024-09-11T06:37:25.611+0000 7f3f8d3bc640 -1 bdev(0x563d17e7cc00 /var/lib/ceph/osd/cluster2-0/block) aio_submit retries 4 2024-09-11T06:37:25.626 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:37:25.613+0000 7fb45318f640 -1 reset not still connected to 0x56134d81fd40 2024-09-11T06:37:25.629 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:37:25.613+0000 7fb45318f640 -1 reset not still connected to 0x561355040c30 2024-09-11T06:37:25.925 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ error_code=124 2024-09-11T06:37:25.926 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set -e 2024-09-11T06:37:25.926 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T06:37:25.926 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:37:25.926 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:37:25.926 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_43 2024-09-11T06:37:25.926 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_image_and_enable_mirror cluster2 mirror image_43 journal 128M 2024-09-11T06:37:25.926 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:37:25.926 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:37:25.926 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:37:25.926 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:37:25.926 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_43 2024-09-11T06:37:25.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:37:25.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local mode=journal 2024-09-11T06:37:25.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n journal ']' 2024-09-11T06:37:25.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:37:25.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_image cluster2 mirror image_43 128M 2024-09-11T06:37:25.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:37:25.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:37:25.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:37:25.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:37:25.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_43 2024-09-11T06:37:25.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:37:25.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local size=128 2024-09-11T06:37:25.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n 128M ']' 2024-09-11T06:37:25.928 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ size=128M 2024-09-11T06:37:25.928 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:37:25.928 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_43 2024-09-11T06:37:26.095 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T06:37:26.095 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:37:26.095 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' pool = image ']' 2024-09-11T06:37:26.095 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ case "${pool}" in 2024-09-11T06:37:26.096 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 1 2024-09-11T06:37:26.096 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n '' ']' 2024-09-11T06:37:26.096 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ stress_write_image cluster2 mirror image_43 2024-09-11T06:37:26.096 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:37:26.096 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:37:26.096 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_43 2024-09-11T06:37:26.096 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T06:37:26.097 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local duration=14 2024-09-11T06:37:26.097 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set +e 2024-09-11T06:37:26.098 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_43 --debug-rbd=20 --debug-journaler=20 2024-09-11T06:37:27.157 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:37:27.149+0000 7fb45318f640 -1 reset not still connected to 0x561315ba4270 2024-09-11T06:37:27.157 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:37:27.149+0000 7fb45318f640 -1 reset not still connected to 0x561315ba5520 2024-09-11T06:37:27.158 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:37:27.149+0000 7fb45318f640 -1 reset not still connected to 0x56131a1ffe10 2024-09-11T06:37:27.158 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:37:27.149+0000 7fb45318f640 -1 reset not still connected to 0x561322931860 2024-09-11T06:37:27.158 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:37:27.149+0000 7fb45318f640 -1 reset not still connected to 0x561322ded930 2024-09-11T06:37:27.159 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:37:27.149+0000 7fb45318f640 -1 reset not still connected to 0x561330ed0410 2024-09-11T06:37:27.159 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:37:27.149+0000 7fb45318f640 -1 reset not still connected to 0x56133339c410 2024-09-11T06:37:27.159 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:37:27.149+0000 7fb45318f640 -1 reset not still connected to 0x5613480e41a0 2024-09-11T06:37:27.159 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:37:27.149+0000 7fb45318f640 -1 reset not still connected to 0x56134d81fd40 2024-09-11T06:37:27.159 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:37:27.149+0000 7fb45318f640 -1 reset not still connected to 0x561354d77110 2024-09-11T06:37:27.159 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:37:27.149+0000 7fb45318f640 -1 reset not still connected to 0x561355040c30 2024-09-11T06:37:32.558 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:100 IOs, 54545408 bytes 2024-09-11T06:37:33.386 INFO:tasks.ceph.cluster2.osd.1.smithi112.stderr:2024-09-11T06:37:33.386+0000 7f2cecaac640 -1 reset not still connected to 0x55da93619110 2024-09-11T06:37:33.391 INFO:tasks.ceph.cluster2.osd.1.smithi112.stderr:2024-09-11T06:37:33.390+0000 7f2cecaac640 -1 reset not still connected to 0x55da941c60d0 2024-09-11T06:37:33.391 INFO:tasks.ceph.cluster2.osd.1.smithi112.stderr:2024-09-11T06:37:33.390+0000 7f2cecaac640 -1 reset not still connected to 0x55da97fd9ba0 2024-09-11T06:37:33.395 INFO:tasks.ceph.cluster2.osd.1.smithi112.stderr:2024-09-11T06:37:33.394+0000 7f2cecaac640 -1 reset not still connected to 0x55daa159cf70 2024-09-11T06:37:33.395 INFO:tasks.ceph.cluster2.osd.1.smithi112.stderr:2024-09-11T06:37:33.398+0000 7f2cecaac640 -1 reset not still connected to 0x55daa7a52d00 2024-09-11T06:37:33.395 INFO:tasks.ceph.cluster2.osd.1.smithi112.stderr:2024-09-11T06:37:33.398+0000 7f2cecaac640 -1 reset not still connected to 0x55daaabf31e0 2024-09-11T06:37:33.395 INFO:tasks.ceph.cluster2.osd.1.smithi112.stderr:2024-09-11T06:37:33.398+0000 7f2cecaac640 -1 reset not still connected to 0x55dab8b8c270 2024-09-11T06:37:34.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:200 IOs, 78373888 bytes 2024-09-11T06:37:35.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:300 IOs, 82097152 bytes 2024-09-11T06:37:38.357 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:400 IOs, 130480128 bytes 2024-09-11T06:37:40.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ error_code=124 2024-09-11T06:37:40.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set -e 2024-09-11T06:37:40.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T06:37:40.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:37:40.112 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:37:40.112 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_44 2024-09-11T06:37:40.112 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_image_and_enable_mirror cluster2 mirror image_44 journal 128M 2024-09-11T06:37:40.112 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:37:40.112 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:37:40.112 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:37:40.112 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:37:40.112 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_44 2024-09-11T06:37:40.112 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:37:40.112 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local mode=journal 2024-09-11T06:37:40.112 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n journal ']' 2024-09-11T06:37:40.112 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:37:40.112 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_image cluster2 mirror image_44 128M 2024-09-11T06:37:40.113 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:37:40.113 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:37:40.113 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:37:40.113 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:37:40.113 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_44 2024-09-11T06:37:40.113 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:37:40.113 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local size=128 2024-09-11T06:37:40.113 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n 128M ']' 2024-09-11T06:37:40.113 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ size=128M 2024-09-11T06:37:40.113 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:37:40.113 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_44 2024-09-11T06:37:40.250 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T06:37:40.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:37:40.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' pool = image ']' 2024-09-11T06:37:40.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ case "${pool}" in 2024-09-11T06:37:40.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 1 2024-09-11T06:37:40.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n '' ']' 2024-09-11T06:37:40.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ stress_write_image cluster2 mirror image_44 2024-09-11T06:37:40.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:37:40.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:37:40.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_44 2024-09-11T06:37:40.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T06:37:40.255 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local duration=5 2024-09-11T06:37:40.255 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set +e 2024-09-11T06:37:40.255 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_44 --debug-rbd=20 --debug-journaler=20 2024-09-11T06:37:41.293 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:37:41.285+0000 7fb45318f640 -1 reset not still connected to 0x561354d77110 2024-09-11T06:37:41.294 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:37:41.289+0000 7fb45318f640 -1 reset not still connected to 0x561355040c30 2024-09-11T06:37:44.989 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:44.985+0000 7fb2ca654640 -1 reset not still connected to 0x55ed0e363ee0 2024-09-11T06:37:44.992 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:44.985+0000 7fb2ca654640 -1 reset not still connected to 0x55ed0e47ba00 2024-09-11T06:37:44.992 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:44.985+0000 7fb2ca654640 -1 reset not still connected to 0x55ed0fd8cf70 2024-09-11T06:37:44.992 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:44.985+0000 7fb2ca654640 -1 reset not still connected to 0x55ed11f509c0 2024-09-11T06:37:44.992 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:44.985+0000 7fb2ca654640 -1 reset not still connected to 0x55ed15c15ad0 2024-09-11T06:37:44.994 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:44.989+0000 7fb2ca654640 -1 reset not still connected to 0x55ed15c36d00 2024-09-11T06:37:44.994 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:44.989+0000 7fb2ca654640 -1 reset not still connected to 0x55ed169d91e0 2024-09-11T06:37:44.994 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:44.989+0000 7fb2ca654640 -1 reset not still connected to 0x55ed1b91f450 2024-09-11T06:37:44.996 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:44.989+0000 7fb2ca654640 -1 reset not still connected to 0x55ed288cd110 2024-09-11T06:37:44.997 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:44.989+0000 7fb2ca654640 -1 reset not still connected to 0x55ed32b6bba0 2024-09-11T06:37:44.997 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:44.989+0000 7fb2ca654640 -1 reset not still connected to 0x55ed369ccc30 2024-09-11T06:37:44.997 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:44.989+0000 7fb2ca654640 -1 reset not still connected to 0x55ed3a812820 2024-09-11T06:37:44.997 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:44.989+0000 7fb2ca654640 -1 reset not still connected to 0x55ed3ecf6b60 2024-09-11T06:37:45.001 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:44.997+0000 7fb2ca654640 -1 reset not still connected to 0x55ed3ecf76c0 2024-09-11T06:37:45.001 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:44.997+0000 7fb2ca654640 -1 reset not still connected to 0x55ed43611450 2024-09-11T06:37:45.001 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:44.997+0000 7fb2ca654640 -1 reset not still connected to 0x55ed436c8750 2024-09-11T06:37:45.001 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:44.997+0000 7fb2ca654640 -1 reset not still connected to 0x55ed44536340 2024-09-11T06:37:45.001 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:44.997+0000 7fb2ca654640 -1 reset not still connected to 0x55ed4b336c30 2024-09-11T06:37:45.001 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:44.997+0000 7fb2ca654640 -1 reset not still connected to 0x55ed4b37ca90 2024-09-11T06:37:45.266 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ error_code=124 2024-09-11T06:37:45.266 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set -e 2024-09-11T06:37:45.266 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T06:37:45.266 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:37:45.266 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:37:45.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_45 2024-09-11T06:37:45.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_image_and_enable_mirror cluster2 mirror image_45 journal 128M 2024-09-11T06:37:45.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:37:45.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:37:45.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:37:45.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:37:45.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_45 2024-09-11T06:37:45.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:37:45.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local mode=journal 2024-09-11T06:37:45.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n journal ']' 2024-09-11T06:37:45.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:37:45.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_image cluster2 mirror image_45 128M 2024-09-11T06:37:45.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:37:45.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:37:45.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:37:45.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:37:45.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_45 2024-09-11T06:37:45.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:37:45.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local size=128 2024-09-11T06:37:45.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n 128M ']' 2024-09-11T06:37:45.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ size=128M 2024-09-11T06:37:45.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:37:45.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_45 2024-09-11T06:37:45.303 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:45.293+0000 7fb2ca654640 -1 reset not still connected to 0x55ed0d8cdee0 2024-09-11T06:37:45.303 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:45.293+0000 7fb2ca654640 -1 reset not still connected to 0x55ed0e363ee0 2024-09-11T06:37:45.303 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:45.293+0000 7fb2ca654640 -1 reset not still connected to 0x55ed0fd8cf70 2024-09-11T06:37:45.303 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:45.293+0000 7fb2ca654640 -1 reset not still connected to 0x55ed11f509c0 2024-09-11T06:37:45.303 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:45.293+0000 7fb2ca654640 -1 reset not still connected to 0x55ed128edba0 2024-09-11T06:37:45.303 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:45.293+0000 7fb2ca654640 -1 reset not still connected to 0x55ed15c15ad0 2024-09-11T06:37:45.304 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:45.293+0000 7fb2ca654640 -1 reset not still connected to 0x55ed15c36d00 2024-09-11T06:37:45.304 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:45.297+0000 7fb2ca654640 -1 reset not still connected to 0x55ed169d91e0 2024-09-11T06:37:45.304 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:45.297+0000 7fb2ca654640 -1 reset not still connected to 0x55ed1b91f450 2024-09-11T06:37:45.304 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:45.297+0000 7fb2ca654640 -1 reset not still connected to 0x55ed1c2d45b0 2024-09-11T06:37:45.305 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:45.301+0000 7fb2ca654640 -1 reset not still connected to 0x55ed288cd110 2024-09-11T06:37:45.305 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:45.301+0000 7fb2ca654640 -1 reset not still connected to 0x55ed2b074410 2024-09-11T06:37:45.305 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:45.301+0000 7fb2ca654640 -1 reset not still connected to 0x55ed32b6bba0 2024-09-11T06:37:45.305 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:45.301+0000 7fb2ca654640 -1 reset not still connected to 0x55ed32ea5110 2024-09-11T06:37:45.305 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:45.301+0000 7fb2ca654640 -1 reset not still connected to 0x55ed369ccc30 2024-09-11T06:37:45.305 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:45.301+0000 7fb2ca654640 -1 reset not still connected to 0x55ed3a812820 2024-09-11T06:37:45.306 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:45.301+0000 7fb2ca654640 -1 reset not still connected to 0x55ed3ecf6b60 2024-09-11T06:37:45.306 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:45.301+0000 7fb2ca654640 -1 reset not still connected to 0x55ed3ecf76c0 2024-09-11T06:37:45.306 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:45.301+0000 7fb2ca654640 -1 reset not still connected to 0x55ed43611450 2024-09-11T06:37:45.306 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:45.301+0000 7fb2ca654640 -1 reset not still connected to 0x55ed436c8750 2024-09-11T06:37:45.306 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:45.301+0000 7fb2ca654640 -1 reset not still connected to 0x55ed44536340 2024-09-11T06:37:45.306 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:45.301+0000 7fb2ca654640 -1 reset not still connected to 0x55ed4b336c30 2024-09-11T06:37:45.306 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:45.301+0000 7fb2ca654640 -1 reset not still connected to 0x55ed4b37ca90 2024-09-11T06:37:45.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T06:37:45.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:37:45.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' pool = image ']' 2024-09-11T06:37:45.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ case "${pool}" in 2024-09-11T06:37:45.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 1 2024-09-11T06:37:45.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n '' ']' 2024-09-11T06:37:45.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ stress_write_image cluster2 mirror image_45 2024-09-11T06:37:45.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:37:45.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:37:45.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_45 2024-09-11T06:37:45.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T06:37:45.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local duration=12 2024-09-11T06:37:45.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set +e 2024-09-11T06:37:45.372 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_45 --debug-rbd=20 --debug-journaler=20 2024-09-11T06:37:49.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:100 IOs, 43436032 bytes 2024-09-11T06:37:53.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:200 IOs, 96591872 bytes 2024-09-11T06:37:55.637 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:37:55.629+0000 7fb45318f640 -1 reset not still connected to 0x561315ba5520 2024-09-11T06:37:55.647 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:37:55.641+0000 7fb45318f640 -1 reset not still connected to 0x5613184ebad0 2024-09-11T06:37:55.647 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:37:55.641+0000 7fb45318f640 -1 reset not still connected to 0x56131a1ffe10 2024-09-11T06:37:55.647 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:37:55.641+0000 7fb45318f640 -1 reset not still connected to 0x561322931860 2024-09-11T06:37:55.647 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:37:55.641+0000 7fb45318f640 -1 reset not still connected to 0x561322ded930 2024-09-11T06:37:55.647 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:37:55.641+0000 7fb45318f640 -1 reset not still connected to 0x561330ed0410 2024-09-11T06:37:55.647 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:37:55.641+0000 7fb45318f640 -1 reset not still connected to 0x56133339c410 2024-09-11T06:37:55.647 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:37:55.641+0000 7fb45318f640 -1 reset not still connected to 0x56133c1909c0 2024-09-11T06:37:55.647 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:37:55.641+0000 7fb45318f640 -1 reset not still connected to 0x56133c2a2dd0 2024-09-11T06:37:55.647 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:37:55.641+0000 7fb45318f640 -1 reset not still connected to 0x5613480872b0 2024-09-11T06:37:55.648 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:37:55.641+0000 7fb45318f640 -1 reset not still connected to 0x5613480e41a0 2024-09-11T06:37:55.648 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:37:55.641+0000 7fb45318f640 -1 reset not still connected to 0x5613488b0820 2024-09-11T06:37:55.648 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:37:55.641+0000 7fb45318f640 -1 reset not still connected to 0x5613488b0d00 2024-09-11T06:37:55.648 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:37:55.641+0000 7fb45318f640 -1 reset not still connected to 0x56134d76fe10 2024-09-11T06:37:55.648 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:37:55.641+0000 7fb45318f640 -1 reset not still connected to 0x56134d81fd40 2024-09-11T06:37:55.648 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:37:55.641+0000 7fb45318f640 -1 reset not still connected to 0x561354d57040 2024-09-11T06:37:55.648 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:37:55.641+0000 7fb45318f640 -1 reset not still connected to 0x561354d77110 2024-09-11T06:37:55.648 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:37:55.641+0000 7fb45318f640 -1 reset not still connected to 0x561355040c30 2024-09-11T06:37:55.648 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:37:55.641+0000 7fb45318f640 -1 reset not still connected to 0x5613551fd110 2024-09-11T06:37:55.648 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:37:55.641+0000 7fb45318f640 -1 reset not still connected to 0x5613557c4680 2024-09-11T06:37:55.768 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:300 IOs, 108695552 bytes 2024-09-11T06:37:55.787 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:37:55.781+0000 7fb45318f640 -1 reset not still connected to 0x5613488b0820 2024-09-11T06:37:55.787 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:37:55.781+0000 7fb45318f640 -1 reset not still connected to 0x5613488b0d00 2024-09-11T06:37:55.790 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:37:55.781+0000 7fb45318f640 -1 reset not still connected to 0x56134d76fe10 2024-09-11T06:37:55.796 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:37:55.789+0000 7fb45318f640 -1 reset not still connected to 0x56134d81fd40 2024-09-11T06:37:55.796 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:37:55.789+0000 7fb45318f640 -1 reset not still connected to 0x561354d57040 2024-09-11T06:37:55.796 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:37:55.789+0000 7fb45318f640 -1 reset not still connected to 0x561354d77110 2024-09-11T06:37:55.796 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:37:55.789+0000 7fb45318f640 -1 reset not still connected to 0x561355040c30 2024-09-11T06:37:55.796 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:37:55.789+0000 7fb45318f640 -1 reset not still connected to 0x5613551fd110 2024-09-11T06:37:55.796 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:37:55.789+0000 7fb45318f640 -1 reset not still connected to 0x5613557c4680 2024-09-11T06:37:57.378 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ error_code=124 2024-09-11T06:37:57.378 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set -e 2024-09-11T06:37:57.378 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T06:37:57.379 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:37:57.379 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:37:57.379 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_46 2024-09-11T06:37:57.379 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_image_and_enable_mirror cluster2 mirror image_46 journal 128M 2024-09-11T06:37:57.379 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:37:57.379 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:37:57.379 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:37:57.379 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:37:57.379 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_46 2024-09-11T06:37:57.379 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:37:57.379 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local mode=journal 2024-09-11T06:37:57.379 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n journal ']' 2024-09-11T06:37:57.379 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:37:57.380 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_image cluster2 mirror image_46 128M 2024-09-11T06:37:57.380 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:37:57.380 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:37:57.380 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:37:57.380 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:37:57.380 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_46 2024-09-11T06:37:57.380 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:37:57.380 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local size=128 2024-09-11T06:37:57.380 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n 128M ']' 2024-09-11T06:37:57.380 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ size=128M 2024-09-11T06:37:57.380 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:37:57.380 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_46 2024-09-11T06:37:57.452 INFO:tasks.ceph.cluster2.osd.0.smithi112.stderr:2024-09-11T06:37:57.454+0000 7f3fa0017640 -1 reset not still connected to 0x563d407f7ad0 2024-09-11T06:37:57.452 INFO:tasks.ceph.cluster2.osd.0.smithi112.stderr:2024-09-11T06:37:57.454+0000 7f3fa0017640 -1 reset not still connected to 0x563d43515c70 2024-09-11T06:37:57.493 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T06:37:57.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:37:57.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' pool = image ']' 2024-09-11T06:37:57.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ case "${pool}" in 2024-09-11T06:37:57.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 1 2024-09-11T06:37:57.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n '' ']' 2024-09-11T06:37:57.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ stress_write_image cluster2 mirror image_46 2024-09-11T06:37:57.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:37:57.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:37:57.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_46 2024-09-11T06:37:57.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T06:37:57.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local duration=8 2024-09-11T06:37:57.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set +e 2024-09-11T06:37:57.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_46 --debug-rbd=20 --debug-journaler=20 2024-09-11T06:37:57.589 INFO:tasks.ceph.cluster2.osd.0.smithi112.stderr:2024-09-11T06:37:57.590+0000 7f3fa0017640 -1 reset not still connected to 0x563d407f7ad0 2024-09-11T06:37:57.592 INFO:tasks.ceph.cluster2.osd.0.smithi112.stderr:2024-09-11T06:37:57.594+0000 7f3fa0017640 -1 reset not still connected to 0x563d4619c8f0 2024-09-11T06:37:57.831 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:57.825+0000 7fb2ca654640 -1 reset not still connected to 0x55ed44536340 2024-09-11T06:37:57.831 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:57.825+0000 7fb2ca654640 -1 reset not still connected to 0x55ed4b336c30 2024-09-11T06:37:57.831 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:37:57.825+0000 7fb2ca654640 -1 reset not still connected to 0x55ed4b37ca90 2024-09-11T06:37:58.347 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:37:58.341+0000 7fb45318f640 -1 reset not still connected to 0x56131a1ffe10 2024-09-11T06:37:58.347 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:37:58.341+0000 7fb45318f640 -1 reset not still connected to 0x561322931860 2024-09-11T06:37:58.350 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:37:58.345+0000 7fb45318f640 -1 reset not still connected to 0x561322ded930 2024-09-11T06:37:58.350 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:37:58.345+0000 7fb45318f640 -1 reset not still connected to 0x561330ed0410 2024-09-11T06:37:58.350 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:37:58.345+0000 7fb45318f640 -1 reset not still connected to 0x56133c1909c0 2024-09-11T06:37:58.356 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:37:58.349+0000 7fb45318f640 -1 reset not still connected to 0x5613480e41a0 2024-09-11T06:37:58.356 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:37:58.349+0000 7fb45318f640 -1 reset not still connected to 0x5613488b0820 2024-09-11T06:37:58.356 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:37:58.349+0000 7fb45318f640 -1 reset not still connected to 0x5613488b0d00 2024-09-11T06:37:58.356 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:37:58.349+0000 7fb45318f640 -1 reset not still connected to 0x56134d81fd40 2024-09-11T06:37:58.356 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:37:58.349+0000 7fb45318f640 -1 reset not still connected to 0x561354d77110 2024-09-11T06:37:58.356 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:37:58.349+0000 7fb45318f640 -1 reset not still connected to 0x561355040c30 2024-09-11T06:38:00.774 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:38:00.774+0000 7efd718c5640 -1 reset not still connected to 0x560b3a5f0750 2024-09-11T06:38:00.774 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:38:00.774+0000 7efd718c5640 -1 reset not still connected to 0x560b3b89c0d0 2024-09-11T06:38:00.774 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:38:00.774+0000 7efd718c5640 -1 reset not still connected to 0x560b3e445040 2024-09-11T06:38:02.710 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:02.705+0000 7fb45318f640 -1 reset not still connected to 0x561315ba4270 2024-09-11T06:38:02.710 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:02.705+0000 7fb45318f640 -1 reset not still connected to 0x56131a1ffe10 2024-09-11T06:38:02.710 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:02.705+0000 7fb45318f640 -1 reset not still connected to 0x561322931860 2024-09-11T06:38:02.711 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:02.705+0000 7fb45318f640 -1 reset not still connected to 0x561322ded930 2024-09-11T06:38:02.711 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:02.705+0000 7fb45318f640 -1 reset not still connected to 0x561330ed0410 2024-09-11T06:38:02.711 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:02.705+0000 7fb45318f640 -1 reset not still connected to 0x56133c1909c0 2024-09-11T06:38:02.715 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:02.709+0000 7fb45318f640 -1 reset not still connected to 0x5613480e41a0 2024-09-11T06:38:02.715 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:02.709+0000 7fb45318f640 -1 reset not still connected to 0x5613488b0820 2024-09-11T06:38:02.715 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:02.709+0000 7fb45318f640 -1 reset not still connected to 0x5613488b0d00 2024-09-11T06:38:02.715 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:02.709+0000 7fb45318f640 -1 reset not still connected to 0x56134d81fd40 2024-09-11T06:38:02.715 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:02.709+0000 7fb45318f640 -1 reset not still connected to 0x561354d77110 2024-09-11T06:38:02.716 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:02.709+0000 7fb45318f640 -1 reset not still connected to 0x561355040c30 2024-09-11T06:38:04.593 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:04.585+0000 7fb45318f640 -1 reset not still connected to 0x561355040c30 2024-09-11T06:38:05.503 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ error_code=124 2024-09-11T06:38:05.503 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set -e 2024-09-11T06:38:05.503 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T06:38:05.504 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:38:05.504 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:38:05.504 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_47 2024-09-11T06:38:05.504 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_image_and_enable_mirror cluster2 mirror image_47 journal 128M 2024-09-11T06:38:05.504 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:38:05.504 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:38:05.504 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:38:05.504 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:38:05.504 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_47 2024-09-11T06:38:05.504 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:38:05.504 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local mode=journal 2024-09-11T06:38:05.504 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n journal ']' 2024-09-11T06:38:05.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:38:05.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_image cluster2 mirror image_47 128M 2024-09-11T06:38:05.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:38:05.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:38:05.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:38:05.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:38:05.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_47 2024-09-11T06:38:05.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:38:05.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local size=128 2024-09-11T06:38:05.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n 128M ']' 2024-09-11T06:38:05.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ size=128M 2024-09-11T06:38:05.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:38:05.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_47 2024-09-11T06:38:05.617 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T06:38:05.618 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:38:05.618 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' pool = image ']' 2024-09-11T06:38:05.618 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ case "${pool}" in 2024-09-11T06:38:05.618 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 1 2024-09-11T06:38:05.618 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n '' ']' 2024-09-11T06:38:05.618 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ stress_write_image cluster2 mirror image_47 2024-09-11T06:38:05.618 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:38:05.618 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:38:05.618 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_47 2024-09-11T06:38:05.618 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T06:38:05.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local duration=8 2024-09-11T06:38:05.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set +e 2024-09-11T06:38:05.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_47 --debug-rbd=20 --debug-journaler=20 2024-09-11T06:38:05.637 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:38:05.638+0000 7efd718c5640 -1 reset not still connected to 0x560b28ee1ee0 2024-09-11T06:38:05.637 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:38:05.638+0000 7efd718c5640 -1 reset not still connected to 0x560b29059520 2024-09-11T06:38:05.637 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:38:05.638+0000 7efd718c5640 -1 reset not still connected to 0x560b2f4745b0 2024-09-11T06:38:05.637 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:38:05.638+0000 7efd718c5640 -1 reset not still connected to 0x560b2fc524e0 2024-09-11T06:38:05.639 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:38:05.642+0000 7efd718c5640 -1 reset not still connected to 0x560b37798410 2024-09-11T06:38:05.647 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:38:05.650+0000 7efd718c5640 -1 reset not still connected to 0x560b389c9ee0 2024-09-11T06:38:05.647 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:38:05.650+0000 7efd718c5640 -1 reset not still connected to 0x560b41b0f380 2024-09-11T06:38:05.699 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:38:05.693+0000 7fb2ca654640 -1 reset not still connected to 0x55ed0d8655f0 2024-09-11T06:38:05.700 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:38:05.693+0000 7fb2ca654640 -1 reset not still connected to 0x55ed0d8cdee0 2024-09-11T06:38:05.703 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:38:05.693+0000 7fb2ca654640 -1 reset not still connected to 0x55ed0e363ee0 2024-09-11T06:38:05.706 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:38:05.701+0000 7fb2ca654640 -1 reset not still connected to 0x55ed0e47ba00 2024-09-11T06:38:05.707 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:38:05.701+0000 7fb2ca654640 -1 reset not still connected to 0x55ed11f509c0 2024-09-11T06:38:05.707 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:38:05.701+0000 7fb2ca654640 -1 reset not still connected to 0x55ed128edba0 2024-09-11T06:38:05.707 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:38:05.701+0000 7fb2ca654640 -1 reset not still connected to 0x55ed15c15ad0 2024-09-11T06:38:05.707 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:38:05.701+0000 7fb2ca654640 -1 reset not still connected to 0x55ed15c36d00 2024-09-11T06:38:05.707 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:38:05.701+0000 7fb2ca654640 -1 reset not still connected to 0x55ed169d91e0 2024-09-11T06:38:05.707 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:38:05.701+0000 7fb2ca654640 -1 reset not still connected to 0x55ed1b91f450 2024-09-11T06:38:05.707 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:38:05.701+0000 7fb2ca654640 -1 reset not still connected to 0x55ed1c2d45b0 2024-09-11T06:38:05.708 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:38:05.701+0000 7fb2ca654640 -1 reset not still connected to 0x55ed288cd110 2024-09-11T06:38:05.708 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:38:05.701+0000 7fb2ca654640 -1 reset not still connected to 0x55ed2b074410 2024-09-11T06:38:05.708 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:38:05.701+0000 7fb2ca654640 -1 reset not still connected to 0x55ed32b6bba0 2024-09-11T06:38:05.708 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:38:05.701+0000 7fb2ca654640 -1 reset not still connected to 0x55ed369ccc30 2024-09-11T06:38:05.708 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:38:05.701+0000 7fb2ca654640 -1 reset not still connected to 0x55ed3a812820 2024-09-11T06:38:05.713 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:38:05.705+0000 7fb2ca654640 -1 reset not still connected to 0x55ed3ecf6b60 2024-09-11T06:38:05.713 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:38:05.705+0000 7fb2ca654640 -1 reset not still connected to 0x55ed3ecf76c0 2024-09-11T06:38:05.713 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:38:05.705+0000 7fb2ca654640 -1 reset not still connected to 0x55ed43611450 2024-09-11T06:38:05.713 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:38:05.705+0000 7fb2ca654640 -1 reset not still connected to 0x55ed436c8750 2024-09-11T06:38:05.713 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:38:05.705+0000 7fb2ca654640 -1 reset not still connected to 0x55ed44536340 2024-09-11T06:38:05.713 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:38:05.705+0000 7fb2ca654640 -1 reset not still connected to 0x55ed4b336c30 2024-09-11T06:38:05.713 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:38:05.705+0000 7fb2ca654640 -1 reset not still connected to 0x55ed4b37ca90 2024-09-11T06:38:06.398 INFO:tasks.ceph.cluster2.osd.1.smithi112.stderr:2024-09-11T06:38:06.398+0000 7f2cecaac640 -1 reset not still connected to 0x55daa56ae8f0 2024-09-11T06:38:06.399 INFO:tasks.ceph.cluster2.osd.1.smithi112.stderr:2024-09-11T06:38:06.398+0000 7f2cecaac640 -1 reset not still connected to 0x55daaa0475f0 2024-09-11T06:38:06.399 INFO:tasks.ceph.cluster2.osd.1.smithi112.stderr:2024-09-11T06:38:06.398+0000 7f2cecaac640 -1 reset not still connected to 0x55daaabf3520 2024-09-11T06:38:06.399 INFO:tasks.ceph.cluster2.osd.1.smithi112.stderr:2024-09-11T06:38:06.398+0000 7f2cecaac640 -1 reset not still connected to 0x55dab0f5ef70 2024-09-11T06:38:07.724 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:100 IOs, 14766080 bytes 2024-09-11T06:38:07.873 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:07.861+0000 7fb45318f640 -1 reset not still connected to 0x561337a04410 2024-09-11T06:38:07.874 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:07.861+0000 7fb45318f640 -1 reset not still connected to 0x561337b67e10 2024-09-11T06:38:07.876 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:07.861+0000 7fb45318f640 -1 reset not still connected to 0x56133cabc000 2024-09-11T06:38:07.876 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:07.861+0000 7fb45318f640 -1 reset not still connected to 0x561342690270 2024-09-11T06:38:07.876 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:07.861+0000 7fb45318f640 -1 reset not still connected to 0x5613480c0a90 2024-09-11T06:38:07.878 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:07.861+0000 7fb45318f640 -1 reset not still connected to 0x5613480c1040 2024-09-11T06:38:07.878 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:07.861+0000 7fb45318f640 -1 reset not still connected to 0x5613545c3790 2024-09-11T06:38:07.879 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:07.869+0000 7fb45318f640 -1 reset not still connected to 0x5613551aac30 2024-09-11T06:38:08.185 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:08.181+0000 7fb45318f640 -1 reset not still connected to 0x56133c1909c0 2024-09-11T06:38:08.187 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:08.181+0000 7fb45318f640 -1 reset not still connected to 0x561341f69450 2024-09-11T06:38:08.188 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:08.181+0000 7fb45318f640 -1 reset not still connected to 0x5613480e41a0 2024-09-11T06:38:08.188 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:08.181+0000 7fb45318f640 -1 reset not still connected to 0x5613488b0820 2024-09-11T06:38:08.188 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:08.181+0000 7fb45318f640 -1 reset not still connected to 0x5613488b0d00 2024-09-11T06:38:08.188 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:08.181+0000 7fb45318f640 -1 reset not still connected to 0x56134d76fe10 2024-09-11T06:38:08.189 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:08.181+0000 7fb45318f640 -1 reset not still connected to 0x56134d81fd40 2024-09-11T06:38:08.189 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:08.181+0000 7fb45318f640 -1 reset not still connected to 0x561354d77110 2024-09-11T06:38:08.189 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:08.181+0000 7fb45318f640 -1 reset not still connected to 0x561355040c30 2024-09-11T06:38:08.874 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:38:08.874+0000 7efd718c5640 -1 reset not still connected to 0x560b29059520 2024-09-11T06:38:08.875 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:38:08.878+0000 7efd718c5640 -1 reset not still connected to 0x560b2fc524e0 2024-09-11T06:38:08.875 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:38:08.878+0000 7efd718c5640 -1 reset not still connected to 0x560b37798410 2024-09-11T06:38:08.944 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:38:08.946+0000 7efd718c5640 -1 reset not still connected to 0x560b41b0f380 2024-09-11T06:38:09.343 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:09.333+0000 7fb45318f640 -1 reset not still connected to 0x561322bdf450 2024-09-11T06:38:09.344 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:09.337+0000 7fb45318f640 -1 reset not still connected to 0x5613237a41a0 2024-09-11T06:38:09.344 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:09.337+0000 7fb45318f640 -1 reset not still connected to 0x5613273cfd40 2024-09-11T06:38:09.347 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:09.341+0000 7fb45318f640 -1 reset not still connected to 0x561328b5c680 2024-09-11T06:38:09.347 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:09.341+0000 7fb45318f640 -1 reset not still connected to 0x561328b5c8f0 2024-09-11T06:38:09.348 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:09.341+0000 7fb45318f640 -1 reset not still connected to 0x56132e90e340 2024-09-11T06:38:09.348 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:09.341+0000 7fb45318f640 -1 reset not still connected to 0x561333ca9450 2024-09-11T06:38:09.348 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:09.341+0000 7fb45318f640 -1 reset not still connected to 0x561337a04410 2024-09-11T06:38:09.348 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:09.341+0000 7fb45318f640 -1 reset not still connected to 0x561337b67e10 2024-09-11T06:38:09.348 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:09.341+0000 7fb45318f640 -1 reset not still connected to 0x56133cabc000 2024-09-11T06:38:09.348 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:09.341+0000 7fb45318f640 -1 reset not still connected to 0x561342690270 2024-09-11T06:38:09.348 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:09.341+0000 7fb45318f640 -1 reset not still connected to 0x5613480c0a90 2024-09-11T06:38:09.348 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:09.341+0000 7fb45318f640 -1 reset not still connected to 0x5613480c1040 2024-09-11T06:38:09.348 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:09.341+0000 7fb45318f640 -1 reset not still connected to 0x5613545c3790 2024-09-11T06:38:09.348 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:09.341+0000 7fb45318f640 -1 reset not still connected to 0x5613551aac30 2024-09-11T06:38:11.163 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:11.157+0000 7fb45318f640 -1 reset not still connected to 0x56133339c410 2024-09-11T06:38:11.169 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:11.165+0000 7fb45318f640 -1 reset not still connected to 0x56133c1909c0 2024-09-11T06:38:11.169 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:11.165+0000 7fb45318f640 -1 reset not still connected to 0x561341f69450 2024-09-11T06:38:11.169 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:11.165+0000 7fb45318f640 -1 reset not still connected to 0x5613480e41a0 2024-09-11T06:38:11.169 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:11.165+0000 7fb45318f640 -1 reset not still connected to 0x5613488b0820 2024-09-11T06:38:11.169 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:11.165+0000 7fb45318f640 -1 reset not still connected to 0x5613488b0d00 2024-09-11T06:38:11.169 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:11.165+0000 7fb45318f640 -1 reset not still connected to 0x56134d76fe10 2024-09-11T06:38:11.170 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:11.165+0000 7fb45318f640 -1 reset not still connected to 0x56134d81fd40 2024-09-11T06:38:11.170 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:11.165+0000 7fb45318f640 -1 reset not still connected to 0x561354d77110 2024-09-11T06:38:11.170 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:11.165+0000 7fb45318f640 -1 reset not still connected to 0x561355040c30 2024-09-11T06:38:12.011 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:38:12.001+0000 7fc41084f640 -1 reset not still connected to 0x55eb26304820 2024-09-11T06:38:12.011 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:38:12.001+0000 7fc41084f640 -1 reset not still connected to 0x55eb33e98270 2024-09-11T06:38:12.011 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:38:12.001+0000 7fc41084f640 -1 reset not still connected to 0x55eb38da3a00 2024-09-11T06:38:12.011 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:38:12.001+0000 7fc41084f640 -1 reset not still connected to 0x55eb43fe4340 2024-09-11T06:38:13.628 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ error_code=124 2024-09-11T06:38:13.628 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set -e 2024-09-11T06:38:13.628 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T06:38:13.628 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:38:13.628 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:38:13.628 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_48 2024-09-11T06:38:13.628 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_image_and_enable_mirror cluster2 mirror image_48 journal 128M 2024-09-11T06:38:13.629 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:38:13.629 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:38:13.629 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:38:13.629 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:38:13.629 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_48 2024-09-11T06:38:13.629 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:38:13.629 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local mode=journal 2024-09-11T06:38:13.629 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n journal ']' 2024-09-11T06:38:13.629 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:38:13.629 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_image cluster2 mirror image_48 128M 2024-09-11T06:38:13.629 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:38:13.629 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:38:13.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:38:13.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:38:13.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_48 2024-09-11T06:38:13.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:38:13.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local size=128 2024-09-11T06:38:13.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n 128M ']' 2024-09-11T06:38:13.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ size=128M 2024-09-11T06:38:13.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:38:13.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_48 2024-09-11T06:38:13.751 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T06:38:13.751 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:38:13.751 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' pool = image ']' 2024-09-11T06:38:13.751 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ case "${pool}" in 2024-09-11T06:38:13.751 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 1 2024-09-11T06:38:13.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n '' ']' 2024-09-11T06:38:13.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ stress_write_image cluster2 mirror image_48 2024-09-11T06:38:13.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:38:13.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:38:13.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_48 2024-09-11T06:38:13.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T06:38:13.754 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local duration=9 2024-09-11T06:38:13.754 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set +e 2024-09-11T06:38:13.754 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_48 --debug-rbd=20 --debug-journaler=20 2024-09-11T06:38:13.779 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:38:13.773+0000 7fb2ca654640 -1 reset not still connected to 0x55ed0d759d40 2024-09-11T06:38:13.783 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:38:13.777+0000 7fb2ca654640 -1 reset not still connected to 0x55ed0d863040 2024-09-11T06:38:13.783 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:38:13.777+0000 7fb2ca654640 -1 reset not still connected to 0x55ed0d865e10 2024-09-11T06:38:13.788 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:38:13.777+0000 7fb2ca654640 -1 reset not still connected to 0x55ed0d8cd5f0 2024-09-11T06:38:13.788 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:38:13.777+0000 7fb2ca654640 -1 reset not still connected to 0x55ed0e550dd0 2024-09-11T06:38:13.788 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:38:13.777+0000 7fb2ca654640 -1 reset not still connected to 0x55ed12570820 2024-09-11T06:38:13.789 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:38:13.777+0000 7fb2ca654640 -1 reset not still connected to 0x55ed149fe8f0 2024-09-11T06:38:13.789 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:38:13.777+0000 7fb2ca654640 -1 reset not still connected to 0x55ed161c2000 2024-09-11T06:38:13.789 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:38:13.777+0000 7fb2ca654640 -1 reset not still connected to 0x55ed194eaea0 2024-09-11T06:38:13.789 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:38:13.781+0000 7fb2ca654640 -1 reset not still connected to 0x55ed1bd1add0 2024-09-11T06:38:13.790 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:38:13.781+0000 7fb2ca654640 -1 reset not still connected to 0x55ed2a1e0ea0 2024-09-11T06:38:13.790 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:38:13.781+0000 7fb2ca654640 -1 reset not still connected to 0x55ed38d48270 2024-09-11T06:38:13.790 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:38:13.781+0000 7fb2ca654640 -1 reset not still connected to 0x55ed3d399d40 2024-09-11T06:38:13.790 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:38:13.781+0000 7fb2ca654640 -1 reset not still connected to 0x55ed50ca3380 2024-09-11T06:38:14.136 INFO:tasks.ceph.cluster2.osd.0.smithi112.stderr:2024-09-11T06:38:14.138+0000 7f3fa0017640 -1 reset not still connected to 0x563d2e638dd0 2024-09-11T06:38:14.137 INFO:tasks.ceph.cluster2.osd.0.smithi112.stderr:2024-09-11T06:38:14.138+0000 7f3fa0017640 -1 reset not still connected to 0x563d39483520 2024-09-11T06:38:14.617 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:38:14.618+0000 7efd718c5640 -1 reset not still connected to 0x560b15a70a90 2024-09-11T06:38:14.628 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:38:14.630+0000 7efd718c5640 -1 reset not still connected to 0x560b15a729c0 2024-09-11T06:38:14.628 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:38:14.630+0000 7efd718c5640 -1 reset not still connected to 0x560b1899a410 2024-09-11T06:38:14.628 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:38:14.630+0000 7efd718c5640 -1 reset not still connected to 0x560b1d3f3380 2024-09-11T06:38:14.628 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:38:14.630+0000 7efd718c5640 -1 reset not still connected to 0x560b28ee1ee0 2024-09-11T06:38:14.628 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:38:14.630+0000 7efd718c5640 -1 reset not still connected to 0x560b29059520 2024-09-11T06:38:14.628 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:38:14.630+0000 7efd718c5640 -1 reset not still connected to 0x560b2f4745b0 2024-09-11T06:38:14.628 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:38:14.630+0000 7efd718c5640 -1 reset not still connected to 0x560b2fc524e0 2024-09-11T06:38:14.628 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:38:14.630+0000 7efd718c5640 -1 reset not still connected to 0x560b34cafee0 2024-09-11T06:38:14.628 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:38:14.630+0000 7efd718c5640 -1 reset not still connected to 0x560b37798410 2024-09-11T06:38:14.629 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:38:14.630+0000 7efd718c5640 -1 reset not still connected to 0x560b389c9ee0 2024-09-11T06:38:14.629 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:38:14.630+0000 7efd718c5640 -1 reset not still connected to 0x560b41b0f380 2024-09-11T06:38:15.823 INFO:tasks.ceph.cluster2.osd.1.smithi112.stderr:2024-09-11T06:38:15.826+0000 7f2cecaac640 -1 reset not still connected to 0x55daa159cf70 2024-09-11T06:38:15.823 INFO:tasks.ceph.cluster2.osd.1.smithi112.stderr:2024-09-11T06:38:15.826+0000 7f2cecaac640 -1 reset not still connected to 0x55daa7a52d00 2024-09-11T06:38:15.823 INFO:tasks.ceph.cluster2.osd.1.smithi112.stderr:2024-09-11T06:38:15.826+0000 7f2cecaac640 -1 reset not still connected to 0x55daaabf31e0 2024-09-11T06:38:15.823 INFO:tasks.ceph.cluster2.osd.1.smithi112.stderr:2024-09-11T06:38:15.826+0000 7f2cecaac640 -1 reset not still connected to 0x55daaf92cdd0 2024-09-11T06:38:15.823 INFO:tasks.ceph.cluster2.osd.1.smithi112.stderr:2024-09-11T06:38:15.826+0000 7f2cecaac640 -1 reset not still connected to 0x55dab8b8c270 2024-09-11T06:38:16.076 INFO:tasks.ceph.cluster2.osd.1.smithi112.stderr:2024-09-11T06:38:16.078+0000 7f2cecaac640 -1 reset not still connected to 0x55da95964a90 2024-09-11T06:38:16.077 INFO:tasks.ceph.cluster2.osd.1.smithi112.stderr:2024-09-11T06:38:16.078+0000 7f2cecaac640 -1 reset not still connected to 0x55daa56ae8f0 2024-09-11T06:38:16.077 INFO:tasks.ceph.cluster2.osd.1.smithi112.stderr:2024-09-11T06:38:16.078+0000 7f2cecaac640 -1 reset not still connected to 0x55daaa0475f0 2024-09-11T06:38:16.078 INFO:tasks.ceph.cluster2.osd.1.smithi112.stderr:2024-09-11T06:38:16.078+0000 7f2cecaac640 -1 reset not still connected to 0x55daaabf3520 2024-09-11T06:38:16.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:100 IOs, 21953536 bytes 2024-09-11T06:38:18.123 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:200 IOs, 40268800 bytes 2024-09-11T06:38:18.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:300 IOs, 40869888 bytes 2024-09-11T06:38:20.347 INFO:tasks.ceph.cluster2.osd.0.smithi112.stderr:2024-09-11T06:38:20.350+0000 7f3fa0017640 -1 reset not still connected to 0x563d44f31a00 2024-09-11T06:38:20.350 INFO:tasks.ceph.cluster2.osd.0.smithi112.stderr:2024-09-11T06:38:20.350+0000 7f3fa0017640 -1 reset not still connected to 0x563d465ff380 2024-09-11T06:38:20.605 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:38:20.597+0000 7fc41084f640 -1 reset not still connected to 0x55eb1afbe4e0 2024-09-11T06:38:20.605 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:38:20.601+0000 7fc41084f640 -1 reset not still connected to 0x55eb2458e680 2024-09-11T06:38:20.605 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:38:20.601+0000 7fc41084f640 -1 reset not still connected to 0x55eb24ad96c0 2024-09-11T06:38:20.605 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:38:20.601+0000 7fc41084f640 -1 reset not still connected to 0x55eb2601e820 2024-09-11T06:38:20.606 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:38:20.601+0000 7fc41084f640 -1 reset not still connected to 0x55eb2da77ba0 2024-09-11T06:38:20.606 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:38:20.601+0000 7fc41084f640 -1 reset not still connected to 0x55eb345e5ad0 2024-09-11T06:38:20.606 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:38:20.601+0000 7fc41084f640 -1 reset not still connected to 0x55eb35243040 2024-09-11T06:38:20.606 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:38:20.601+0000 7fc41084f640 -1 reset not still connected to 0x55eb354a21a0 2024-09-11T06:38:20.606 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:38:20.601+0000 7fc41084f640 -1 reset not still connected to 0x55eb3893ac30 2024-09-11T06:38:20.606 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:38:20.601+0000 7fc41084f640 -1 reset not still connected to 0x55eb3add8270 2024-09-11T06:38:20.606 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:38:20.601+0000 7fc41084f640 -1 reset not still connected to 0x55eb4480b930 2024-09-11T06:38:20.606 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:38:20.601+0000 7fc41084f640 -1 reset not still connected to 0x55eb4f1329c0 2024-09-11T06:38:20.606 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:38:20.601+0000 7fc41084f640 -1 reset not still connected to 0x55eb53d56f70 2024-09-11T06:38:20.645 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:20.641+0000 7fb45318f640 -1 reset not still connected to 0x5613480c0a90 2024-09-11T06:38:20.645 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:20.641+0000 7fb45318f640 -1 reset not still connected to 0x5613480c1040 2024-09-11T06:38:20.645 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:20.641+0000 7fb45318f640 -1 reset not still connected to 0x56134f7f48f0 2024-09-11T06:38:20.645 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:20.641+0000 7fb45318f640 -1 reset not still connected to 0x5613551aac30 2024-09-11T06:38:21.238 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:21.233+0000 7fb45318f640 -1 reset not still connected to 0x5613480c1040 2024-09-11T06:38:21.239 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:21.233+0000 7fb45318f640 -1 reset not still connected to 0x56134f7f48f0 2024-09-11T06:38:21.239 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:21.233+0000 7fb45318f640 -1 reset not still connected to 0x5613545c3790 2024-09-11T06:38:21.239 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:21.233+0000 7fb45318f640 -1 reset not still connected to 0x5613551aac30 2024-09-11T06:38:21.268 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:38:21.261+0000 7fc41084f640 -1 reset not still connected to 0x55eb43fe4340 2024-09-11T06:38:21.268 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:38:21.261+0000 7fc41084f640 -1 reset not still connected to 0x55eb4464a410 2024-09-11T06:38:21.580 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:400 IOs, 77296640 bytes 2024-09-11T06:38:21.917 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:500 IOs, 77931520 bytes 2024-09-11T06:38:21.922 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:38:21.913+0000 7fc41084f640 -1 reset not still connected to 0x55eb1afbe4e0 2024-09-11T06:38:21.922 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:38:21.913+0000 7fc41084f640 -1 reset not still connected to 0x55eb2458e680 2024-09-11T06:38:21.922 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:38:21.913+0000 7fc41084f640 -1 reset not still connected to 0x55eb24ad96c0 2024-09-11T06:38:21.922 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:38:21.913+0000 7fc41084f640 -1 reset not still connected to 0x55eb2601e820 2024-09-11T06:38:21.923 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:38:21.913+0000 7fc41084f640 -1 reset not still connected to 0x55eb2da77ba0 2024-09-11T06:38:21.923 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:38:21.913+0000 7fc41084f640 -1 reset not still connected to 0x55eb345e5ad0 2024-09-11T06:38:21.923 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:38:21.913+0000 7fc41084f640 -1 reset not still connected to 0x55eb35243040 2024-09-11T06:38:21.923 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:38:21.913+0000 7fc41084f640 -1 reset not still connected to 0x55eb354a21a0 2024-09-11T06:38:21.923 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:38:21.913+0000 7fc41084f640 -1 reset not still connected to 0x55eb3893ac30 2024-09-11T06:38:21.923 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:38:21.913+0000 7fc41084f640 -1 reset not still connected to 0x55eb3add8270 2024-09-11T06:38:21.923 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:38:21.913+0000 7fc41084f640 -1 reset not still connected to 0x55eb4480b930 2024-09-11T06:38:21.923 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:38:21.913+0000 7fc41084f640 -1 reset not still connected to 0x55eb4f1329c0 2024-09-11T06:38:21.923 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:38:21.913+0000 7fc41084f640 -1 reset not still connected to 0x55eb53d56f70 2024-09-11T06:38:22.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ error_code=124 2024-09-11T06:38:22.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set -e 2024-09-11T06:38:22.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T06:38:22.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:38:22.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:38:22.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_49 2024-09-11T06:38:22.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_image_and_enable_mirror cluster2 mirror image_49 journal 128M 2024-09-11T06:38:22.764 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:38:22.764 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:38:22.764 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:38:22.764 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:38:22.764 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_49 2024-09-11T06:38:22.764 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:38:22.764 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local mode=journal 2024-09-11T06:38:22.764 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n journal ']' 2024-09-11T06:38:22.764 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:38:22.764 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_image cluster2 mirror image_49 128M 2024-09-11T06:38:22.764 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:38:22.764 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:38:22.764 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:38:22.764 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:38:22.765 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_49 2024-09-11T06:38:22.765 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:38:22.765 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local size=128 2024-09-11T06:38:22.765 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n 128M ']' 2024-09-11T06:38:22.765 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ size=128M 2024-09-11T06:38:22.765 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:38:22.765 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_49 2024-09-11T06:38:22.890 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T06:38:22.890 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:38:22.890 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' pool = image ']' 2024-09-11T06:38:22.890 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ case "${pool}" in 2024-09-11T06:38:22.890 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 1 2024-09-11T06:38:22.890 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n '' ']' 2024-09-11T06:38:22.890 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ stress_write_image cluster2 mirror image_49 2024-09-11T06:38:22.890 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:38:22.891 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:38:22.891 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_49 2024-09-11T06:38:22.891 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T06:38:22.892 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local duration=8 2024-09-11T06:38:22.892 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set +e 2024-09-11T06:38:22.892 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_49 --debug-rbd=20 --debug-journaler=20 2024-09-11T06:38:25.895 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:100 IOs, 31194112 bytes 2024-09-11T06:38:28.728 INFO:tasks.ceph.cluster2.osd.1.smithi112.stderr:2024-09-11T06:38:28.729+0000 7f2cecaac640 -1 reset not still connected to 0x55da97fd9ba0 2024-09-11T06:38:28.728 INFO:tasks.ceph.cluster2.osd.1.smithi112.stderr:2024-09-11T06:38:28.729+0000 7f2cecaac640 -1 reset not still connected to 0x55daa159cf70 2024-09-11T06:38:28.734 INFO:tasks.ceph.cluster2.osd.1.smithi112.stderr:2024-09-11T06:38:28.733+0000 7f2cecaac640 -1 reset not still connected to 0x55daaabf31e0 2024-09-11T06:38:28.734 INFO:tasks.ceph.cluster2.osd.1.smithi112.stderr:2024-09-11T06:38:28.733+0000 7f2cecaac640 -1 reset not still connected to 0x55daaf92cdd0 2024-09-11T06:38:28.734 INFO:tasks.ceph.cluster2.osd.1.smithi112.stderr:2024-09-11T06:38:28.733+0000 7f2cecaac640 -1 reset not still connected to 0x55dab8b8c270 2024-09-11T06:38:28.803 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:38:28.797+0000 7fc41084f640 -1 reset not still connected to 0x55eb24ad96c0 2024-09-11T06:38:28.803 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:38:28.797+0000 7fc41084f640 -1 reset not still connected to 0x55eb2601e820 2024-09-11T06:38:28.803 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:38:28.797+0000 7fc41084f640 -1 reset not still connected to 0x55eb2613e680 2024-09-11T06:38:28.803 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:38:28.797+0000 7fc41084f640 -1 reset not still connected to 0x55eb2da77ba0 2024-09-11T06:38:28.805 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:38:28.801+0000 7fc41084f640 -1 reset not still connected to 0x55eb345e5ad0 2024-09-11T06:38:28.805 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:38:28.801+0000 7fc41084f640 -1 reset not still connected to 0x55eb34898a90 2024-09-11T06:38:28.805 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:38:28.801+0000 7fc41084f640 -1 reset not still connected to 0x55eb35243040 2024-09-11T06:38:28.805 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:38:28.801+0000 7fc41084f640 -1 reset not still connected to 0x55eb354a21a0 2024-09-11T06:38:28.805 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:38:28.801+0000 7fc41084f640 -1 reset not still connected to 0x55eb3893ac30 2024-09-11T06:38:28.806 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:38:28.801+0000 7fc41084f640 -1 reset not still connected to 0x55eb3add8270 2024-09-11T06:38:28.806 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:38:28.801+0000 7fc41084f640 -1 reset not still connected to 0x55eb4480b930 2024-09-11T06:38:28.807 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:38:28.801+0000 7fc41084f640 -1 reset not still connected to 0x55eb4f1329c0 2024-09-11T06:38:28.807 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:38:28.801+0000 7fc41084f640 -1 reset not still connected to 0x55eb53d56f70 2024-09-11T06:38:30.902 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ error_code=124 2024-09-11T06:38:30.902 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set -e 2024-09-11T06:38:30.902 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T06:38:30.902 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:38:30.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:38:30.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_50 2024-09-11T06:38:30.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_image_and_enable_mirror cluster2 mirror image_50 journal 128M 2024-09-11T06:38:30.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:38:30.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:38:30.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:38:30.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:38:30.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_50 2024-09-11T06:38:30.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:38:30.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local mode=journal 2024-09-11T06:38:30.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n journal ']' 2024-09-11T06:38:30.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:38:30.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_image cluster2 mirror image_50 128M 2024-09-11T06:38:30.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:38:30.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:38:30.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:38:30.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:38:30.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_50 2024-09-11T06:38:30.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:38:30.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local size=128 2024-09-11T06:38:30.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n 128M ']' 2024-09-11T06:38:30.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ size=128M 2024-09-11T06:38:30.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:38:30.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,journaling mirror/image_50 2024-09-11T06:38:31.036 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T06:38:31.036 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:38:31.036 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' pool = image ']' 2024-09-11T06:38:31.036 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ case "${pool}" in 2024-09-11T06:38:31.036 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 1 2024-09-11T06:38:31.036 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n '' ']' 2024-09-11T06:38:31.037 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ stress_write_image cluster2 mirror image_50 2024-09-11T06:38:31.037 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:38:31.037 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:38:31.037 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_50 2024-09-11T06:38:31.038 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T06:38:31.043 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local duration=13 2024-09-11T06:38:31.043 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set +e 2024-09-11T06:38:31.044 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_50 --debug-rbd=20 --debug-journaler=20 2024-09-11T06:38:34.495 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:38:34.489+0000 7fb2ca654640 -1 reset not still connected to 0x55ed288cd110 2024-09-11T06:38:34.499 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:38:34.493+0000 7fb2ca654640 -1 reset not still connected to 0x55ed32b6bba0 2024-09-11T06:38:34.499 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:38:34.493+0000 7fb2ca654640 -1 reset not still connected to 0x55ed32ea5110 2024-09-11T06:38:34.499 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:38:34.493+0000 7fb2ca654640 -1 reset not still connected to 0x55ed369ccc30 2024-09-11T06:38:34.499 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:38:34.493+0000 7fb2ca654640 -1 reset not still connected to 0x55ed3a812820 2024-09-11T06:38:34.505 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:38:34.497+0000 7fb2ca654640 -1 reset not still connected to 0x55ed3ecf6b60 2024-09-11T06:38:34.505 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:38:34.497+0000 7fb2ca654640 -1 reset not still connected to 0x55ed3ecf76c0 2024-09-11T06:38:34.505 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:38:34.497+0000 7fb2ca654640 -1 reset not still connected to 0x55ed43611450 2024-09-11T06:38:34.505 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:38:34.497+0000 7fb2ca654640 -1 reset not still connected to 0x55ed436c8750 2024-09-11T06:38:34.505 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:38:34.497+0000 7fb2ca654640 -1 reset not still connected to 0x55ed44536340 2024-09-11T06:38:34.505 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:38:34.497+0000 7fb2ca654640 -1 reset not still connected to 0x55ed44538a90 2024-09-11T06:38:34.506 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:38:34.497+0000 7fb2ca654640 -1 reset not still connected to 0x55ed44538dd0 2024-09-11T06:38:34.506 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:38:34.497+0000 7fb2ca654640 -1 reset not still connected to 0x55ed4b336c30 2024-09-11T06:38:34.506 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:38:34.497+0000 7fb2ca654640 -1 reset not still connected to 0x55ed4b37ca90 2024-09-11T06:38:35.785 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:38:35.785+0000 7efd718c5640 -1 reset not still connected to 0x560b15a71ad0 2024-09-11T06:38:36.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:100 IOs, 67194880 bytes 2024-09-11T06:38:38.563 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:200 IOs, 87885824 bytes 2024-09-11T06:38:39.765 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:39.757+0000 7fb45318f640 -1 reset not still connected to 0x56131a1ffe10 2024-09-11T06:38:39.765 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:39.757+0000 7fb45318f640 -1 reset not still connected to 0x561322931860 2024-09-11T06:38:39.765 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:39.757+0000 7fb45318f640 -1 reset not still connected to 0x561322ded930 2024-09-11T06:38:39.765 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:39.757+0000 7fb45318f640 -1 reset not still connected to 0x56132c963520 2024-09-11T06:38:39.765 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:39.757+0000 7fb45318f640 -1 reset not still connected to 0x561330ed0410 2024-09-11T06:38:39.766 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:39.757+0000 7fb45318f640 -1 reset not still connected to 0x56133c1909c0 2024-09-11T06:38:39.766 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:39.757+0000 7fb45318f640 -1 reset not still connected to 0x561341f69450 2024-09-11T06:38:39.766 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:39.757+0000 7fb45318f640 -1 reset not still connected to 0x561342043930 2024-09-11T06:38:39.766 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:39.757+0000 7fb45318f640 -1 reset not still connected to 0x5613480e41a0 2024-09-11T06:38:39.766 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:39.757+0000 7fb45318f640 -1 reset not still connected to 0x5613488b0820 2024-09-11T06:38:39.766 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:39.757+0000 7fb45318f640 -1 reset not still connected to 0x5613488b0d00 2024-09-11T06:38:39.766 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:39.757+0000 7fb45318f640 -1 reset not still connected to 0x56134d81fd40 2024-09-11T06:38:39.766 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:39.757+0000 7fb45318f640 -1 reset not still connected to 0x561354d77110 2024-09-11T06:38:39.766 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:39.757+0000 7fb45318f640 -1 reset not still connected to 0x561355040c30 2024-09-11T06:38:40.265 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:40.257+0000 7fb45318f640 -1 reset not still connected to 0x56131a1ffe10 2024-09-11T06:38:40.267 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:40.257+0000 7fb45318f640 -1 reset not still connected to 0x561322931860 2024-09-11T06:38:40.267 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:40.257+0000 7fb45318f640 -1 reset not still connected to 0x561322ded930 2024-09-11T06:38:40.267 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:40.257+0000 7fb45318f640 -1 reset not still connected to 0x56132c963520 2024-09-11T06:38:40.267 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:40.257+0000 7fb45318f640 -1 reset not still connected to 0x561330ed0410 2024-09-11T06:38:40.267 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:40.257+0000 7fb45318f640 -1 reset not still connected to 0x56133c1909c0 2024-09-11T06:38:40.267 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:40.257+0000 7fb45318f640 -1 reset not still connected to 0x561341f69450 2024-09-11T06:38:40.267 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:40.257+0000 7fb45318f640 -1 reset not still connected to 0x561342043930 2024-09-11T06:38:40.268 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:40.257+0000 7fb45318f640 -1 reset not still connected to 0x5613480e41a0 2024-09-11T06:38:40.268 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:40.257+0000 7fb45318f640 -1 reset not still connected to 0x5613488b0820 2024-09-11T06:38:40.268 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:40.257+0000 7fb45318f640 -1 reset not still connected to 0x5613488b0d00 2024-09-11T06:38:40.268 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:40.257+0000 7fb45318f640 -1 reset not still connected to 0x56134d81fd40 2024-09-11T06:38:40.268 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:40.257+0000 7fb45318f640 -1 reset not still connected to 0x561354d77110 2024-09-11T06:38:40.268 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:40.257+0000 7fb45318f640 -1 reset not still connected to 0x561355040c30 2024-09-11T06:38:40.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:300 IOs, 105291776 bytes 2024-09-11T06:38:40.553 INFO:tasks.ceph.cluster2.osd.0.smithi112.stderr:2024-09-11T06:38:40.553+0000 7f3fa0017640 -1 reset not still connected to 0x563d39483520 2024-09-11T06:38:40.554 INFO:tasks.ceph.cluster2.osd.0.smithi112.stderr:2024-09-11T06:38:40.553+0000 7f3fa0017640 -1 reset not still connected to 0x563d44f31a00 2024-09-11T06:38:40.554 INFO:tasks.ceph.cluster2.osd.0.smithi112.stderr:2024-09-11T06:38:40.553+0000 7f3fa0017640 -1 reset not still connected to 0x563d465ff380 2024-09-11T06:38:41.991 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:400 IOs, 124322816 bytes 2024-09-11T06:38:43.462 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:500 IOs, 132093952 bytes 2024-09-11T06:38:44.053 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ error_code=124 2024-09-11T06:38:44.053 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set -e 2024-09-11T06:38:44.054 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T06:38:44.054 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:38:44.054 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_pool_images cluster2 mirror 50 2024-09-11T06:38:44.054 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:38:44.054 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:38:44.054 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_count=50 2024-09-11T06:38:44.054 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:38:44.054 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local count 2024-09-11T06:38:44.054 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local last_count=0 2024-09-11T06:38:44.054 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:38:44.054 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 1 40 2024-09-11T06:38:44.206 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in `seq 1 40` 2024-09-11T06:38:44.207 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 1 -ne 1 2024-09-11T06:38:44.208 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep 'images: ' 2024-09-11T06:38:44.209 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut '-d ' -f 2 2024-09-11T06:38:44.209 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 -p mirror mirror pool status 2024-09-11T06:38:44.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: failed to query services: (13) Permission denied 2024-09-11T06:38:44.277 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ count=50 2024-09-11T06:38:44.277 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 50 = 50 2024-09-11T06:38:44.277 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:38:44.277 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_pool_healthy cluster2 mirror 2024-09-11T06:38:44.277 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:38:44.277 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:38:44.277 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:38:44.277 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state 2024-09-11T06:38:44.278 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 1 40 2024-09-11T06:38:44.281 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in `seq 1 40` 2024-09-11T06:38:44.281 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 1 -ne 1 2024-09-11T06:38:44.281 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 -p mirror mirror pool status 2024-09-11T06:38:44.282 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut '-d ' -f 3 2024-09-11T06:38:44.283 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep 'image health:' 2024-09-11T06:38:44.336 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:38:44.332+0000 7fb2ca654640 -1 reset not still connected to 0x55ed0e47ba00 2024-09-11T06:38:44.336 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:38:44.332+0000 7fb2ca654640 -1 reset not still connected to 0x55ed11f509c0 2024-09-11T06:38:44.340 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:38:44.336+0000 7fb2ca654640 -1 reset not still connected to 0x55ed15c15ad0 2024-09-11T06:38:44.340 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:38:44.336+0000 7fb2ca654640 -1 reset not still connected to 0x55ed15c36d00 2024-09-11T06:38:44.340 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:38:44.336+0000 7fb2ca654640 -1 reset not still connected to 0x55ed169d91e0 2024-09-11T06:38:44.340 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:38:44.336+0000 7fb2ca654640 -1 reset not still connected to 0x55ed1b91f450 2024-09-11T06:38:44.340 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:38:44.336+0000 7fb2ca654640 -1 reset not still connected to 0x55ed288cd110 2024-09-11T06:38:44.340 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:38:44.336+0000 7fb2ca654640 -1 reset not still connected to 0x55ed32b6bba0 2024-09-11T06:38:44.341 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:38:44.336+0000 7fb2ca654640 -1 reset not still connected to 0x55ed369ccc30 2024-09-11T06:38:44.341 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:38:44.336+0000 7fb2ca654640 -1 reset not still connected to 0x55ed3a812820 2024-09-11T06:38:44.341 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:38:44.336+0000 7fb2ca654640 -1 reset not still connected to 0x55ed43611450 2024-09-11T06:38:44.341 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:38:44.336+0000 7fb2ca654640 -1 reset not still connected to 0x55ed436c8750 2024-09-11T06:38:44.341 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:38:44.336+0000 7fb2ca654640 -1 reset not still connected to 0x55ed44536340 2024-09-11T06:38:44.342 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:38:44.336+0000 7fb2ca654640 -1 reset not still connected to 0x55ed4b336c30 2024-09-11T06:38:44.342 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:38:44.336+0000 7fb2ca654640 -1 reset not still connected to 0x55ed4b37ca90 2024-09-11T06:38:44.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: failed to query services: (13) Permission denied 2024-09-11T06:38:44.350 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ state=WARNING 2024-09-11T06:38:44.350 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test WARNING = ERROR 2024-09-11T06:38:44.350 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test WARNING = OK 2024-09-11T06:38:44.350 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in `seq 1 40` 2024-09-11T06:38:44.350 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 2 -ne 1 2024-09-11T06:38:44.351 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 30 2024-09-11T06:38:45.103 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:45.100+0000 7fb45318f640 -1 reset not still connected to 0x5613273cfd40 2024-09-11T06:38:45.103 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:45.100+0000 7fb45318f640 -1 reset not still connected to 0x561328b5c680 2024-09-11T06:38:45.103 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:45.100+0000 7fb45318f640 -1 reset not still connected to 0x561328b5c8f0 2024-09-11T06:38:45.104 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:45.100+0000 7fb45318f640 -1 reset not still connected to 0x56132e90e340 2024-09-11T06:38:45.104 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:45.100+0000 7fb45318f640 -1 reset not still connected to 0x561333ca9450 2024-09-11T06:38:45.104 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:45.100+0000 7fb45318f640 -1 reset not still connected to 0x561337a04410 2024-09-11T06:38:45.104 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:45.100+0000 7fb45318f640 -1 reset not still connected to 0x561337b67e10 2024-09-11T06:38:45.104 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:45.100+0000 7fb45318f640 -1 reset not still connected to 0x56133cabc000 2024-09-11T06:38:45.104 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:45.100+0000 7fb45318f640 -1 reset not still connected to 0x561342690270 2024-09-11T06:38:45.104 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:45.100+0000 7fb45318f640 -1 reset not still connected to 0x5613480c1040 2024-09-11T06:38:45.105 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:45.100+0000 7fb45318f640 -1 reset not still connected to 0x56134f7f48f0 2024-09-11T06:38:45.105 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:45.100+0000 7fb45318f640 -1 reset not still connected to 0x5613545c3790 2024-09-11T06:38:45.105 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:45.100+0000 7fb45318f640 -1 reset not still connected to 0x5613551aac30 2024-09-11T06:38:48.043 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:38:48.045+0000 7efd718c5640 -1 reset not still connected to 0x560b2f4745b0 2024-09-11T06:38:48.043 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:38:48.045+0000 7efd718c5640 -1 reset not still connected to 0x560b2fc524e0 2024-09-11T06:38:48.048 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:38:48.049+0000 7efd718c5640 -1 reset not still connected to 0x560b34cafee0 2024-09-11T06:38:48.048 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:38:48.049+0000 7efd718c5640 -1 reset not still connected to 0x560b37798410 2024-09-11T06:38:48.049 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:38:48.049+0000 7efd718c5640 -1 reset not still connected to 0x560b389c9ee0 2024-09-11T06:38:48.049 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:38:48.049+0000 7efd718c5640 -1 reset not still connected to 0x560b41b0f380 2024-09-11T06:38:48.768 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:38:48.764+0000 7fc41084f640 -1 reset not still connected to 0x55eb2613e680 2024-09-11T06:38:48.769 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:38:48.764+0000 7fc41084f640 -1 reset not still connected to 0x55eb3add8270 2024-09-11T06:38:48.774 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:38:48.772+0000 7fc41084f640 -1 reset not still connected to 0x55eb4480b930 2024-09-11T06:38:48.774 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:38:48.772+0000 7fc41084f640 -1 reset not still connected to 0x55eb4f1329c0 2024-09-11T06:38:48.774 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:38:48.772+0000 7fc41084f640 -1 reset not still connected to 0x55eb53d56f70 2024-09-11T06:38:49.965 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:38:49.960+0000 7fc41084f640 -1 reset not still connected to 0x55eb1afbe4e0 2024-09-11T06:38:49.965 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:38:49.960+0000 7fc41084f640 -1 reset not still connected to 0x55eb1fefeb60 2024-09-11T06:38:49.966 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:38:49.960+0000 7fc41084f640 -1 reset not still connected to 0x55eb2458e680 2024-09-11T06:38:49.966 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:38:49.960+0000 7fc41084f640 -1 reset not still connected to 0x55eb24ad96c0 2024-09-11T06:38:49.966 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:38:49.960+0000 7fc41084f640 -1 reset not still connected to 0x55eb2601e820 2024-09-11T06:38:49.966 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:38:49.960+0000 7fc41084f640 -1 reset not still connected to 0x55eb2613e680 2024-09-11T06:38:49.966 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:38:49.960+0000 7fc41084f640 -1 reset not still connected to 0x55eb2da77ba0 2024-09-11T06:38:49.966 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:38:49.960+0000 7fc41084f640 -1 reset not still connected to 0x55eb34898a90 2024-09-11T06:38:49.966 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:38:49.960+0000 7fc41084f640 -1 reset not still connected to 0x55eb35243040 2024-09-11T06:38:49.984 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:38:49.980+0000 7fc41084f640 -1 reset not still connected to 0x55eb354a21a0 2024-09-11T06:38:50.005 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:38:50.000+0000 7fc41084f640 -1 reset not still connected to 0x55eb3add8270 2024-09-11T06:38:50.005 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:38:50.000+0000 7fc41084f640 -1 reset not still connected to 0x55eb4480b930 2024-09-11T06:38:50.005 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:38:50.000+0000 7fc41084f640 -1 reset not still connected to 0x55eb4f1329c0 2024-09-11T06:38:50.005 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:38:50.000+0000 7fc41084f640 -1 reset not still connected to 0x55eb53d56f70 2024-09-11T06:38:50.399 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:50.396+0000 7fb45318f640 -1 reset not still connected to 0x561330ed0410 2024-09-11T06:38:50.404 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:50.400+0000 7fb45318f640 -1 reset not still connected to 0x56133339c410 2024-09-11T06:38:50.407 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:50.400+0000 7fb45318f640 -1 reset not still connected to 0x5613480e41a0 2024-09-11T06:38:50.408 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:50.404+0000 7fb45318f640 -1 reset not still connected to 0x5613488b0820 2024-09-11T06:38:50.408 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:50.404+0000 7fb45318f640 -1 reset not still connected to 0x5613488b0d00 2024-09-11T06:38:50.408 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:50.404+0000 7fb45318f640 -1 reset not still connected to 0x56134d76fe10 2024-09-11T06:38:50.408 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:50.404+0000 7fb45318f640 -1 reset not still connected to 0x56134d81fd40 2024-09-11T06:38:50.410 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:50.408+0000 7fb45318f640 -1 reset not still connected to 0x561354d57040 2024-09-11T06:38:50.410 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:50.408+0000 7fb45318f640 -1 reset not still connected to 0x561354d77110 2024-09-11T06:38:50.410 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:50.408+0000 7fb45318f640 -1 reset not still connected to 0x561355040c30 2024-09-11T06:38:50.410 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:50.408+0000 7fb45318f640 -1 reset not still connected to 0x5613551fd110 2024-09-11T06:38:51.549 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:38:51.548+0000 7fb2ca654640 -1 reset not still connected to 0x55ed161c2000 2024-09-11T06:38:51.550 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:38:51.548+0000 7fb2ca654640 -1 reset not still connected to 0x55ed194eaea0 2024-09-11T06:38:51.550 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:38:51.548+0000 7fb2ca654640 -1 reset not still connected to 0x55ed1bd1add0 2024-09-11T06:38:51.551 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:38:51.548+0000 7fb2ca654640 -1 reset not still connected to 0x55ed2a1e0ea0 2024-09-11T06:38:51.551 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:38:51.548+0000 7fb2ca654640 -1 reset not still connected to 0x55ed2b00a000 2024-09-11T06:38:51.552 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:38:51.548+0000 7fb2ca654640 -1 reset not still connected to 0x55ed38d48270 2024-09-11T06:38:51.552 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:38:51.548+0000 7fb2ca654640 -1 reset not still connected to 0x55ed3d399d40 2024-09-11T06:38:51.552 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:38:51.548+0000 7fb2ca654640 -1 reset not still connected to 0x55ed43824c30 2024-09-11T06:38:51.552 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:38:51.548+0000 7fb2ca654640 -1 reset not still connected to 0x55ed50ca3380 2024-09-11T06:38:53.802 INFO:tasks.ceph.cluster2.osd.0.smithi112.stderr:2024-09-11T06:38:53.801+0000 7f3fa0017640 -1 reset not still connected to 0x563d46606410 2024-09-11T06:38:58.865 INFO:tasks.ceph.cluster2.osd.1.smithi112.stderr:2024-09-11T06:38:58.864+0000 7f2cd8e4f640 -1 bdev(0x55da8dafac00 /var/lib/ceph/osd/cluster2-1/block) aio_submit retries 5 2024-09-11T06:38:58.871 INFO:tasks.ceph.cluster2.osd.1.smithi112.stderr:2024-09-11T06:38:58.872+0000 7f2cdce57640 -1 bdev(0x55da8dafac00 /var/lib/ceph/osd/cluster2-1/block) aio_submit retries 11 2024-09-11T06:38:59.019 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:59.016+0000 7fb45318f640 -1 reset not still connected to 0x561341f69450 2024-09-11T06:38:59.026 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:59.020+0000 7fb45318f640 -1 reset not still connected to 0x5613480e41a0 2024-09-11T06:38:59.026 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:59.020+0000 7fb45318f640 -1 reset not still connected to 0x5613488b0820 2024-09-11T06:38:59.026 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:59.020+0000 7fb45318f640 -1 reset not still connected to 0x5613488b0d00 2024-09-11T06:38:59.026 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:59.020+0000 7fb45318f640 -1 reset not still connected to 0x56134d76fe10 2024-09-11T06:38:59.026 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:59.020+0000 7fb45318f640 -1 reset not still connected to 0x56134d81fd40 2024-09-11T06:38:59.027 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:59.020+0000 7fb45318f640 -1 reset not still connected to 0x561354d57040 2024-09-11T06:38:59.027 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:59.020+0000 7fb45318f640 -1 reset not still connected to 0x561354d77110 2024-09-11T06:38:59.027 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:59.020+0000 7fb45318f640 -1 reset not still connected to 0x561355040c30 2024-09-11T06:38:59.027 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:38:59.020+0000 7fb45318f640 -1 reset not still connected to 0x5613551fd110 2024-09-11T06:38:59.086 INFO:tasks.ceph.cluster2.osd.0.smithi112.stderr:2024-09-11T06:38:59.080+0000 7f3fa0017640 -1 reset not still connected to 0x563d1fa34a90 2024-09-11T06:38:59.087 INFO:tasks.ceph.cluster2.osd.0.smithi112.stderr:2024-09-11T06:38:59.080+0000 7f3fa0017640 -1 reset not still connected to 0x563d25dba8f0 2024-09-11T06:38:59.087 INFO:tasks.ceph.cluster2.osd.0.smithi112.stderr:2024-09-11T06:38:59.088+0000 7f3fa0017640 -1 reset not still connected to 0x563d2e638dd0 2024-09-11T06:38:59.088 INFO:tasks.ceph.cluster2.osd.0.smithi112.stderr:2024-09-11T06:38:59.088+0000 7f3fa0017640 -1 reset not still connected to 0x563d2fa8a1a0 2024-09-11T06:38:59.092 INFO:tasks.ceph.cluster2.osd.0.smithi112.stderr:2024-09-11T06:38:59.092+0000 7f3fa0017640 -1 reset not still connected to 0x563d35f7c1a0 2024-09-11T06:38:59.093 INFO:tasks.ceph.cluster2.osd.0.smithi112.stderr:2024-09-11T06:38:59.092+0000 7f3fa0017640 -1 reset not still connected to 0x563d35fe6750 2024-09-11T06:38:59.093 INFO:tasks.ceph.cluster2.osd.0.smithi112.stderr:2024-09-11T06:38:59.092+0000 7f3fa0017640 -1 reset not still connected to 0x563d44f31a00 2024-09-11T06:38:59.093 INFO:tasks.ceph.cluster2.osd.0.smithi112.stderr:2024-09-11T06:38:59.092+0000 7f3fa0017640 -1 reset not still connected to 0x563d4633c1a0 2024-09-11T06:38:59.093 INFO:tasks.ceph.cluster2.osd.0.smithi112.stderr:2024-09-11T06:38:59.092+0000 7f3fa0017640 -1 reset not still connected to 0x563d465ff380 2024-09-11T06:38:59.093 INFO:tasks.ceph.cluster2.osd.0.smithi112.stderr:2024-09-11T06:38:59.092+0000 7f3fa0017640 -1 reset not still connected to 0x563d48a57e10 2024-09-11T06:39:03.451 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:03.444+0000 7fb2ca654640 -1 reset not still connected to 0x55ed0d8cdee0 2024-09-11T06:39:03.451 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:03.444+0000 7fb2ca654640 -1 reset not still connected to 0x55ed0e363ee0 2024-09-11T06:39:03.451 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:03.448+0000 7fb2ca654640 -1 reset not still connected to 0x55ed0e47ba00 2024-09-11T06:39:03.452 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:03.448+0000 7fb2ca654640 -1 reset not still connected to 0x55ed0fd8cf70 2024-09-11T06:39:03.452 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:03.448+0000 7fb2ca654640 -1 reset not still connected to 0x55ed11f509c0 2024-09-11T06:39:03.452 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:03.448+0000 7fb2ca654640 -1 reset not still connected to 0x55ed128edba0 2024-09-11T06:39:03.452 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:03.448+0000 7fb2ca654640 -1 reset not still connected to 0x55ed15c15ad0 2024-09-11T06:39:03.465 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:03.460+0000 7fb2ca654640 -1 reset not still connected to 0x55ed15c36d00 2024-09-11T06:39:03.465 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:03.460+0000 7fb2ca654640 -1 reset not still connected to 0x55ed169d91e0 2024-09-11T06:39:03.465 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:03.460+0000 7fb2ca654640 -1 reset not still connected to 0x55ed1b91f450 2024-09-11T06:39:03.465 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:03.460+0000 7fb2ca654640 -1 reset not still connected to 0x55ed1c2d45b0 2024-09-11T06:39:03.465 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:03.460+0000 7fb2ca654640 -1 reset not still connected to 0x55ed288cd110 2024-09-11T06:39:03.465 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:03.460+0000 7fb2ca654640 -1 reset not still connected to 0x55ed2b074410 2024-09-11T06:39:03.465 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:03.460+0000 7fb2ca654640 -1 reset not still connected to 0x55ed32b6bba0 2024-09-11T06:39:03.465 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:03.460+0000 7fb2ca654640 -1 reset not still connected to 0x55ed32ea5110 2024-09-11T06:39:03.465 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:03.460+0000 7fb2ca654640 -1 reset not still connected to 0x55ed369ccc30 2024-09-11T06:39:03.466 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:03.460+0000 7fb2ca654640 -1 reset not still connected to 0x55ed3a812820 2024-09-11T06:39:03.466 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:03.460+0000 7fb2ca654640 -1 reset not still connected to 0x55ed3ecf6b60 2024-09-11T06:39:03.466 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:03.460+0000 7fb2ca654640 -1 reset not still connected to 0x55ed3ecf76c0 2024-09-11T06:39:03.466 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:03.460+0000 7fb2ca654640 -1 reset not still connected to 0x55ed43611450 2024-09-11T06:39:03.466 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:03.460+0000 7fb2ca654640 -1 reset not still connected to 0x55ed436c8750 2024-09-11T06:39:03.466 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:03.460+0000 7fb2ca654640 -1 reset not still connected to 0x55ed44536340 2024-09-11T06:39:03.466 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:03.460+0000 7fb2ca654640 -1 reset not still connected to 0x55ed44538a90 2024-09-11T06:39:03.466 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:03.460+0000 7fb2ca654640 -1 reset not still connected to 0x55ed44538dd0 2024-09-11T06:39:03.466 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:03.460+0000 7fb2ca654640 -1 reset not still connected to 0x55ed4b336c30 2024-09-11T06:39:03.466 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:03.460+0000 7fb2ca654640 -1 reset not still connected to 0x55ed4b37ca90 2024-09-11T06:39:03.977 INFO:tasks.ceph.cluster2.osd.0.smithi112.stderr:2024-09-11T06:39:03.976+0000 7f3fa0017640 -1 reset not still connected to 0x563d31a47e10 2024-09-11T06:39:03.977 INFO:tasks.ceph.cluster2.osd.0.smithi112.stderr:2024-09-11T06:39:03.976+0000 7f3fa0017640 -1 reset not still connected to 0x563d332f71e0 2024-09-11T06:39:03.977 INFO:tasks.ceph.cluster2.osd.0.smithi112.stderr:2024-09-11T06:39:03.976+0000 7f3fa0017640 -1 reset not still connected to 0x563d394c8ea0 2024-09-11T06:39:03.977 INFO:tasks.ceph.cluster2.osd.0.smithi112.stderr:2024-09-11T06:39:03.976+0000 7f3fa0017640 -1 reset not still connected to 0x563d407f7ad0 2024-09-11T06:39:03.977 INFO:tasks.ceph.cluster2.osd.0.smithi112.stderr:2024-09-11T06:39:03.976+0000 7f3fa0017640 -1 reset not still connected to 0x563d4587c9c0 2024-09-11T06:39:03.977 INFO:tasks.ceph.cluster2.osd.0.smithi112.stderr:2024-09-11T06:39:03.976+0000 7f3fa0017640 -1 reset not still connected to 0x563d4619c8f0 2024-09-11T06:39:03.978 INFO:tasks.ceph.cluster2.osd.0.smithi112.stderr:2024-09-11T06:39:03.980+0000 7f3fa0017640 -1 reset not still connected to 0x563d46606410 2024-09-11T06:39:03.978 INFO:tasks.ceph.cluster2.osd.0.smithi112.stderr:2024-09-11T06:39:03.980+0000 7f3fa0017640 -1 reset not still connected to 0x563d48757a00 2024-09-11T06:39:06.114 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:06.112+0000 7fb45318f640 -1 reset not still connected to 0x561315ba5520 2024-09-11T06:39:06.116 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:06.112+0000 7fb45318f640 -1 reset not still connected to 0x56131a1ffe10 2024-09-11T06:39:06.116 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:06.112+0000 7fb45318f640 -1 reset not still connected to 0x561322931860 2024-09-11T06:39:06.119 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:06.116+0000 7fb45318f640 -1 reset not still connected to 0x561322ded930 2024-09-11T06:39:06.119 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:06.116+0000 7fb45318f640 -1 reset not still connected to 0x561330ed0410 2024-09-11T06:39:06.122 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:06.120+0000 7fb45318f640 -1 reset not still connected to 0x56133339c410 2024-09-11T06:39:06.125 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:06.124+0000 7fb45318f640 -1 reset not still connected to 0x56133c1909c0 2024-09-11T06:39:06.129 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:06.128+0000 7fb45318f640 -1 reset not still connected to 0x56133c2a2dd0 2024-09-11T06:39:06.129 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:06.128+0000 7fb45318f640 -1 reset not still connected to 0x561341f69450 2024-09-11T06:39:06.134 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:06.132+0000 7fb45318f640 -1 reset not still connected to 0x561342043930 2024-09-11T06:39:06.134 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:06.132+0000 7fb45318f640 -1 reset not still connected to 0x5613480872b0 2024-09-11T06:39:06.134 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:06.132+0000 7fb45318f640 -1 reset not still connected to 0x5613480e41a0 2024-09-11T06:39:06.134 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:06.132+0000 7fb45318f640 -1 reset not still connected to 0x5613488b0820 2024-09-11T06:39:06.134 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:06.132+0000 7fb45318f640 -1 reset not still connected to 0x5613488b0d00 2024-09-11T06:39:06.134 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:06.132+0000 7fb45318f640 -1 reset not still connected to 0x56134d76fe10 2024-09-11T06:39:06.134 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:06.132+0000 7fb45318f640 -1 reset not still connected to 0x56134d81fd40 2024-09-11T06:39:06.134 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:06.132+0000 7fb45318f640 -1 reset not still connected to 0x561354d57040 2024-09-11T06:39:06.134 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:06.132+0000 7fb45318f640 -1 reset not still connected to 0x561354d77110 2024-09-11T06:39:06.134 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:06.132+0000 7fb45318f640 -1 reset not still connected to 0x561355040c30 2024-09-11T06:39:06.135 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:06.132+0000 7fb45318f640 -1 reset not still connected to 0x5613551fd110 2024-09-11T06:39:06.135 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:06.132+0000 7fb45318f640 -1 reset not still connected to 0x5613557c4680 2024-09-11T06:39:08.113 INFO:tasks.ceph.cluster2.osd.0.smithi112.stderr:2024-09-11T06:39:08.116+0000 7f3fa0017640 -1 reset not still connected to 0x563d332f71e0 2024-09-11T06:39:08.114 INFO:tasks.ceph.cluster2.osd.0.smithi112.stderr:2024-09-11T06:39:08.116+0000 7f3fa0017640 -1 reset not still connected to 0x563d394c8ea0 2024-09-11T06:39:08.114 INFO:tasks.ceph.cluster2.osd.0.smithi112.stderr:2024-09-11T06:39:08.116+0000 7f3fa0017640 -1 reset not still connected to 0x563d407f7ad0 2024-09-11T06:39:08.114 INFO:tasks.ceph.cluster2.osd.0.smithi112.stderr:2024-09-11T06:39:08.116+0000 7f3fa0017640 -1 reset not still connected to 0x563d4619c8f0 2024-09-11T06:39:08.114 INFO:tasks.ceph.cluster2.osd.0.smithi112.stderr:2024-09-11T06:39:08.116+0000 7f3fa0017640 -1 reset not still connected to 0x563d46606410 2024-09-11T06:39:08.114 INFO:tasks.ceph.cluster2.osd.0.smithi112.stderr:2024-09-11T06:39:08.116+0000 7f3fa0017640 -1 reset not still connected to 0x563d48757a00 2024-09-11T06:39:11.012 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:39:11.008+0000 7fc41084f640 -1 reset not still connected to 0x55eb3893ac30 2024-09-11T06:39:11.012 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:39:11.008+0000 7fc41084f640 -1 reset not still connected to 0x55eb3add8270 2024-09-11T06:39:11.015 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:39:11.012+0000 7fc41084f640 -1 reset not still connected to 0x55eb4f1329c0 2024-09-11T06:39:11.015 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:39:11.012+0000 7fc41084f640 -1 reset not still connected to 0x55eb53d56f70 2024-09-11T06:39:11.913 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:11.908+0000 7fb45318f640 -1 reset not still connected to 0x561322931860 2024-09-11T06:39:11.913 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:11.908+0000 7fb45318f640 -1 reset not still connected to 0x561322ded930 2024-09-11T06:39:11.913 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:11.908+0000 7fb45318f640 -1 reset not still connected to 0x56132c963520 2024-09-11T06:39:11.913 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:11.908+0000 7fb45318f640 -1 reset not still connected to 0x561330ed0410 2024-09-11T06:39:11.913 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:11.908+0000 7fb45318f640 -1 reset not still connected to 0x56133339c410 2024-09-11T06:39:11.913 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:11.908+0000 7fb45318f640 -1 reset not still connected to 0x56133c1909c0 2024-09-11T06:39:11.914 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:11.912+0000 7fb45318f640 -1 reset not still connected to 0x56133c2a2dd0 2024-09-11T06:39:11.914 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:11.912+0000 7fb45318f640 -1 reset not still connected to 0x561342043930 2024-09-11T06:39:11.914 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:11.912+0000 7fb45318f640 -1 reset not still connected to 0x5613480872b0 2024-09-11T06:39:11.922 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:11.920+0000 7fb45318f640 -1 reset not still connected to 0x5613488b0820 2024-09-11T06:39:11.923 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:11.920+0000 7fb45318f640 -1 reset not still connected to 0x5613488b0d00 2024-09-11T06:39:11.923 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:11.920+0000 7fb45318f640 -1 reset not still connected to 0x56134d76fe10 2024-09-11T06:39:11.923 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:11.920+0000 7fb45318f640 -1 reset not still connected to 0x56134d81fd40 2024-09-11T06:39:11.923 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:11.920+0000 7fb45318f640 -1 reset not still connected to 0x561354d57040 2024-09-11T06:39:11.923 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:11.920+0000 7fb45318f640 -1 reset not still connected to 0x561354d77110 2024-09-11T06:39:11.923 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:11.920+0000 7fb45318f640 -1 reset not still connected to 0x5613551fd110 2024-09-11T06:39:11.923 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:11.920+0000 7fb45318f640 -1 reset not still connected to 0x5613557c4680 2024-09-11T06:39:13.068 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:13.064+0000 7fb45318f640 -1 reset not still connected to 0x56134d76fe10 2024-09-11T06:39:13.068 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:13.064+0000 7fb45318f640 -1 reset not still connected to 0x56134d81fd40 2024-09-11T06:39:13.068 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:13.064+0000 7fb45318f640 -1 reset not still connected to 0x561354d77110 2024-09-11T06:39:13.069 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:13.068+0000 7fb45318f640 -1 reset not still connected to 0x561355040c30 2024-09-11T06:39:13.071 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:13.068+0000 7fb45318f640 -1 reset not still connected to 0x5613551fd110 2024-09-11T06:39:14.373 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 -p mirror mirror pool status 2024-09-11T06:39:14.373 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut '-d ' -f 3 2024-09-11T06:39:14.373 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep 'image health:' 2024-09-11T06:39:14.751 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: failed to query services: (13) Permission denied 2024-09-11T06:39:14.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ state=WARNING 2024-09-11T06:39:14.756 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test WARNING = ERROR 2024-09-11T06:39:14.756 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test WARNING = OK 2024-09-11T06:39:14.756 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in `seq 1 40` 2024-09-11T06:39:14.756 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 3 -ne 1 2024-09-11T06:39:14.756 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 30 2024-09-11T06:39:14.884 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:39:14.880+0000 7fc41084f640 -1 reset not still connected to 0x55eb15565d40 2024-09-11T06:39:14.884 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:39:14.880+0000 7fc41084f640 -1 reset not still connected to 0x55eb15806dd0 2024-09-11T06:39:14.884 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:39:14.880+0000 7fc41084f640 -1 reset not still connected to 0x55eb16561450 2024-09-11T06:39:14.884 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:39:14.880+0000 7fc41084f640 -1 reset not still connected to 0x55eb1afbe4e0 2024-09-11T06:39:14.884 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:39:14.880+0000 7fc41084f640 -1 reset not still connected to 0x55eb1fefeb60 2024-09-11T06:39:14.885 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:39:14.880+0000 7fc41084f640 -1 reset not still connected to 0x55eb2458e680 2024-09-11T06:39:14.885 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:39:14.880+0000 7fc41084f640 -1 reset not still connected to 0x55eb24ad96c0 2024-09-11T06:39:14.885 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:39:14.880+0000 7fc41084f640 -1 reset not still connected to 0x55eb2601e820 2024-09-11T06:39:14.885 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:39:14.880+0000 7fc41084f640 -1 reset not still connected to 0x55eb2613e680 2024-09-11T06:39:14.885 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:39:14.880+0000 7fc41084f640 -1 reset not still connected to 0x55eb2da77ba0 2024-09-11T06:39:14.885 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:39:14.880+0000 7fc41084f640 -1 reset not still connected to 0x55eb34898a90 2024-09-11T06:39:14.885 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:39:14.880+0000 7fc41084f640 -1 reset not still connected to 0x55eb35243040 2024-09-11T06:39:14.885 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:39:14.880+0000 7fc41084f640 -1 reset not still connected to 0x55eb354a21a0 2024-09-11T06:39:14.887 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:39:14.884+0000 7fc41084f640 -1 reset not still connected to 0x55eb3893ac30 2024-09-11T06:39:14.889 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:39:14.884+0000 7fc41084f640 -1 reset not still connected to 0x55eb3add8270 2024-09-11T06:39:14.889 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:39:14.884+0000 7fc41084f640 -1 reset not still connected to 0x55eb4480b930 2024-09-11T06:39:14.889 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:39:14.888+0000 7fc41084f640 -1 reset not still connected to 0x55eb4f1329c0 2024-09-11T06:39:14.889 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:39:14.888+0000 7fc41084f640 -1 reset not still connected to 0x55eb53d56f70 2024-09-11T06:39:16.063 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:39:16.068+0000 7efd718c5640 -1 reset not still connected to 0x560b2f4745b0 2024-09-11T06:39:16.063 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:39:16.068+0000 7efd718c5640 -1 reset not still connected to 0x560b2fc524e0 2024-09-11T06:39:16.063 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:39:16.068+0000 7efd718c5640 -1 reset not still connected to 0x560b34cafee0 2024-09-11T06:39:16.063 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:39:16.068+0000 7efd718c5640 -1 reset not still connected to 0x560b37798410 2024-09-11T06:39:16.893 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:39:16.888+0000 7fc41084f640 -1 reset not still connected to 0x55eb2613e680 2024-09-11T06:39:16.897 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:39:16.892+0000 7fc41084f640 -1 reset not still connected to 0x55eb2da77ba0 2024-09-11T06:39:16.897 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:39:16.892+0000 7fc41084f640 -1 reset not still connected to 0x55eb345e5ad0 2024-09-11T06:39:16.897 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:39:16.892+0000 7fc41084f640 -1 reset not still connected to 0x55eb34898a90 2024-09-11T06:39:16.897 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:39:16.892+0000 7fc41084f640 -1 reset not still connected to 0x55eb354a21a0 2024-09-11T06:39:16.897 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:39:16.892+0000 7fc41084f640 -1 reset not still connected to 0x55eb3893ac30 2024-09-11T06:39:16.897 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:39:16.896+0000 7fc41084f640 -1 reset not still connected to 0x55eb4480b930 2024-09-11T06:39:16.897 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:39:16.896+0000 7fc41084f640 -1 reset not still connected to 0x55eb4f1329c0 2024-09-11T06:39:16.897 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:39:16.896+0000 7fc41084f640 -1 reset not still connected to 0x55eb53d56f70 2024-09-11T06:39:19.772 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:39:19.768+0000 7fc41084f640 -1 reset not still connected to 0x55eb43fe4340 2024-09-11T06:39:19.772 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:39:19.768+0000 7fc41084f640 -1 reset not still connected to 0x55eb4464a410 2024-09-11T06:39:20.039 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:20.028+0000 7fb2ca654640 -1 reset not still connected to 0x55ed43611450 2024-09-11T06:39:20.040 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:20.032+0000 7fb2ca654640 -1 reset not still connected to 0x55ed44536340 2024-09-11T06:39:20.040 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:20.032+0000 7fb2ca654640 -1 reset not still connected to 0x55ed4b336c30 2024-09-11T06:39:21.071 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:39:21.068+0000 7fc41084f640 -1 reset not still connected to 0x55eb16561450 2024-09-11T06:39:21.073 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:39:21.072+0000 7fc41084f640 -1 reset not still connected to 0x55eb1afbe4e0 2024-09-11T06:39:21.074 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:39:21.072+0000 7fc41084f640 -1 reset not still connected to 0x55eb1fefeb60 2024-09-11T06:39:21.074 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:39:21.072+0000 7fc41084f640 -1 reset not still connected to 0x55eb2458e680 2024-09-11T06:39:21.075 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:39:21.072+0000 7fc41084f640 -1 reset not still connected to 0x55eb24ad96c0 2024-09-11T06:39:21.078 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:39:21.076+0000 7fc41084f640 -1 reset not still connected to 0x55eb2601e820 2024-09-11T06:39:21.080 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:39:21.076+0000 7fc41084f640 -1 reset not still connected to 0x55eb2613e680 2024-09-11T06:39:21.081 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:39:21.076+0000 7fc41084f640 -1 reset not still connected to 0x55eb2da77ba0 2024-09-11T06:39:21.081 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:39:21.076+0000 7fc41084f640 -1 reset not still connected to 0x55eb345e5ad0 2024-09-11T06:39:21.082 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:39:21.076+0000 7fc41084f640 -1 reset not still connected to 0x55eb34898a90 2024-09-11T06:39:21.088 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:39:21.080+0000 7fc41084f640 -1 reset not still connected to 0x55eb35243040 2024-09-11T06:39:21.088 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:39:21.080+0000 7fc41084f640 -1 reset not still connected to 0x55eb354a21a0 2024-09-11T06:39:21.088 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:39:21.080+0000 7fc41084f640 -1 reset not still connected to 0x55eb3893ac30 2024-09-11T06:39:21.088 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:39:21.080+0000 7fc41084f640 -1 reset not still connected to 0x55eb3add8270 2024-09-11T06:39:21.088 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:39:21.080+0000 7fc41084f640 -1 reset not still connected to 0x55eb4480b930 2024-09-11T06:39:21.088 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:39:21.080+0000 7fc41084f640 -1 reset not still connected to 0x55eb4f1329c0 2024-09-11T06:39:21.088 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:39:21.080+0000 7fc41084f640 -1 reset not still connected to 0x55eb53d56f70 2024-09-11T06:39:22.887 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:22.884+0000 7fb45318f640 -1 reset not still connected to 0x5613488b0820 2024-09-11T06:39:22.897 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:22.892+0000 7fb45318f640 -1 reset not still connected to 0x56134d76fe10 2024-09-11T06:39:22.897 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:22.892+0000 7fb45318f640 -1 reset not still connected to 0x56134d81fd40 2024-09-11T06:39:22.902 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:22.900+0000 7fb45318f640 -1 reset not still connected to 0x561354d57040 2024-09-11T06:39:22.908 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:22.904+0000 7fb45318f640 -1 reset not still connected to 0x561354d77110 2024-09-11T06:39:22.908 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:22.904+0000 7fb45318f640 -1 reset not still connected to 0x561355040c30 2024-09-11T06:39:22.909 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:22.904+0000 7fb45318f640 -1 reset not still connected to 0x5613551fd110 2024-09-11T06:39:22.909 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:22.904+0000 7fb45318f640 -1 reset not still connected to 0x5613557c4680 2024-09-11T06:39:23.002 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:22.992+0000 7fb2ca654640 -1 reset not still connected to 0x55ed4b336c30 2024-09-11T06:39:23.273 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:23.272+0000 7fb2ca654640 -1 reset not still connected to 0x55ed0d8cd5f0 2024-09-11T06:39:23.273 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:23.272+0000 7fb2ca654640 -1 reset not still connected to 0x55ed0e550dd0 2024-09-11T06:39:23.277 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:23.272+0000 7fb2ca654640 -1 reset not still connected to 0x55ed12570820 2024-09-11T06:39:23.277 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:23.272+0000 7fb2ca654640 -1 reset not still connected to 0x55ed149fe8f0 2024-09-11T06:39:23.277 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:23.272+0000 7fb2ca654640 -1 reset not still connected to 0x55ed161c2000 2024-09-11T06:39:23.277 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:23.272+0000 7fb2ca654640 -1 reset not still connected to 0x55ed194eaea0 2024-09-11T06:39:23.277 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:23.272+0000 7fb2ca654640 -1 reset not still connected to 0x55ed1bd1add0 2024-09-11T06:39:23.277 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:23.272+0000 7fb2ca654640 -1 reset not still connected to 0x55ed2a1e0ea0 2024-09-11T06:39:23.277 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:23.272+0000 7fb2ca654640 -1 reset not still connected to 0x55ed2b00a000 2024-09-11T06:39:23.277 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:23.272+0000 7fb2ca654640 -1 reset not still connected to 0x55ed38d48270 2024-09-11T06:39:23.277 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:23.272+0000 7fb2ca654640 -1 reset not still connected to 0x55ed3d399d40 2024-09-11T06:39:23.277 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:23.272+0000 7fb2ca654640 -1 reset not still connected to 0x55ed43824c30 2024-09-11T06:39:23.277 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:23.272+0000 7fb2ca654640 -1 reset not still connected to 0x55ed50ca3380 2024-09-11T06:39:23.545 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:23.532+0000 7fb2ca654640 -1 reset not still connected to 0x55ed149fe8f0 2024-09-11T06:39:23.546 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:23.532+0000 7fb2ca654640 -1 reset not still connected to 0x55ed161c2000 2024-09-11T06:39:23.546 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:23.532+0000 7fb2ca654640 -1 reset not still connected to 0x55ed194eaea0 2024-09-11T06:39:23.547 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:23.540+0000 7fb2ca654640 -1 reset not still connected to 0x55ed2a1e0ea0 2024-09-11T06:39:23.552 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:23.548+0000 7fb2ca654640 -1 reset not still connected to 0x55ed50ca3380 2024-09-11T06:39:23.765 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:23.764+0000 7fb45318f640 -1 reset not still connected to 0x561315ba5520 2024-09-11T06:39:23.769 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:23.768+0000 7fb45318f640 -1 reset not still connected to 0x56131a1ffe10 2024-09-11T06:39:23.777 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:23.768+0000 7fb45318f640 -1 reset not still connected to 0x561322931860 2024-09-11T06:39:23.779 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:23.768+0000 7fb45318f640 -1 reset not still connected to 0x561322ded930 2024-09-11T06:39:23.779 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:23.768+0000 7fb45318f640 -1 reset not still connected to 0x561330ed0410 2024-09-11T06:39:23.779 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:23.768+0000 7fb45318f640 -1 reset not still connected to 0x56133339c410 2024-09-11T06:39:23.779 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:23.772+0000 7fb45318f640 -1 reset not still connected to 0x56133c1909c0 2024-09-11T06:39:23.783 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:23.780+0000 7fb45318f640 -1 reset not still connected to 0x561341f69450 2024-09-11T06:39:23.785 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:23.784+0000 7fb45318f640 -1 reset not still connected to 0x561342043930 2024-09-11T06:39:23.789 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:23.784+0000 7fb45318f640 -1 reset not still connected to 0x5613480872b0 2024-09-11T06:39:23.792 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:23.788+0000 7fb45318f640 -1 reset not still connected to 0x5613480e41a0 2024-09-11T06:39:23.794 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:23.788+0000 7fb45318f640 -1 reset not still connected to 0x5613488b0820 2024-09-11T06:39:23.794 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:23.788+0000 7fb45318f640 -1 reset not still connected to 0x5613488b0d00 2024-09-11T06:39:23.794 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:23.788+0000 7fb45318f640 -1 reset not still connected to 0x56134d76fe10 2024-09-11T06:39:23.794 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:23.788+0000 7fb45318f640 -1 reset not still connected to 0x56134d81fd40 2024-09-11T06:39:23.795 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:23.792+0000 7fb45318f640 -1 reset not still connected to 0x561354d57040 2024-09-11T06:39:23.795 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:23.792+0000 7fb45318f640 -1 reset not still connected to 0x561354d77110 2024-09-11T06:39:23.795 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:23.792+0000 7fb45318f640 -1 reset not still connected to 0x561355040c30 2024-09-11T06:39:23.795 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:23.792+0000 7fb45318f640 -1 reset not still connected to 0x5613551fd110 2024-09-11T06:39:23.795 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:23.792+0000 7fb45318f640 -1 reset not still connected to 0x5613557c4680 2024-09-11T06:39:24.779 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:24.772+0000 7fb2ca654640 -1 reset not still connected to 0x55ed44536340 2024-09-11T06:39:24.788 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:24.784+0000 7fb2ca654640 -1 reset not still connected to 0x55ed44538dd0 2024-09-11T06:39:24.788 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:24.784+0000 7fb2ca654640 -1 reset not still connected to 0x55ed4b336c30 2024-09-11T06:39:24.788 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:24.784+0000 7fb2ca654640 -1 reset not still connected to 0x55ed4b37ca90 2024-09-11T06:39:26.267 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:26.264+0000 7fb45318f640 -1 reset not still connected to 0x56131a1ffe10 2024-09-11T06:39:26.267 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:26.264+0000 7fb45318f640 -1 reset not still connected to 0x561322931860 2024-09-11T06:39:26.271 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:26.268+0000 7fb45318f640 -1 reset not still connected to 0x561322ded930 2024-09-11T06:39:26.272 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:26.268+0000 7fb45318f640 -1 reset not still connected to 0x561330ed0410 2024-09-11T06:39:26.273 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:26.272+0000 7fb45318f640 -1 reset not still connected to 0x56133339c410 2024-09-11T06:39:26.275 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:26.272+0000 7fb45318f640 -1 reset not still connected to 0x56133c1909c0 2024-09-11T06:39:26.282 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:26.280+0000 7fb45318f640 -1 reset not still connected to 0x561341f69450 2024-09-11T06:39:26.282 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:26.280+0000 7fb45318f640 -1 reset not still connected to 0x561342043930 2024-09-11T06:39:26.289 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:26.284+0000 7fb45318f640 -1 reset not still connected to 0x5613480872b0 2024-09-11T06:39:26.289 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:26.284+0000 7fb45318f640 -1 reset not still connected to 0x5613480e41a0 2024-09-11T06:39:26.289 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:26.284+0000 7fb45318f640 -1 reset not still connected to 0x5613488b0820 2024-09-11T06:39:26.289 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:26.284+0000 7fb45318f640 -1 reset not still connected to 0x5613488b0d00 2024-09-11T06:39:26.289 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:26.284+0000 7fb45318f640 -1 reset not still connected to 0x56134d76fe10 2024-09-11T06:39:26.289 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:26.284+0000 7fb45318f640 -1 reset not still connected to 0x56134d81fd40 2024-09-11T06:39:26.289 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:26.284+0000 7fb45318f640 -1 reset not still connected to 0x561354d57040 2024-09-11T06:39:26.289 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:26.284+0000 7fb45318f640 -1 reset not still connected to 0x561354d77110 2024-09-11T06:39:26.289 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:26.284+0000 7fb45318f640 -1 reset not still connected to 0x561355040c30 2024-09-11T06:39:26.289 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:26.284+0000 7fb45318f640 -1 reset not still connected to 0x5613551fd110 2024-09-11T06:39:27.355 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:27.344+0000 7fb45318f640 -1 reset not still connected to 0x56131a1ffe10 2024-09-11T06:39:27.356 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:27.344+0000 7fb45318f640 -1 reset not still connected to 0x561322931860 2024-09-11T06:39:27.356 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:27.348+0000 7fb45318f640 -1 reset not still connected to 0x561322ded930 2024-09-11T06:39:27.356 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:27.348+0000 7fb45318f640 -1 reset not still connected to 0x561330ed0410 2024-09-11T06:39:27.356 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:27.348+0000 7fb45318f640 -1 reset not still connected to 0x56133c1909c0 2024-09-11T06:39:27.357 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:27.348+0000 7fb45318f640 -1 reset not still connected to 0x561341f69450 2024-09-11T06:39:27.357 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:27.352+0000 7fb45318f640 -1 reset not still connected to 0x561342043930 2024-09-11T06:39:27.357 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:27.356+0000 7fb45318f640 -1 reset not still connected to 0x5613480e41a0 2024-09-11T06:39:27.357 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:27.356+0000 7fb45318f640 -1 reset not still connected to 0x5613488b0820 2024-09-11T06:39:27.357 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:27.356+0000 7fb45318f640 -1 reset not still connected to 0x5613488b0d00 2024-09-11T06:39:27.362 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:27.360+0000 7fb45318f640 -1 reset not still connected to 0x56134d76fe10 2024-09-11T06:39:27.362 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:27.360+0000 7fb45318f640 -1 reset not still connected to 0x56134d81fd40 2024-09-11T06:39:27.362 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:27.360+0000 7fb45318f640 -1 reset not still connected to 0x561354d77110 2024-09-11T06:39:27.362 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:27.360+0000 7fb45318f640 -1 reset not still connected to 0x561355040c30 2024-09-11T06:39:27.372 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:27.368+0000 7fb45318f640 -1 reset not still connected to 0x5613551fd110 2024-09-11T06:39:27.497 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:27.496+0000 7fb2ca654640 -1 reset not still connected to 0x55ed0d863040 2024-09-11T06:39:27.499 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:27.496+0000 7fb2ca654640 -1 reset not still connected to 0x55ed0d865e10 2024-09-11T06:39:27.499 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:27.496+0000 7fb2ca654640 -1 reset not still connected to 0x55ed0d8cd5f0 2024-09-11T06:39:27.499 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:27.496+0000 7fb2ca654640 -1 reset not still connected to 0x55ed12570820 2024-09-11T06:39:27.499 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:27.496+0000 7fb2ca654640 -1 reset not still connected to 0x55ed149fe8f0 2024-09-11T06:39:27.499 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:27.496+0000 7fb2ca654640 -1 reset not still connected to 0x55ed161c2000 2024-09-11T06:39:27.499 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:27.496+0000 7fb2ca654640 -1 reset not still connected to 0x55ed194eaea0 2024-09-11T06:39:27.499 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:27.496+0000 7fb2ca654640 -1 reset not still connected to 0x55ed1bd1add0 2024-09-11T06:39:27.502 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:27.500+0000 7fb2ca654640 -1 reset not still connected to 0x55ed38d48270 2024-09-11T06:39:27.502 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:27.500+0000 7fb2ca654640 -1 reset not still connected to 0x55ed3d399d40 2024-09-11T06:39:27.510 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:27.504+0000 7fb2ca654640 -1 reset not still connected to 0x55ed43824c30 2024-09-11T06:39:27.510 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:27.504+0000 7fb2ca654640 -1 reset not still connected to 0x55ed50ca3380 2024-09-11T06:39:27.889 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:27.884+0000 7fb2ca654640 -1 reset not still connected to 0x55ed0d8655f0 2024-09-11T06:39:27.889 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:27.884+0000 7fb2ca654640 -1 reset not still connected to 0x55ed0d8cdee0 2024-09-11T06:39:27.889 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:27.884+0000 7fb2ca654640 -1 reset not still connected to 0x55ed0e363ee0 2024-09-11T06:39:27.893 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:27.888+0000 7fb2ca654640 -1 reset not still connected to 0x55ed0e47ba00 2024-09-11T06:39:27.893 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:27.888+0000 7fb2ca654640 -1 reset not still connected to 0x55ed11f509c0 2024-09-11T06:39:27.893 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:27.888+0000 7fb2ca654640 -1 reset not still connected to 0x55ed128edba0 2024-09-11T06:39:27.893 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:27.888+0000 7fb2ca654640 -1 reset not still connected to 0x55ed15c15ad0 2024-09-11T06:39:27.893 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:27.888+0000 7fb2ca654640 -1 reset not still connected to 0x55ed15c36d00 2024-09-11T06:39:27.893 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:27.888+0000 7fb2ca654640 -1 reset not still connected to 0x55ed169d91e0 2024-09-11T06:39:27.893 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:27.888+0000 7fb2ca654640 -1 reset not still connected to 0x55ed1b91f450 2024-09-11T06:39:27.906 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:27.904+0000 7fb2ca654640 -1 reset not still connected to 0x55ed1c2d45b0 2024-09-11T06:39:27.906 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:27.904+0000 7fb2ca654640 -1 reset not still connected to 0x55ed288cd110 2024-09-11T06:39:27.907 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:27.904+0000 7fb2ca654640 -1 reset not still connected to 0x55ed2b074410 2024-09-11T06:39:27.907 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:27.904+0000 7fb2ca654640 -1 reset not still connected to 0x55ed32b6bba0 2024-09-11T06:39:27.907 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:27.904+0000 7fb2ca654640 -1 reset not still connected to 0x55ed32ea5110 2024-09-11T06:39:27.907 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:27.904+0000 7fb2ca654640 -1 reset not still connected to 0x55ed369ccc30 2024-09-11T06:39:27.907 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:27.904+0000 7fb2ca654640 -1 reset not still connected to 0x55ed3a812820 2024-09-11T06:39:27.914 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:27.912+0000 7fb2ca654640 -1 reset not still connected to 0x55ed3ecf6b60 2024-09-11T06:39:27.915 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:27.912+0000 7fb2ca654640 -1 reset not still connected to 0x55ed3ecf76c0 2024-09-11T06:39:27.915 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:27.912+0000 7fb2ca654640 -1 reset not still connected to 0x55ed43611450 2024-09-11T06:39:27.915 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:27.912+0000 7fb2ca654640 -1 reset not still connected to 0x55ed436c8750 2024-09-11T06:39:27.915 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:27.912+0000 7fb2ca654640 -1 reset not still connected to 0x55ed44536340 2024-09-11T06:39:27.915 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:27.912+0000 7fb2ca654640 -1 reset not still connected to 0x55ed44538a90 2024-09-11T06:39:27.916 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:27.912+0000 7fb2ca654640 -1 reset not still connected to 0x55ed44538dd0 2024-09-11T06:39:27.916 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:27.912+0000 7fb2ca654640 -1 reset not still connected to 0x55ed4b336c30 2024-09-11T06:39:27.916 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:27.912+0000 7fb2ca654640 -1 reset not still connected to 0x55ed4b37ca90 2024-09-11T06:39:31.145 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:31.140+0000 7fb2ca654640 -1 reset not still connected to 0x55ed1bd1add0 2024-09-11T06:39:31.145 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:31.140+0000 7fb2ca654640 -1 reset not still connected to 0x55ed2a1e0ea0 2024-09-11T06:39:31.146 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:31.140+0000 7fb2ca654640 -1 reset not still connected to 0x55ed2b00a000 2024-09-11T06:39:31.146 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:31.140+0000 7fb2ca654640 -1 reset not still connected to 0x55ed38d48270 2024-09-11T06:39:31.146 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:31.140+0000 7fb2ca654640 -1 reset not still connected to 0x55ed3d399d40 2024-09-11T06:39:31.146 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:31.140+0000 7fb2ca654640 -1 reset not still connected to 0x55ed43824c30 2024-09-11T06:39:31.146 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:31.140+0000 7fb2ca654640 -1 reset not still connected to 0x55ed50ca3380 2024-09-11T06:39:33.946 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:33.936+0000 7fb2ca654640 -1 reset not still connected to 0x55ed0e363ee0 2024-09-11T06:39:33.953 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:33.948+0000 7fb2ca654640 -1 reset not still connected to 0x55ed15c36d00 2024-09-11T06:39:33.953 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:33.952+0000 7fb2ca654640 -1 reset not still connected to 0x55ed169d91e0 2024-09-11T06:39:33.954 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:33.952+0000 7fb2ca654640 -1 reset not still connected to 0x55ed1b91f450 2024-09-11T06:39:33.963 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:33.960+0000 7fb2ca654640 -1 reset not still connected to 0x55ed288cd110 2024-09-11T06:39:33.963 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:33.960+0000 7fb2ca654640 -1 reset not still connected to 0x55ed32b6bba0 2024-09-11T06:39:33.963 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:33.960+0000 7fb2ca654640 -1 reset not still connected to 0x55ed369ccc30 2024-09-11T06:39:33.972 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:33.968+0000 7fb2ca654640 -1 reset not still connected to 0x55ed3a812820 2024-09-11T06:39:33.972 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:33.968+0000 7fb2ca654640 -1 reset not still connected to 0x55ed43611450 2024-09-11T06:39:33.972 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:33.968+0000 7fb2ca654640 -1 reset not still connected to 0x55ed436c8750 2024-09-11T06:39:33.973 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:33.968+0000 7fb2ca654640 -1 reset not still connected to 0x55ed44536340 2024-09-11T06:39:33.985 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:33.980+0000 7fb2ca654640 -1 reset not still connected to 0x55ed44538dd0 2024-09-11T06:39:33.986 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:33.980+0000 7fb2ca654640 -1 reset not still connected to 0x55ed4b336c30 2024-09-11T06:39:33.986 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:33.980+0000 7fb2ca654640 -1 reset not still connected to 0x55ed4b37ca90 2024-09-11T06:39:37.752 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:39:37.748+0000 7fc41084f640 -1 reset not still connected to 0x55eb38da3a00 2024-09-11T06:39:37.752 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:39:37.748+0000 7fc41084f640 -1 reset not still connected to 0x55eb43fe4340 2024-09-11T06:39:37.909 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:37.900+0000 7fb45318f640 -1 reset not still connected to 0x561315ba4270 2024-09-11T06:39:37.909 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:37.900+0000 7fb45318f640 -1 reset not still connected to 0x561315ba5520 2024-09-11T06:39:37.910 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:37.904+0000 7fb45318f640 -1 reset not still connected to 0x56131a1ffe10 2024-09-11T06:39:37.910 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:37.904+0000 7fb45318f640 -1 reset not still connected to 0x561322931860 2024-09-11T06:39:37.910 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:37.904+0000 7fb45318f640 -1 reset not still connected to 0x561322ded930 2024-09-11T06:39:37.913 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:37.908+0000 7fb45318f640 -1 reset not still connected to 0x561330ed0410 2024-09-11T06:39:37.913 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:37.908+0000 7fb45318f640 -1 reset not still connected to 0x56133339c410 2024-09-11T06:39:37.930 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:37.924+0000 7fb45318f640 -1 reset not still connected to 0x56133c1909c0 2024-09-11T06:39:37.931 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:37.924+0000 7fb45318f640 -1 reset not still connected to 0x56133c2a2dd0 2024-09-11T06:39:37.931 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:37.924+0000 7fb45318f640 -1 reset not still connected to 0x561341f69450 2024-09-11T06:39:37.932 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:37.924+0000 7fb45318f640 -1 reset not still connected to 0x561342043930 2024-09-11T06:39:37.932 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:37.924+0000 7fb45318f640 -1 reset not still connected to 0x5613480872b0 2024-09-11T06:39:37.932 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:37.924+0000 7fb45318f640 -1 reset not still connected to 0x5613480e41a0 2024-09-11T06:39:37.932 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:37.924+0000 7fb45318f640 -1 reset not still connected to 0x5613488b0820 2024-09-11T06:39:37.932 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:37.924+0000 7fb45318f640 -1 reset not still connected to 0x5613488b0d00 2024-09-11T06:39:37.932 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:37.924+0000 7fb45318f640 -1 reset not still connected to 0x56134d76fe10 2024-09-11T06:39:37.932 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:37.928+0000 7fb45318f640 -1 reset not still connected to 0x56134d81fd40 2024-09-11T06:39:37.933 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:37.928+0000 7fb45318f640 -1 reset not still connected to 0x561354d57040 2024-09-11T06:39:37.933 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:37.928+0000 7fb45318f640 -1 reset not still connected to 0x561354d77110 2024-09-11T06:39:37.933 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:37.928+0000 7fb45318f640 -1 reset not still connected to 0x561355040c30 2024-09-11T06:39:37.933 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:37.928+0000 7fb45318f640 -1 reset not still connected to 0x5613551fd110 2024-09-11T06:39:37.933 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:37.928+0000 7fb45318f640 -1 reset not still connected to 0x5613557c4680 2024-09-11T06:39:38.990 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:39:38.988+0000 7fc41084f640 -1 reset not still connected to 0x55eb24ad96c0 2024-09-11T06:39:38.990 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:39:38.988+0000 7fc41084f640 -1 reset not still connected to 0x55eb2601e820 2024-09-11T06:39:38.990 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:39:38.988+0000 7fc41084f640 -1 reset not still connected to 0x55eb2613e680 2024-09-11T06:39:38.990 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:39:38.988+0000 7fc41084f640 -1 reset not still connected to 0x55eb2da77ba0 2024-09-11T06:39:38.993 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:39:38.988+0000 7fc41084f640 -1 reset not still connected to 0x55eb345e5ad0 2024-09-11T06:39:38.993 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:39:38.988+0000 7fc41084f640 -1 reset not still connected to 0x55eb34898a90 2024-09-11T06:39:38.993 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:39:38.988+0000 7fc41084f640 -1 reset not still connected to 0x55eb35243040 2024-09-11T06:39:38.995 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:39:38.992+0000 7fc41084f640 -1 reset not still connected to 0x55eb354a21a0 2024-09-11T06:39:38.995 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:39:38.992+0000 7fc41084f640 -1 reset not still connected to 0x55eb3893ac30 2024-09-11T06:39:38.995 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:39:38.992+0000 7fc41084f640 -1 reset not still connected to 0x55eb3add8270 2024-09-11T06:39:38.995 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:39:38.992+0000 7fc41084f640 -1 reset not still connected to 0x55eb4f1329c0 2024-09-11T06:39:38.995 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:39:38.992+0000 7fc41084f640 -1 reset not still connected to 0x55eb53d56f70 2024-09-11T06:39:39.898 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:39:39.899+0000 7efd718c5640 -1 reset not still connected to 0x560b15963110 2024-09-11T06:39:39.898 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:39:39.899+0000 7efd718c5640 -1 reset not still connected to 0x560b15a70d00 2024-09-11T06:39:39.898 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:39:39.899+0000 7efd718c5640 -1 reset not still connected to 0x560b15a71ad0 2024-09-11T06:39:41.278 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:39:41.268+0000 7fc41084f640 -1 reset not still connected to 0x55eb16561450 2024-09-11T06:39:41.278 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:39:41.268+0000 7fc41084f640 -1 reset not still connected to 0x55eb1afbe4e0 2024-09-11T06:39:41.278 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:39:41.268+0000 7fc41084f640 -1 reset not still connected to 0x55eb2458e680 2024-09-11T06:39:41.278 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:39:41.268+0000 7fc41084f640 -1 reset not still connected to 0x55eb24ad96c0 2024-09-11T06:39:41.278 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:39:41.272+0000 7fc41084f640 -1 reset not still connected to 0x55eb2601e820 2024-09-11T06:39:41.279 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:39:41.276+0000 7fc41084f640 -1 reset not still connected to 0x55eb2613e680 2024-09-11T06:39:41.279 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:39:41.276+0000 7fc41084f640 -1 reset not still connected to 0x55eb2da77ba0 2024-09-11T06:39:41.279 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:39:41.276+0000 7fc41084f640 -1 reset not still connected to 0x55eb34898a90 2024-09-11T06:39:41.279 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:39:41.276+0000 7fc41084f640 -1 reset not still connected to 0x55eb35243040 2024-09-11T06:39:41.280 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:39:41.276+0000 7fc41084f640 -1 reset not still connected to 0x55eb354a21a0 2024-09-11T06:39:41.286 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:39:41.284+0000 7fc41084f640 -1 reset not still connected to 0x55eb3893ac30 2024-09-11T06:39:41.286 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:39:41.284+0000 7fc41084f640 -1 reset not still connected to 0x55eb3add8270 2024-09-11T06:39:41.286 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:39:41.284+0000 7fc41084f640 -1 reset not still connected to 0x55eb4480b930 2024-09-11T06:39:41.286 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:39:41.284+0000 7fc41084f640 -1 reset not still connected to 0x55eb4f1329c0 2024-09-11T06:39:41.286 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:39:41.284+0000 7fc41084f640 -1 reset not still connected to 0x55eb53d56f70 2024-09-11T06:39:43.537 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:43.532+0000 7fb45318f640 -1 reset not still connected to 0x561315ba4270 2024-09-11T06:39:43.541 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:43.536+0000 7fb45318f640 -1 reset not still connected to 0x561315ba5520 2024-09-11T06:39:43.541 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:43.536+0000 7fb45318f640 -1 reset not still connected to 0x56131a1ffe10 2024-09-11T06:39:43.541 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:43.536+0000 7fb45318f640 -1 reset not still connected to 0x561322931860 2024-09-11T06:39:43.542 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:43.540+0000 7fb45318f640 -1 reset not still connected to 0x561322ded930 2024-09-11T06:39:43.548 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:43.544+0000 7fb45318f640 -1 reset not still connected to 0x56133339c410 2024-09-11T06:39:43.548 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:43.544+0000 7fb45318f640 -1 reset not still connected to 0x56133c1909c0 2024-09-11T06:39:43.549 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:43.544+0000 7fb45318f640 -1 reset not still connected to 0x561341f69450 2024-09-11T06:39:43.549 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:43.544+0000 7fb45318f640 -1 reset not still connected to 0x561342043930 2024-09-11T06:39:43.553 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:43.552+0000 7fb45318f640 -1 reset not still connected to 0x5613480e41a0 2024-09-11T06:39:43.553 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:43.552+0000 7fb45318f640 -1 reset not still connected to 0x5613488b0820 2024-09-11T06:39:43.553 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:43.552+0000 7fb45318f640 -1 reset not still connected to 0x5613488b0d00 2024-09-11T06:39:43.554 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:43.552+0000 7fb45318f640 -1 reset not still connected to 0x56134d76fe10 2024-09-11T06:39:43.554 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:43.552+0000 7fb45318f640 -1 reset not still connected to 0x56134d81fd40 2024-09-11T06:39:43.557 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:43.552+0000 7fb45318f640 -1 reset not still connected to 0x561354d57040 2024-09-11T06:39:43.557 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:43.552+0000 7fb45318f640 -1 reset not still connected to 0x561354d77110 2024-09-11T06:39:43.557 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:43.552+0000 7fb45318f640 -1 reset not still connected to 0x561355040c30 2024-09-11T06:39:43.557 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:43.552+0000 7fb45318f640 -1 reset not still connected to 0x5613551fd110 2024-09-11T06:39:44.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 -p mirror mirror pool status 2024-09-11T06:39:44.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep 'image health:' 2024-09-11T06:39:44.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut '-d ' -f 3 2024-09-11T06:39:44.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: failed to query services: (13) Permission denied 2024-09-11T06:39:44.854 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ state=OK 2024-09-11T06:39:44.854 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test OK = ERROR 2024-09-11T06:39:44.854 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test OK = OK 2024-09-11T06:39:44.854 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:39:44.854 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_pool_images cluster1 mirror 50 2024-09-11T06:39:44.855 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:39:44.855 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:39:44.855 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_count=50 2024-09-11T06:39:44.855 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:39:44.855 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local count 2024-09-11T06:39:44.855 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local last_count=0 2024-09-11T06:39:44.855 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:39:44.855 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 1 40 2024-09-11T06:39:44.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in `seq 1 40` 2024-09-11T06:39:44.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 1 -ne 1 2024-09-11T06:39:44.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2024-09-11T06:39:44.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep 'images: ' 2024-09-11T06:39:44.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut '-d ' -f 2 2024-09-11T06:39:44.908 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:39:44.900+0000 7fc41084f640 -1 reset not still connected to 0x55eb1fefeb60 2024-09-11T06:39:44.909 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:39:44.900+0000 7fc41084f640 -1 reset not still connected to 0x55eb2458e680 2024-09-11T06:39:44.909 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:39:44.900+0000 7fc41084f640 -1 reset not still connected to 0x55eb24ad96c0 2024-09-11T06:39:44.909 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:39:44.900+0000 7fc41084f640 -1 reset not still connected to 0x55eb2601e820 2024-09-11T06:39:44.909 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:39:44.900+0000 7fc41084f640 -1 reset not still connected to 0x55eb2613e680 2024-09-11T06:39:44.909 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:39:44.900+0000 7fc41084f640 -1 reset not still connected to 0x55eb2da77ba0 2024-09-11T06:39:44.910 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:39:44.904+0000 7fc41084f640 -1 reset not still connected to 0x55eb345e5ad0 2024-09-11T06:39:44.910 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:39:44.904+0000 7fc41084f640 -1 reset not still connected to 0x55eb34898a90 2024-09-11T06:39:44.910 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:39:44.904+0000 7fc41084f640 -1 reset not still connected to 0x55eb35243040 2024-09-11T06:39:44.910 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:39:44.908+0000 7fc41084f640 -1 reset not still connected to 0x55eb354a21a0 2024-09-11T06:39:44.910 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:39:44.908+0000 7fc41084f640 -1 reset not still connected to 0x55eb3893ac30 2024-09-11T06:39:44.911 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:39:44.908+0000 7fc41084f640 -1 reset not still connected to 0x55eb4480b930 2024-09-11T06:39:44.911 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:39:44.908+0000 7fc41084f640 -1 reset not still connected to 0x55eb4f1329c0 2024-09-11T06:39:44.911 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:39:44.908+0000 7fc41084f640 -1 reset not still connected to 0x55eb53d56f70 2024-09-11T06:39:44.977 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: failed to query services: (13) Permission denied 2024-09-11T06:39:44.982 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ count=50 2024-09-11T06:39:44.982 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 50 = 50 2024-09-11T06:39:44.982 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:39:44.982 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_pool_healthy cluster1 mirror 2024-09-11T06:39:44.982 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:39:44.982 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:39:44.982 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:39:44.982 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state 2024-09-11T06:39:44.982 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 1 40 2024-09-11T06:39:44.983 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in `seq 1 40` 2024-09-11T06:39:44.983 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 1 -ne 1 2024-09-11T06:39:44.983 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2024-09-11T06:39:44.983 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep 'image health:' 2024-09-11T06:39:44.983 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut '-d ' -f 3 2024-09-11T06:39:45.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: failed to query services: (13) Permission denied 2024-09-11T06:39:45.037 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ state=OK 2024-09-11T06:39:45.037 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test OK = ERROR 2024-09-11T06:39:45.037 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test OK = OK 2024-09-11T06:39:45.037 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:39:45.037 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ testlog 'TEST: compare many images' 2024-09-11T06:39:45.038 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ tee -a /tmp/tmp.Dxwd1ESqKa/rbd-mirror.test.log 2024-09-11T06:39:45.038 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ date '+%F %T' 2024-09-11T06:39:45.057 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ echo 2024-09-11 06:39:45 TEST: compare many images 2024-09-11T06:39:45.057 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:2024-09-11 06:39:45 TEST: compare many images 2024-09-11T06:39:45.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 1 50 2024-09-11T06:39:45.064 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:39:45.064 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_1 2024-09-11T06:39:45.064 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_snap cluster2 mirror image_1 snap 2024-09-11T06:39:45.064 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:39:45.064 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:39:45.064 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_1 2024-09-11T06:39:45.064 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:39:45.064 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror snap create image_1@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T06:39:45.260 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:45.256+0000 7fb45318f640 -1 reset not still connected to 0x56134d81fd40 2024-09-11T06:39:45.270 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:45.268+0000 7fb45318f640 -1 reset not still connected to 0x561354d77110 2024-09-11T06:39:45.272 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:45.268+0000 7fb45318f640 -1 reset not still connected to 0x561355040c30 2024-09-11T06:39:45.275 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:45.272+0000 7fb45318f640 -1 reset not still connected to 0x561315884d00 2024-09-11T06:39:45.275 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:45.272+0000 7fb45318f640 -1 reset not still connected to 0x561315ba4270 2024-09-11T06:39:45.275 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:45.272+0000 7fb45318f640 -1 reset not still connected to 0x561315ba5520 2024-09-11T06:39:45.276 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:45.272+0000 7fb45318f640 -1 reset not still connected to 0x56131a1ffe10 2024-09-11T06:39:45.276 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:45.272+0000 7fb45318f640 -1 reset not still connected to 0x561322931860 2024-09-11T06:39:45.276 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:45.272+0000 7fb45318f640 -1 reset not still connected to 0x561330ed0410 2024-09-11T06:39:45.276 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:45.272+0000 7fb45318f640 -1 reset not still connected to 0x5613480e41a0 2024-09-11T06:39:45.276 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:45.272+0000 7fb45318f640 -1 reset not still connected to 0x5613488b0820 2024-09-11T06:39:45.276 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:45.272+0000 7fb45318f640 -1 reset not still connected to 0x56134d81fd40 2024-09-11T06:39:45.276 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:45.272+0000 7fb45318f640 -1 reset not still connected to 0x561354d77110 2024-09-11T06:39:45.276 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:45.272+0000 7fb45318f640 -1 reset not still connected to 0x561355040c30 2024-09-11T06:39:45.372 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:45.368+0000 7fb2ca654640 -1 reset not still connected to 0x55ed43824c30 2024-09-11T06:39:45.372 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:45.368+0000 7fb2ca654640 -1 reset not still connected to 0x55ed50ca3380 2024-09-11T06:39:45.605 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_started cluster1 mirror image_1 2024-09-11T06:39:45.605 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:39:45.606 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:39:45.606 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_1 2024-09-11T06:39:45.606 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_state cluster1 mirror image_1 started 2024-09-11T06:39:45.606 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:39:45.606 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:39:45.606 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_1 2024-09-11T06:39:45.606 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=started 2024-09-11T06:39:45.606 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:39:45.606 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:39:45.606 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:39:45.890 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:45.880+0000 7fb2ca654640 -1 reset not still connected to 0x55ed1b91f450 2024-09-11T06:39:45.890 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:45.880+0000 7fb2ca654640 -1 reset not still connected to 0x55ed288cd110 2024-09-11T06:39:45.892 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:45.884+0000 7fb2ca654640 -1 reset not still connected to 0x55ed32b6bba0 2024-09-11T06:39:45.892 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:45.884+0000 7fb2ca654640 -1 reset not still connected to 0x55ed369ccc30 2024-09-11T06:39:45.893 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:45.884+0000 7fb2ca654640 -1 reset not still connected to 0x55ed3a812820 2024-09-11T06:39:45.896 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:45.888+0000 7fb2ca654640 -1 reset not still connected to 0x55ed43611450 2024-09-11T06:39:45.897 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:45.888+0000 7fb2ca654640 -1 reset not still connected to 0x55ed44536340 2024-09-11T06:39:45.897 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:45.892+0000 7fb2ca654640 -1 reset not still connected to 0x55ed44538dd0 2024-09-11T06:39:45.897 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:45.892+0000 7fb2ca654640 -1 reset not still connected to 0x55ed4b336c30 2024-09-11T06:39:45.897 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:45.892+0000 7fb2ca654640 -1 reset not still connected to 0x55ed4b37ca90 2024-09-11T06:39:46.606 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_replay_state cluster1 mirror image_1 started 2024-09-11T06:39:46.606 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:39:46.606 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:39:46.606 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_1 2024-09-11T06:39:46.606 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=started 2024-09-11T06:39:46.607 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local status_result 2024-09-11T06:39:46.607 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=stopped 2024-09-11T06:39:46.608 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_1 2024-09-11T06:39:46.608 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -i state 2024-09-11T06:39:46.608 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_instance=cluster1 2024-09-11T06:39:46.608 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:39:46.608 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:39:46.608 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance=cluster1 2024-09-11T06:39:46.608 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local loop_instance 2024-09-11T06:39:46.608 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:39:46.608 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sleep 0 2024-09-11T06:39:46.609 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:39:46.610 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ seq 0 1 2024-09-11T06:39:46.613 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:39:46.613 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_1 2024-09-11T06:39:46.613 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1:0 2024-09-11T06:39:46.613 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:39:46.613 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:39:46.614 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:39:46.614 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:39:46.614 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=cluster 2024-09-11T06:39:46.614 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:39:46.614 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:39:46.614 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:39:46.614 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:39:46.614 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval cluster=cluster1 2024-09-11T06:39:46.614 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:39:46.614 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:39:46.614 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:39:46.614 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:39:46.614 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local local_cluster=cluster1:0 2024-09-11T06:39:46.614 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster=cluster1 2024-09-11T06:39:46.615 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance 2024-09-11T06:39:46.615 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:39:46.615 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local val=cluster1:0 2024-09-11T06:39:46.615 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T06:39:46.615 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance_var_name=instance 2024-09-11T06:39:46.615 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:39:46.615 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:39:46.615 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T06:39:46.615 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval local_cluster=cluster1 2024-09-11T06:39:46.615 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ local_cluster=cluster1 2024-09-11T06:39:46.615 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval instance=0 2024-09-11T06:39:46.615 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ instance=0 2024-09-11T06:39:46.615 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:39:46.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:39:46.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:39:46.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:39:46.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_1 2024-09-11T06:39:46.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:no valid command found; 5 closest matches: 2024-09-11T06:39:46.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd mirror status mirror/image_11 2024-09-11T06:39:46.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd mirror status mirror/image_13 2024-09-11T06:39:46.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd mirror status mirror/image_15 2024-09-11T06:39:46.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd mirror status mirror/image_17 2024-09-11T06:39:46.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd mirror status mirror/image_19 2024-09-11T06:39:46.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:admin_socket: invalid command 2024-09-11T06:39:46.909 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:39:46.909 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_1 2024-09-11T06:39:46.909 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1:1 2024-09-11T06:39:46.909 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:39:46.909 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:39:46.909 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-11T06:39:46.909 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:1 2024-09-11T06:39:46.910 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=cluster 2024-09-11T06:39:46.910 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:39:46.910 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:39:46.910 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=1 2024-09-11T06:39:46.910 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T06:39:46.910 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval cluster=cluster1 2024-09-11T06:39:46.910 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:39:46.910 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=1 2024-09-11T06:39:46.910 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=1 2024-09-11T06:39:46.910 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-11T06:39:46.910 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local local_cluster=cluster1:1 2024-09-11T06:39:46.910 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster=cluster1 2024-09-11T06:39:46.911 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance 2024-09-11T06:39:46.911 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T06:39:46.911 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local val=cluster1:1 2024-09-11T06:39:46.911 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T06:39:46.911 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance_var_name=instance 2024-09-11T06:39:46.911 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:39:46.911 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=1 2024-09-11T06:39:46.911 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-11T06:39:46.912 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval local_cluster=cluster1 2024-09-11T06:39:46.912 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ local_cluster=cluster1 2024-09-11T06:39:46.912 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval instance=1 2024-09-11T06:39:46.912 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ instance=1 2024-09-11T06:39:46.912 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T06:39:46.929 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:39:46.929 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:39:46.929 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:39:46.930 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_1 2024-09-11T06:39:47.068 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ return 0 2024-09-11T06:39:47.068 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T06:39:47.069 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ echo ' "state": "Replaying"' 2024-09-11T06:39:47.069 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ grep -i Replaying 2024-09-11T06:39:47.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: "state": "Replaying" 2024-09-11T06:39:47.072 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_state=started 2024-09-11T06:39:47.073 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test started = started 2024-09-11T06:39:47.073 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:39:47.073 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_1 2024-09-11T06:39:47.073 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:39:47.073 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:39:47.073 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:39:47.073 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_1 2024-09-11T06:39:47.073 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' journal = journal ']' 2024-09-11T06:39:47.073 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_1 2024-09-11T06:39:47.073 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:39:47.073 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:39:47.073 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:39:47.073 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_1 2024-09-11T06:39:47.073 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-11T06:39:47.074 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-11T06:39:47.074 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:39:47.074 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:39:47.074 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:39:47.274 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror image_1 2024-09-11T06:39:47.274 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:39:47.274 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:39:47.274 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_1 2024-09-11T06:39:47.274 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:39:47.274 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n image_1 ']' 2024-09-11T06:39:47.275 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/image_1' 2024-09-11T06:39:47.275 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_1 2024-09-11T06:39:47.275 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:39:47.275 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:39:47.275 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:39:47.275 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:39:47.275 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:39:47.275 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:39:47.275 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:39:47.276 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:39:47.276 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:39:47.279 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:39:47.279 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_1 2024-09-11T06:39:47.280 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:39:47.280 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:39:47.280 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:39:47.280 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:39:47.280 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:39:47.280 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:39:47.280 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:39:47.280 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:39:47.280 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:39:47.280 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:39:47.280 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:39:47.280 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:39:47.280 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:39:47.281 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:39:47.281 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:39:47.281 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:39:47.281 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:39:47.281 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:39:47.281 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:39:47.281 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:39:47.281 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:39:47.281 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:39:47.281 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:39:47.281 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:39:47.281 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:39:47.281 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:39:47.282 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:39:47.282 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:39:47.282 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:39:47.282 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:39:47.306 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:39:47.306 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:39:47.306 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:39:47.306 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_1 2024-09-11T06:39:47.432 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:no valid command found; 5 closest matches: 2024-09-11T06:39:47.432 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd mirror flush mirror/image_11 2024-09-11T06:39:47.433 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd mirror flush mirror/image_13 2024-09-11T06:39:47.433 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd mirror flush mirror/image_15 2024-09-11T06:39:47.433 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd mirror flush mirror/image_17 2024-09-11T06:39:47.433 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd mirror flush mirror/image_19 2024-09-11T06:39:47.433 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:admin_socket: invalid command 2024-09-11T06:39:47.436 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:39:47.436 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_1 2024-09-11T06:39:47.436 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:1 2024-09-11T06:39:47.436 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:39:47.436 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:39:47.437 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-11T06:39:47.437 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:1 2024-09-11T06:39:47.437 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:39:47.437 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:39:47.437 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:39:47.437 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=1 2024-09-11T06:39:47.437 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-11T06:39:47.437 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:39:47.437 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:39:47.437 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=1 2024-09-11T06:39:47.437 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=1 2024-09-11T06:39:47.437 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-11T06:39:47.437 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:1 2024-09-11T06:39:47.437 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:39:47.438 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:39:47.438 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T06:39:47.438 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:1 2024-09-11T06:39:47.438 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:39:47.438 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:39:47.438 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:39:47.438 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=1 2024-09-11T06:39:47.438 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T06:39:47.438 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:39:47.438 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:39:47.438 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=1 2024-09-11T06:39:47.438 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=1 2024-09-11T06:39:47.438 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T06:39:47.456 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:39:47.456 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:39:47.456 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:39:47.457 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_1 2024-09-11T06:39:47.594 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:39:47.594 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror image_1 2024-09-11T06:39:47.594 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:39:47.594 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:39:47.594 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_1 2024-09-11T06:39:47.594 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror image_1 '' 2024-09-11T06:39:47.594 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:39:47.594 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:39:47.594 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_1 2024-09-11T06:39:47.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:39:47.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-image_1.status 2024-09-11T06:39:47.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:39:47.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-image_1.status 2024-09-11T06:39:47.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_1.status 2024-09-11T06:39:47.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_1 2024-09-11T06:39:47.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_1.status 2024-09-11T06:39:47.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 1 2024-09-11T06:39:47.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 1 2024-09-11T06:39:47.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:39:47.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5595], [object_number=6, tag_tid=1, entry_tid=5594]]], state=connected] 2024-09-11T06:39:47.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=5, tag_tid=1, entry_tid=5597], [object_number=4, tag_tid=1, entry_tid=5596], [object_number=7, tag_tid=1, entry_tid=5595], [object_number=6, tag_tid=1, entry_tid=5594]]], state=connected] 2024-09-11T06:39:47.677 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_1.status 2024-09-11T06:39:47.678 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-09-11T06:39:47.680 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror image_1 2024-09-11T06:39:47.680 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:39:47.680 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:39:47.680 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_1 2024-09-11T06:39:47.680 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror image_1 '..*' 2024-09-11T06:39:47.680 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:39:47.680 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:39:47.680 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_1 2024-09-11T06:39:47.680 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:39:47.681 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-image_1.status 2024-09-11T06:39:47.681 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-image_1.status 2024-09-11T06:39:47.681 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:39:47.682 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_1.status 2024-09-11T06:39:47.682 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_1 2024-09-11T06:39:47.682 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_1.status 2024-09-11T06:39:47.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 1 2024-09-11T06:39:47.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 1 2024-09-11T06:39:47.729 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:39:47.729 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5595], [object_number=6, tag_tid=1, entry_tid=5594]]], state=connected] 2024-09-11T06:39:47.729 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=5, tag_tid=1, entry_tid=5597], [object_number=4, tag_tid=1, entry_tid=5596], [object_number=7, tag_tid=1, entry_tid=5595], [object_number=6, tag_tid=1, entry_tid=5594]]], state=connected] 2024-09-11T06:39:47.732 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_1.status 2024-09-11T06:39:47.734 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=5, tag_tid=1, entry_tid=5597' 2024-09-11T06:39:47.734 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=1, entry_tid=5597' 2024-09-11T06:39:47.734 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=5, tag_tid=1, entry_tid=5597' '!=' '' 2024-09-11T06:39:47.734 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:39:47.735 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=5, tag_tid=1, entry_tid=5597' = '' 2024-09-11T06:39:47.735 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=5, tag_tid=1, entry_tid=5597' 2024-09-11T06:39:47.735 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=5, tag_tid=3, entry_tid=1' 2024-09-11T06:39:47.735 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:39:47.735 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:39:47.735 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=3 2024-09-11T06:39:47.736 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=5, tag_tid=1, entry_tid=5597' 2024-09-11T06:39:47.736 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:39:47.736 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:39:47.737 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=1 2024-09-11T06:39:47.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=5, tag_tid=3, entry_tid=1' 2024-09-11T06:39:47.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:39:47.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:39:47.739 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:39:47.739 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=5, tag_tid=1, entry_tid=5597' 2024-09-11T06:39:47.740 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:39:47.740 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:39:47.741 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=5597 2024-09-11T06:39:47.741 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 3 = 1 -a 1 -le 5597 2024-09-11T06:39:47.741 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:39:47.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:39:47.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:39:47.943 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror image_1 2024-09-11T06:39:47.943 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:39:47.943 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:39:47.943 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_1 2024-09-11T06:39:47.943 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:39:47.943 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n image_1 ']' 2024-09-11T06:39:47.943 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/image_1' 2024-09-11T06:39:47.944 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_1 2024-09-11T06:39:47.944 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:39:47.944 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:39:47.944 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:39:47.944 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:39:47.944 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:39:47.944 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:39:47.944 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:39:47.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:39:47.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:39:47.947 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:39:47.947 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_1 2024-09-11T06:39:47.947 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:39:47.948 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:39:47.948 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:39:47.948 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:39:47.948 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:39:47.948 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:39:47.948 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:39:47.948 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:39:47.949 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:39:47.949 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:39:47.949 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:39:47.949 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:39:47.949 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:39:47.949 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:39:47.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:39:47.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:39:47.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:39:47.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:39:47.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:39:47.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:39:47.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:39:47.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:39:47.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:39:47.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:39:47.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:39:47.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:39:47.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:39:47.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:39:47.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:39:47.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:39:47.976 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:39:47.977 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:39:47.977 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:39:47.977 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_1 2024-09-11T06:39:48.105 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:no valid command found; 5 closest matches: 2024-09-11T06:39:48.105 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd mirror flush mirror/image_11 2024-09-11T06:39:48.105 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd mirror flush mirror/image_13 2024-09-11T06:39:48.105 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd mirror flush mirror/image_15 2024-09-11T06:39:48.105 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd mirror flush mirror/image_17 2024-09-11T06:39:48.105 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd mirror flush mirror/image_19 2024-09-11T06:39:48.105 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:admin_socket: invalid command 2024-09-11T06:39:48.108 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:39:48.108 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_1 2024-09-11T06:39:48.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:1 2024-09-11T06:39:48.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:39:48.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:39:48.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-11T06:39:48.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:1 2024-09-11T06:39:48.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:39:48.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:39:48.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:39:48.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=1 2024-09-11T06:39:48.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-11T06:39:48.110 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:39:48.110 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:39:48.110 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=1 2024-09-11T06:39:48.110 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=1 2024-09-11T06:39:48.110 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-11T06:39:48.110 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:1 2024-09-11T06:39:48.110 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:39:48.110 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:39:48.110 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T06:39:48.110 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:1 2024-09-11T06:39:48.110 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:39:48.110 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:39:48.110 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:39:48.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=1 2024-09-11T06:39:48.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T06:39:48.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:39:48.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:39:48.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=1 2024-09-11T06:39:48.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=1 2024-09-11T06:39:48.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T06:39:48.128 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:39:48.129 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:39:48.129 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:39:48.129 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_1 2024-09-11T06:39:48.270 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:39:48.270 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror image_1 2024-09-11T06:39:48.271 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:39:48.271 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:39:48.271 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_1 2024-09-11T06:39:48.271 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror image_1 '' 2024-09-11T06:39:48.271 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:39:48.271 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:39:48.271 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_1 2024-09-11T06:39:48.271 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:39:48.271 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-image_1.status 2024-09-11T06:39:48.271 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:39:48.272 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-image_1.status 2024-09-11T06:39:48.272 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_1.status 2024-09-11T06:39:48.272 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_1 2024-09-11T06:39:48.272 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_1.status 2024-09-11T06:39:48.326 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 1 2024-09-11T06:39:48.326 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 1 2024-09-11T06:39:48.326 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:39:48.326 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5595], [object_number=6, tag_tid=1, entry_tid=5594]]], state=connected] 2024-09-11T06:39:48.327 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5595], [object_number=6, tag_tid=1, entry_tid=5594]]], state=connected] 2024-09-11T06:39:48.330 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_1.status 2024-09-11T06:39:48.333 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-09-11T06:39:48.333 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror image_1 2024-09-11T06:39:48.333 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:39:48.333 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:39:48.334 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_1 2024-09-11T06:39:48.334 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror image_1 '..*' 2024-09-11T06:39:48.334 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:39:48.334 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:39:48.334 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_1 2024-09-11T06:39:48.334 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:39:48.334 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-image_1.status 2024-09-11T06:39:48.334 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-image_1.status 2024-09-11T06:39:48.334 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:39:48.335 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_1.status 2024-09-11T06:39:48.335 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_1 2024-09-11T06:39:48.335 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_1.status 2024-09-11T06:39:48.382 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 1 2024-09-11T06:39:48.382 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 1 2024-09-11T06:39:48.382 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:39:48.382 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5595], [object_number=6, tag_tid=1, entry_tid=5594]]], state=connected] 2024-09-11T06:39:48.382 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=5595], [object_number=6, tag_tid=1, entry_tid=5594]]], state=connected] 2024-09-11T06:39:48.385 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_1.status 2024-09-11T06:39:48.387 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-09-11T06:39:48.387 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=3, entry_tid=1' 2024-09-11T06:39:48.387 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:39:48.387 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_snap_present cluster1 mirror image_1 snap 2024-09-11T06:39:48.387 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:39:48.387 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:39:48.388 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_1 2024-09-11T06:39:48.388 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:39:48.388 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:39:48.388 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T06:39:48.388 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:39:49.388 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 info mirror/image_1@snap 2024-09-11T06:39:49.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:rbd image 'image_1': 2024-09-11T06:39:49.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: size 128 MiB in 32 objects 2024-09-11T06:39:49.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: order 22 (4 MiB objects) 2024-09-11T06:39:49.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: snapshot_count: 1 2024-09-11T06:39:49.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: id: 10a9819e4592 2024-09-11T06:39:49.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: block_name_prefix: rbd_data.10a9819e4592 2024-09-11T06:39:49.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: format: 2 2024-09-11T06:39:49.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: features: layering, exclusive-lock, journaling 2024-09-11T06:39:49.453 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: op_features: 2024-09-11T06:39:49.454 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: flags: 2024-09-11T06:39:49.454 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: create_timestamp: Wed Sep 11 06:31:00 2024 2024-09-11T06:39:49.454 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: access_timestamp: Wed Sep 11 06:31:00 2024 2024-09-11T06:39:49.454 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: modify_timestamp: Wed Sep 11 06:32:00 2024 2024-09-11T06:39:49.454 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: protected: False 2024-09-11T06:39:49.454 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: journal: 10a9819e4592 2024-09-11T06:39:49.454 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring state: enabled 2024-09-11T06:39:49.454 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring mode: journal 2024-09-11T06:39:49.454 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring global id: cf5968d2-26be-4ed4-8bbc-11d945800984 2024-09-11T06:39:49.454 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring primary: false 2024-09-11T06:39:49.457 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:39:49.458 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ compare_image_snaps mirror image_1 snap 2024-09-11T06:39:49.458 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:39:49.458 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_1 2024-09-11T06:39:49.458 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:39:49.458 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local ret=0 2024-09-11T06:39:49.458 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local rmt_export=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_1.export 2024-09-11T06:39:49.458 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loc_export=/tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_1.export 2024-09-11T06:39:49.458 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_1.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_1.export 2024-09-11T06:39:49.470 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror export image_1@snap /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_1.export 2024-09-11T06:39:49.649 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T06:39:49.663 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 -p mirror export image_1@snap /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_1.export 2024-09-11T06:39:49.926 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T06:39:49.935 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmp /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_1.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_1.export 2024-09-11T06:39:50.007 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_1.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_1.export 2024-09-11T06:39:50.045 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:39:50.045 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:39:50.045 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_2 2024-09-11T06:39:50.045 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_snap cluster2 mirror image_2 snap 2024-09-11T06:39:50.046 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:39:50.046 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:39:50.046 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_2 2024-09-11T06:39:50.046 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:39:50.046 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror snap create image_2@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T06:39:50.207 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:50.204+0000 7fb2ca654640 -1 reset not still connected to 0x55ed0e363ee0 2024-09-11T06:39:50.211 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:50.208+0000 7fb2ca654640 -1 reset not still connected to 0x55ed0e47ba00 2024-09-11T06:39:50.211 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:50.208+0000 7fb2ca654640 -1 reset not still connected to 0x55ed11f509c0 2024-09-11T06:39:50.218 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:50.216+0000 7fb2ca654640 -1 reset not still connected to 0x55ed15c36d00 2024-09-11T06:39:50.218 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:50.216+0000 7fb2ca654640 -1 reset not still connected to 0x55ed169d91e0 2024-09-11T06:39:50.218 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:50.216+0000 7fb2ca654640 -1 reset not still connected to 0x55ed1b91f450 2024-09-11T06:39:50.219 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:50.216+0000 7fb2ca654640 -1 reset not still connected to 0x55ed1c2d45b0 2024-09-11T06:39:50.219 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:50.216+0000 7fb2ca654640 -1 reset not still connected to 0x55ed288cd110 2024-09-11T06:39:50.222 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:50.220+0000 7fb2ca654640 -1 reset not still connected to 0x55ed2b074410 2024-09-11T06:39:50.224 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:50.220+0000 7fb2ca654640 -1 reset not still connected to 0x55ed32b6bba0 2024-09-11T06:39:50.224 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:50.220+0000 7fb2ca654640 -1 reset not still connected to 0x55ed32ea5110 2024-09-11T06:39:50.224 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:50.220+0000 7fb2ca654640 -1 reset not still connected to 0x55ed369ccc30 2024-09-11T06:39:50.226 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:50.220+0000 7fb2ca654640 -1 reset not still connected to 0x55ed3a812820 2024-09-11T06:39:50.227 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:50.224+0000 7fb2ca654640 -1 reset not still connected to 0x55ed3ecf6b60 2024-09-11T06:39:50.227 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:50.224+0000 7fb2ca654640 -1 reset not still connected to 0x55ed3ecf76c0 2024-09-11T06:39:50.227 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:50.224+0000 7fb2ca654640 -1 reset not still connected to 0x55ed43611450 2024-09-11T06:39:50.227 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:50.224+0000 7fb2ca654640 -1 reset not still connected to 0x55ed436c8750 2024-09-11T06:39:50.227 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:50.224+0000 7fb2ca654640 -1 reset not still connected to 0x55ed44536340 2024-09-11T06:39:50.227 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:50.224+0000 7fb2ca654640 -1 reset not still connected to 0x55ed44538a90 2024-09-11T06:39:50.228 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:50.224+0000 7fb2ca654640 -1 reset not still connected to 0x55ed44538dd0 2024-09-11T06:39:50.228 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:50.224+0000 7fb2ca654640 -1 reset not still connected to 0x55ed4b336c30 2024-09-11T06:39:50.228 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:50.224+0000 7fb2ca654640 -1 reset not still connected to 0x55ed4b37ca90 2024-09-11T06:39:50.455 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:50.448+0000 7fb45318f640 -1 reset not still connected to 0x5613488b0820 2024-09-11T06:39:50.455 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:50.448+0000 7fb45318f640 -1 reset not still connected to 0x56134d81fd40 2024-09-11T06:39:50.459 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:50.456+0000 7fb45318f640 -1 reset not still connected to 0x561354d77110 2024-09-11T06:39:50.459 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:50.456+0000 7fb45318f640 -1 reset not still connected to 0x561355040c30 2024-09-11T06:39:50.463 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:50.460+0000 7fb45318f640 -1 reset not still connected to 0x5613551fd110 2024-09-11T06:39:50.924 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:50.908+0000 7fb45318f640 -1 reset not still connected to 0x561330ed0410 2024-09-11T06:39:50.929 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:50.916+0000 7fb45318f640 -1 reset not still connected to 0x56133339c410 2024-09-11T06:39:50.929 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:50.916+0000 7fb45318f640 -1 reset not still connected to 0x56133c1909c0 2024-09-11T06:39:50.930 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:50.924+0000 7fb45318f640 -1 reset not still connected to 0x561342043930 2024-09-11T06:39:50.930 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:50.924+0000 7fb45318f640 -1 reset not still connected to 0x5613480872b0 2024-09-11T06:39:50.930 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:50.924+0000 7fb45318f640 -1 reset not still connected to 0x5613480e41a0 2024-09-11T06:39:50.930 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:50.924+0000 7fb45318f640 -1 reset not still connected to 0x5613488b0820 2024-09-11T06:39:50.930 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:50.924+0000 7fb45318f640 -1 reset not still connected to 0x5613488b0d00 2024-09-11T06:39:50.931 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:50.924+0000 7fb45318f640 -1 reset not still connected to 0x56134d76fe10 2024-09-11T06:39:50.931 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:50.924+0000 7fb45318f640 -1 reset not still connected to 0x56134d81fd40 2024-09-11T06:39:50.931 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:50.924+0000 7fb45318f640 -1 reset not still connected to 0x561354d57040 2024-09-11T06:39:50.931 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:50.924+0000 7fb45318f640 -1 reset not still connected to 0x561354d77110 2024-09-11T06:39:50.931 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:50.924+0000 7fb45318f640 -1 reset not still connected to 0x561355040c30 2024-09-11T06:39:50.931 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:50.924+0000 7fb45318f640 -1 reset not still connected to 0x5613551fd110 2024-09-11T06:39:50.931 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:50.924+0000 7fb45318f640 -1 reset not still connected to 0x5613557c4680 2024-09-11T06:39:51.497 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:51.492+0000 7fb2ca654640 -1 reset not still connected to 0x55ed43611450 2024-09-11T06:39:51.501 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:51.500+0000 7fb2ca654640 -1 reset not still connected to 0x55ed44536340 2024-09-11T06:39:51.501 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:51.500+0000 7fb2ca654640 -1 reset not still connected to 0x55ed4b336c30 2024-09-11T06:39:51.508 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:51.504+0000 7fb2ca654640 -1 reset not still connected to 0x55ed4b37ca90 2024-09-11T06:39:52.631 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_started cluster1 mirror image_2 2024-09-11T06:39:52.631 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:39:52.631 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:39:52.631 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_2 2024-09-11T06:39:52.631 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_state cluster1 mirror image_2 started 2024-09-11T06:39:52.631 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:39:52.631 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:39:52.631 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_2 2024-09-11T06:39:52.631 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=started 2024-09-11T06:39:52.632 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:39:52.632 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:39:52.632 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:39:53.227 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:53.224+0000 7fb45318f640 -1 reset not still connected to 0x56134d81fd40 2024-09-11T06:39:53.227 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:53.224+0000 7fb45318f640 -1 reset not still connected to 0x561354d77110 2024-09-11T06:39:53.229 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:53.228+0000 7fb45318f640 -1 reset not still connected to 0x561355040c30 2024-09-11T06:39:53.232 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:53.228+0000 7fb45318f640 -1 reset not still connected to 0x561315ba4270 2024-09-11T06:39:53.232 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:53.228+0000 7fb45318f640 -1 reset not still connected to 0x561330ed0410 2024-09-11T06:39:53.232 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:53.228+0000 7fb45318f640 -1 reset not still connected to 0x561354d77110 2024-09-11T06:39:53.233 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:53.228+0000 7fb45318f640 -1 reset not still connected to 0x561355040c30 2024-09-11T06:39:53.632 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_replay_state cluster1 mirror image_2 started 2024-09-11T06:39:53.632 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:39:53.633 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:39:53.633 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_2 2024-09-11T06:39:53.633 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=started 2024-09-11T06:39:53.633 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local status_result 2024-09-11T06:39:53.633 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=stopped 2024-09-11T06:39:53.633 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -i state 2024-09-11T06:39:53.636 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_2 2024-09-11T06:39:53.636 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_instance=cluster1 2024-09-11T06:39:53.636 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:39:53.636 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:39:53.636 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance=cluster1 2024-09-11T06:39:53.636 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local loop_instance 2024-09-11T06:39:53.636 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:39:53.636 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sleep 0 2024-09-11T06:39:53.637 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:39:53.637 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ seq 0 1 2024-09-11T06:39:53.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:39:53.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_2 2024-09-11T06:39:53.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1:0 2024-09-11T06:39:53.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:39:53.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:39:53.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:39:53.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:39:53.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=cluster 2024-09-11T06:39:53.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:39:53.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:39:53.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:39:53.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:39:53.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval cluster=cluster1 2024-09-11T06:39:53.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:39:53.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:39:53.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:39:53.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:39:53.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local local_cluster=cluster1:0 2024-09-11T06:39:53.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster=cluster1 2024-09-11T06:39:53.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance 2024-09-11T06:39:53.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:39:53.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local val=cluster1:0 2024-09-11T06:39:53.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T06:39:53.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance_var_name=instance 2024-09-11T06:39:53.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:39:53.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:39:53.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T06:39:53.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval local_cluster=cluster1 2024-09-11T06:39:53.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ local_cluster=cluster1 2024-09-11T06:39:53.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval instance=0 2024-09-11T06:39:53.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ instance=0 2024-09-11T06:39:53.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:39:53.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:39:53.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:39:53.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:39:53.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_2 2024-09-11T06:39:53.806 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ return 0 2024-09-11T06:39:53.807 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T06:39:53.807 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ echo ' "state": "Replaying"' 2024-09-11T06:39:53.807 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ grep -i Replaying 2024-09-11T06:39:53.808 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: "state": "Replaying" 2024-09-11T06:39:53.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_state=started 2024-09-11T06:39:53.810 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test started = started 2024-09-11T06:39:53.810 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:39:53.810 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_2 2024-09-11T06:39:53.810 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:39:53.810 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:39:53.810 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:39:53.810 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_2 2024-09-11T06:39:53.810 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' journal = journal ']' 2024-09-11T06:39:53.810 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_2 2024-09-11T06:39:53.810 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:39:53.810 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:39:53.810 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:39:53.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_2 2024-09-11T06:39:53.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-11T06:39:53.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-11T06:39:53.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:39:53.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:39:53.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:39:54.011 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror image_2 2024-09-11T06:39:54.011 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:39:54.012 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:39:54.012 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_2 2024-09-11T06:39:54.012 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:39:54.012 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n image_2 ']' 2024-09-11T06:39:54.012 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/image_2' 2024-09-11T06:39:54.012 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_2 2024-09-11T06:39:54.012 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:39:54.012 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:39:54.012 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:39:54.012 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:39:54.012 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:39:54.012 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:39:54.013 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:39:54.015 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:39:54.015 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:39:54.015 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:39:54.015 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_2 2024-09-11T06:39:54.015 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:39:54.015 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:39:54.015 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:39:54.015 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:39:54.016 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:39:54.016 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:39:54.016 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:39:54.016 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:39:54.016 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:39:54.016 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:39:54.016 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:39:54.016 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:39:54.016 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:39:54.016 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:39:54.016 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:39:54.016 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:39:54.016 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:39:54.017 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:39:54.017 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:39:54.017 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:39:54.017 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:39:54.017 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:39:54.017 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:39:54.017 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:39:54.017 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:39:54.017 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:39:54.017 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:39:54.017 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:39:54.017 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:39:54.017 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:39:54.041 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:39:54.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:39:54.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:39:54.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_2 2024-09-11T06:39:54.047 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:54.040+0000 7fb45318f640 -1 reset not still connected to 0x561330ed0410 2024-09-11T06:39:54.057 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:54.052+0000 7fb45318f640 -1 reset not still connected to 0x5613480e41a0 2024-09-11T06:39:54.060 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:54.056+0000 7fb45318f640 -1 reset not still connected to 0x5613488b0820 2024-09-11T06:39:54.060 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:54.056+0000 7fb45318f640 -1 reset not still connected to 0x5613488b0d00 2024-09-11T06:39:54.060 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:54.056+0000 7fb45318f640 -1 reset not still connected to 0x56134d76fe10 2024-09-11T06:39:54.060 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:54.056+0000 7fb45318f640 -1 reset not still connected to 0x56134d81fd40 2024-09-11T06:39:54.060 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:54.056+0000 7fb45318f640 -1 reset not still connected to 0x561354d77110 2024-09-11T06:39:54.060 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:54.056+0000 7fb45318f640 -1 reset not still connected to 0x561355040c30 2024-09-11T06:39:54.067 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:54.064+0000 7fb45318f640 -1 reset not still connected to 0x561315ba4270 2024-09-11T06:39:54.068 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:54.064+0000 7fb45318f640 -1 reset not still connected to 0x561315ba5520 2024-09-11T06:39:54.068 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:54.064+0000 7fb45318f640 -1 reset not still connected to 0x56131a1ffe10 2024-09-11T06:39:54.074 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:54.068+0000 7fb45318f640 -1 reset not still connected to 0x561322931860 2024-09-11T06:39:54.074 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:54.068+0000 7fb45318f640 -1 reset not still connected to 0x561330ed0410 2024-09-11T06:39:54.074 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:54.068+0000 7fb45318f640 -1 reset not still connected to 0x56133339c410 2024-09-11T06:39:54.074 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:54.068+0000 7fb45318f640 -1 reset not still connected to 0x561341f69450 2024-09-11T06:39:54.074 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:54.068+0000 7fb45318f640 -1 reset not still connected to 0x5613480e41a0 2024-09-11T06:39:54.074 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:54.068+0000 7fb45318f640 -1 reset not still connected to 0x5613488b0820 2024-09-11T06:39:54.074 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:54.068+0000 7fb45318f640 -1 reset not still connected to 0x5613488b0d00 2024-09-11T06:39:54.074 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:54.068+0000 7fb45318f640 -1 reset not still connected to 0x56134d76fe10 2024-09-11T06:39:54.074 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:54.068+0000 7fb45318f640 -1 reset not still connected to 0x56134d81fd40 2024-09-11T06:39:54.074 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:54.068+0000 7fb45318f640 -1 reset not still connected to 0x561354d77110 2024-09-11T06:39:54.074 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:54.068+0000 7fb45318f640 -1 reset not still connected to 0x561355040c30 2024-09-11T06:39:54.185 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:39:54.187 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror image_2 2024-09-11T06:39:54.187 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:39:54.187 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:39:54.187 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_2 2024-09-11T06:39:54.187 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror image_2 '' 2024-09-11T06:39:54.187 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:39:54.187 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:39:54.187 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_2 2024-09-11T06:39:54.187 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:39:54.188 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-image_2.status 2024-09-11T06:39:54.188 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-image_2.status 2024-09-11T06:39:54.188 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:39:54.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_2.status 2024-09-11T06:39:54.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_2 2024-09-11T06:39:54.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_2.status 2024-09-11T06:39:54.237 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 4 2024-09-11T06:39:54.237 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 4 2024-09-11T06:39:54.237 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:39:54.238 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=1], [object_number=16, tag_tid=2, entry_tid=0], [object_number=18, tag_tid=1, entry_tid=15422], [object_number=19, tag_tid=1, entry_tid=15419]]], state=connected] 2024-09-11T06:39:54.238 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=1], [object_number=16, tag_tid=2, entry_tid=0], [object_number=18, tag_tid=1, entry_tid=15422], [object_number=19, tag_tid=1, entry_tid=15419]]], state=connected] 2024-09-11T06:39:54.240 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_2.status 2024-09-11T06:39:54.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=17, tag_tid=2, entry_tid=1' 2024-09-11T06:39:54.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror image_2 2024-09-11T06:39:54.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:39:54.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:39:54.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_2 2024-09-11T06:39:54.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror image_2 '..*' 2024-09-11T06:39:54.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:39:54.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:39:54.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_2 2024-09-11T06:39:54.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:39:54.244 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-image_2.status 2024-09-11T06:39:54.244 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-image_2.status 2024-09-11T06:39:54.244 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:39:54.244 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_2.status 2024-09-11T06:39:54.244 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_2 2024-09-11T06:39:54.244 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_2.status 2024-09-11T06:39:54.295 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 4 2024-09-11T06:39:54.296 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 4 2024-09-11T06:39:54.296 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:39:54.296 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=1], [object_number=16, tag_tid=2, entry_tid=0], [object_number=18, tag_tid=1, entry_tid=15422], [object_number=19, tag_tid=1, entry_tid=15419]]], state=connected] 2024-09-11T06:39:54.296 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=17, tag_tid=2, entry_tid=1], [object_number=16, tag_tid=2, entry_tid=0], [object_number=18, tag_tid=1, entry_tid=15422], [object_number=19, tag_tid=1, entry_tid=15419]]], state=connected] 2024-09-11T06:39:54.319 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_2.status 2024-09-11T06:39:54.321 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=17, tag_tid=2, entry_tid=1' 2024-09-11T06:39:54.321 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=17, tag_tid=2, entry_tid=1' -a 'object_number=17, tag_tid=2, entry_tid=1' = 'object_number=17, tag_tid=2, entry_tid=1' 2024-09-11T06:39:54.322 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:39:54.322 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_snap_present cluster1 mirror image_2 snap 2024-09-11T06:39:54.322 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:39:54.322 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:39:54.322 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_2 2024-09-11T06:39:54.322 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:39:54.322 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:39:54.322 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T06:39:54.322 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:39:55.322 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 info mirror/image_2@snap 2024-09-11T06:39:55.388 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:rbd image 'image_2': 2024-09-11T06:39:55.389 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: size 128 MiB in 32 objects 2024-09-11T06:39:55.389 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: order 22 (4 MiB objects) 2024-09-11T06:39:55.389 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: snapshot_count: 1 2024-09-11T06:39:55.389 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: id: 10a6d2a26591 2024-09-11T06:39:55.389 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: block_name_prefix: rbd_data.10a6d2a26591 2024-09-11T06:39:55.389 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: format: 2 2024-09-11T06:39:55.389 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: features: layering, exclusive-lock, journaling 2024-09-11T06:39:55.389 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: op_features: 2024-09-11T06:39:55.389 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: flags: 2024-09-11T06:39:55.389 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: create_timestamp: Wed Sep 11 06:31:05 2024 2024-09-11T06:39:55.389 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: access_timestamp: Wed Sep 11 06:31:05 2024 2024-09-11T06:39:55.390 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: modify_timestamp: Wed Sep 11 06:32:05 2024 2024-09-11T06:39:55.390 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: protected: False 2024-09-11T06:39:55.390 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: journal: 10a6d2a26591 2024-09-11T06:39:55.390 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring state: enabled 2024-09-11T06:39:55.390 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring mode: journal 2024-09-11T06:39:55.390 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring global id: 0a0dd68e-8a5f-4ea9-822f-272c5cb50f3c 2024-09-11T06:39:55.390 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring primary: false 2024-09-11T06:39:55.393 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:39:55.393 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ compare_image_snaps mirror image_2 snap 2024-09-11T06:39:55.393 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:39:55.393 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_2 2024-09-11T06:39:55.394 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:39:55.394 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local ret=0 2024-09-11T06:39:55.394 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local rmt_export=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_2.export 2024-09-11T06:39:55.394 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loc_export=/tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_2.export 2024-09-11T06:39:55.394 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_2.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_2.export 2024-09-11T06:39:55.394 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror export image_2@snap /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_2.export 2024-09-11T06:39:55.449 INFO:tasks.ceph.cluster2.osd.1.smithi112.stderr:2024-09-11T06:39:55.450+0000 7f2cecaac640 -1 reset not still connected to 0x55da97fd9ba0 2024-09-11T06:39:55.449 INFO:tasks.ceph.cluster2.osd.1.smithi112.stderr:2024-09-11T06:39:55.450+0000 7f2cecaac640 -1 reset not still connected to 0x55daa159cf70 2024-09-11T06:39:55.449 INFO:tasks.ceph.cluster2.osd.1.smithi112.stderr:2024-09-11T06:39:55.450+0000 7f2cecaac640 -1 reset not still connected to 0x55daaabf31e0 2024-09-11T06:39:55.449 INFO:tasks.ceph.cluster2.osd.1.smithi112.stderr:2024-09-11T06:39:55.450+0000 7f2cecaac640 -1 reset not still connected to 0x55daaf92cdd0 2024-09-11T06:39:55.449 INFO:tasks.ceph.cluster2.osd.1.smithi112.stderr:2024-09-11T06:39:55.450+0000 7f2cecaac640 -1 reset not still connected to 0x55dab8b8c270 2024-09-11T06:39:55.621 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T06:39:55.628 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 -p mirror export image_2@snap /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_2.export 2024-09-11T06:39:55.785 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:55.780+0000 7fb45318f640 -1 reset not still connected to 0x561322165790 2024-09-11T06:39:55.785 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:55.780+0000 7fb45318f640 -1 reset not still connected to 0x5613224a2d00 2024-09-11T06:39:55.785 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:55.780+0000 7fb45318f640 -1 reset not still connected to 0x5613225fd930 2024-09-11T06:39:55.785 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:55.780+0000 7fb45318f640 -1 reset not still connected to 0x561322bdf450 2024-09-11T06:39:55.790 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:55.788+0000 7fb45318f640 -1 reset not still connected to 0x5613237a41a0 2024-09-11T06:39:55.790 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:55.788+0000 7fb45318f640 -1 reset not still connected to 0x5613273cfd40 2024-09-11T06:39:55.790 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:55.788+0000 7fb45318f640 -1 reset not still connected to 0x561328b5c680 2024-09-11T06:39:55.790 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:55.788+0000 7fb45318f640 -1 reset not still connected to 0x561328b5c8f0 2024-09-11T06:39:55.790 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:55.788+0000 7fb45318f640 -1 reset not still connected to 0x56132e90e340 2024-09-11T06:39:55.790 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:55.788+0000 7fb45318f640 -1 reset not still connected to 0x561333ca9450 2024-09-11T06:39:55.790 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:55.788+0000 7fb45318f640 -1 reset not still connected to 0x561337a04410 2024-09-11T06:39:55.790 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:55.788+0000 7fb45318f640 -1 reset not still connected to 0x561337b67e10 2024-09-11T06:39:55.790 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:55.788+0000 7fb45318f640 -1 reset not still connected to 0x56133cabc000 2024-09-11T06:39:55.790 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:55.788+0000 7fb45318f640 -1 reset not still connected to 0x561342690270 2024-09-11T06:39:55.791 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:55.788+0000 7fb45318f640 -1 reset not still connected to 0x5613480c0a90 2024-09-11T06:39:55.791 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:55.788+0000 7fb45318f640 -1 reset not still connected to 0x5613480c1040 2024-09-11T06:39:55.791 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:55.788+0000 7fb45318f640 -1 reset not still connected to 0x56134f7f48f0 2024-09-11T06:39:55.791 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:55.788+0000 7fb45318f640 -1 reset not still connected to 0x5613545c3790 2024-09-11T06:39:55.791 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:39:55.788+0000 7fb45318f640 -1 reset not still connected to 0x5613551aac30 2024-09-11T06:39:55.976 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T06:39:55.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmp /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_2.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_2.export 2024-09-11T06:39:56.084 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_2.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_2.export 2024-09-11T06:39:56.120 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:39:56.120 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:39:56.120 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_3 2024-09-11T06:39:56.120 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_snap cluster2 mirror image_3 snap 2024-09-11T06:39:56.120 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:39:56.120 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:39:56.120 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_3 2024-09-11T06:39:56.120 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:39:56.120 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror snap create image_3@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T06:39:58.111 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:58.108+0000 7fb2ca654640 -1 reset not still connected to 0x55ed44536340 2024-09-11T06:39:58.112 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:58.108+0000 7fb2ca654640 -1 reset not still connected to 0x55ed4b336c30 2024-09-11T06:39:58.117 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:58.116+0000 7fb2ca654640 -1 reset not still connected to 0x55ed4b37ca90 2024-09-11T06:39:59.046 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:59.040+0000 7fb2ca654640 -1 reset not still connected to 0x55ed0e363ee0 2024-09-11T06:39:59.052 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:59.048+0000 7fb2ca654640 -1 reset not still connected to 0x55ed15c36d00 2024-09-11T06:39:59.053 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:59.048+0000 7fb2ca654640 -1 reset not still connected to 0x55ed169d91e0 2024-09-11T06:39:59.053 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:59.052+0000 7fb2ca654640 -1 reset not still connected to 0x55ed1b91f450 2024-09-11T06:39:59.057 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:59.056+0000 7fb2ca654640 -1 reset not still connected to 0x55ed288cd110 2024-09-11T06:39:59.058 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:59.056+0000 7fb2ca654640 -1 reset not still connected to 0x55ed32b6bba0 2024-09-11T06:39:59.058 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:59.056+0000 7fb2ca654640 -1 reset not still connected to 0x55ed369ccc30 2024-09-11T06:39:59.064 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:59.060+0000 7fb2ca654640 -1 reset not still connected to 0x55ed3a812820 2024-09-11T06:39:59.065 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:59.060+0000 7fb2ca654640 -1 reset not still connected to 0x55ed3ecf6b60 2024-09-11T06:39:59.065 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:59.060+0000 7fb2ca654640 -1 reset not still connected to 0x55ed3ecf76c0 2024-09-11T06:39:59.065 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:59.060+0000 7fb2ca654640 -1 reset not still connected to 0x55ed43611450 2024-09-11T06:39:59.065 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:59.060+0000 7fb2ca654640 -1 reset not still connected to 0x55ed436c8750 2024-09-11T06:39:59.065 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:59.060+0000 7fb2ca654640 -1 reset not still connected to 0x55ed44536340 2024-09-11T06:39:59.065 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:59.060+0000 7fb2ca654640 -1 reset not still connected to 0x55ed44538a90 2024-09-11T06:39:59.069 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:59.064+0000 7fb2ca654640 -1 reset not still connected to 0x55ed4b336c30 2024-09-11T06:39:59.069 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:39:59.064+0000 7fb2ca654640 -1 reset not still connected to 0x55ed4b37ca90 2024-09-11T06:39:59.637 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_started cluster1 mirror image_3 2024-09-11T06:39:59.637 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:39:59.637 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:39:59.638 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_3 2024-09-11T06:39:59.638 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_state cluster1 mirror image_3 started 2024-09-11T06:39:59.638 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:39:59.638 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:39:59.638 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_3 2024-09-11T06:39:59.638 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=started 2024-09-11T06:39:59.638 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:39:59.638 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:39:59.638 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:40:00.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_replay_state cluster1 mirror image_3 started 2024-09-11T06:40:00.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:40:00.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:40:00.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_3 2024-09-11T06:40:00.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=started 2024-09-11T06:40:00.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local status_result 2024-09-11T06:40:00.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=stopped 2024-09-11T06:40:00.640 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_3 2024-09-11T06:40:00.640 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -i state 2024-09-11T06:40:00.640 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_instance=cluster1 2024-09-11T06:40:00.640 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:40:00.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:40:00.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance=cluster1 2024-09-11T06:40:00.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local loop_instance 2024-09-11T06:40:00.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:40:00.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sleep 0 2024-09-11T06:40:00.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:40:00.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ seq 0 1 2024-09-11T06:40:00.646 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:40:00.646 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_3 2024-09-11T06:40:00.646 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1:0 2024-09-11T06:40:00.646 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:40:00.646 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:40:00.646 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:40:00.646 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:40:00.646 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=cluster 2024-09-11T06:40:00.646 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:40:00.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:40:00.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:40:00.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:40:00.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval cluster=cluster1 2024-09-11T06:40:00.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:40:00.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:40:00.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:40:00.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:40:00.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local local_cluster=cluster1:0 2024-09-11T06:40:00.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster=cluster1 2024-09-11T06:40:00.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance 2024-09-11T06:40:00.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:40:00.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local val=cluster1:0 2024-09-11T06:40:00.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T06:40:00.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance_var_name=instance 2024-09-11T06:40:00.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:40:00.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:40:00.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T06:40:00.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval local_cluster=cluster1 2024-09-11T06:40:00.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ local_cluster=cluster1 2024-09-11T06:40:00.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval instance=0 2024-09-11T06:40:00.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ instance=0 2024-09-11T06:40:00.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:40:00.672 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:40:00.672 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:40:00.672 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:40:00.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_3 2024-09-11T06:40:00.735 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:40:00.732+0000 7fb2ca654640 -1 reset not still connected to 0x55ed0e550dd0 2024-09-11T06:40:00.739 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:40:00.732+0000 7fb2ca654640 -1 reset not still connected to 0x55ed12570820 2024-09-11T06:40:00.739 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:40:00.732+0000 7fb2ca654640 -1 reset not still connected to 0x55ed149fe8f0 2024-09-11T06:40:00.739 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:40:00.732+0000 7fb2ca654640 -1 reset not still connected to 0x55ed161c2000 2024-09-11T06:40:00.739 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:40:00.732+0000 7fb2ca654640 -1 reset not still connected to 0x55ed194eaea0 2024-09-11T06:40:00.739 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:40:00.732+0000 7fb2ca654640 -1 reset not still connected to 0x55ed2a1e0ea0 2024-09-11T06:40:00.739 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:40:00.736+0000 7fb2ca654640 -1 reset not still connected to 0x55ed3d399d40 2024-09-11T06:40:00.745 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:40:00.740+0000 7fb2ca654640 -1 reset not still connected to 0x55ed50ca3380 2024-09-11T06:40:00.817 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ return 0 2024-09-11T06:40:00.818 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T06:40:00.818 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ grep -i Replaying 2024-09-11T06:40:00.818 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ echo ' "state": "Replaying"' 2024-09-11T06:40:00.819 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: "state": "Replaying" 2024-09-11T06:40:00.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_state=started 2024-09-11T06:40:00.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test started = started 2024-09-11T06:40:00.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:40:00.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_3 2024-09-11T06:40:00.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:40:00.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:40:00.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:40:00.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_3 2024-09-11T06:40:00.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' journal = journal ']' 2024-09-11T06:40:00.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_3 2024-09-11T06:40:00.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:40:00.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:40:00.822 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:40:00.822 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_3 2024-09-11T06:40:00.822 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-11T06:40:00.822 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-11T06:40:00.822 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:40:00.822 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:40:00.822 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:40:00.972 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:40:00.968+0000 7fb2ca654640 -1 reset not still connected to 0x55ed0d8cd5f0 2024-09-11T06:40:00.972 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:40:00.968+0000 7fb2ca654640 -1 reset not still connected to 0x55ed0e550dd0 2024-09-11T06:40:00.972 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:40:00.968+0000 7fb2ca654640 -1 reset not still connected to 0x55ed149fe8f0 2024-09-11T06:40:00.972 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:40:00.968+0000 7fb2ca654640 -1 reset not still connected to 0x55ed161c2000 2024-09-11T06:40:00.972 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:40:00.968+0000 7fb2ca654640 -1 reset not still connected to 0x55ed194eaea0 2024-09-11T06:40:00.980 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:40:00.976+0000 7fb2ca654640 -1 reset not still connected to 0x55ed1bd1add0 2024-09-11T06:40:00.980 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:40:00.976+0000 7fb2ca654640 -1 reset not still connected to 0x55ed2a1e0ea0 2024-09-11T06:40:00.981 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:40:00.976+0000 7fb2ca654640 -1 reset not still connected to 0x55ed38d48270 2024-09-11T06:40:00.981 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:40:00.976+0000 7fb2ca654640 -1 reset not still connected to 0x55ed3d399d40 2024-09-11T06:40:00.983 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:40:00.980+0000 7fb2ca654640 -1 reset not still connected to 0x55ed43824c30 2024-09-11T06:40:00.984 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:40:00.980+0000 7fb2ca654640 -1 reset not still connected to 0x55ed50ca3380 2024-09-11T06:40:01.022 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror image_3 2024-09-11T06:40:01.022 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:40:01.023 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:40:01.023 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_3 2024-09-11T06:40:01.023 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:40:01.023 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n image_3 ']' 2024-09-11T06:40:01.023 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/image_3' 2024-09-11T06:40:01.023 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_3 2024-09-11T06:40:01.023 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:40:01.023 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:40:01.023 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:40:01.023 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:40:01.023 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:40:01.023 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:40:01.023 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:40:01.024 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:40:01.024 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:40:01.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:40:01.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_3 2024-09-11T06:40:01.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:40:01.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:40:01.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:40:01.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:40:01.028 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:40:01.028 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:40:01.028 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:40:01.028 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:40:01.028 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:40:01.028 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:40:01.028 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:40:01.028 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:40:01.028 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:40:01.028 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:40:01.030 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:40:01.030 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:40:01.030 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:40:01.030 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:40:01.030 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:40:01.030 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:40:01.030 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:40:01.031 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:40:01.031 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:40:01.031 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:40:01.031 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:40:01.031 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:40:01.031 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:40:01.031 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:40:01.031 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:40:01.031 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:40:01.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:40:01.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:40:01.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:40:01.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_3 2024-09-11T06:40:01.200 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:40:01.201 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror image_3 2024-09-11T06:40:01.202 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:40:01.202 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:40:01.202 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_3 2024-09-11T06:40:01.202 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror image_3 '' 2024-09-11T06:40:01.202 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:40:01.202 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:40:01.202 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_3 2024-09-11T06:40:01.202 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:40:01.202 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-image_3.status 2024-09-11T06:40:01.202 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-image_3.status 2024-09-11T06:40:01.203 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:40:01.203 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_3.status 2024-09-11T06:40:01.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_3 2024-09-11T06:40:01.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_3.status 2024-09-11T06:40:01.250 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 2 2024-09-11T06:40:01.250 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 2 2024-09-11T06:40:01.250 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:40:01.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9647], [object_number=10, tag_tid=1, entry_tid=9646]]], state=connected] 2024-09-11T06:40:01.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9647], [object_number=10, tag_tid=1, entry_tid=9646]]], state=connected] 2024-09-11T06:40:01.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_3.status 2024-09-11T06:40:01.255 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-09-11T06:40:01.256 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror image_3 2024-09-11T06:40:01.256 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:40:01.256 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:40:01.257 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_3 2024-09-11T06:40:01.257 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror image_3 '..*' 2024-09-11T06:40:01.257 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:40:01.257 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:40:01.257 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_3 2024-09-11T06:40:01.257 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:40:01.257 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-image_3.status 2024-09-11T06:40:01.257 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-image_3.status 2024-09-11T06:40:01.257 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:40:01.258 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_3.status 2024-09-11T06:40:01.258 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_3 2024-09-11T06:40:01.258 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_3.status 2024-09-11T06:40:01.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 2 2024-09-11T06:40:01.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 2 2024-09-11T06:40:01.305 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:40:01.305 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9647], [object_number=10, tag_tid=1, entry_tid=9646]]], state=connected] 2024-09-11T06:40:01.305 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=9647], [object_number=10, tag_tid=1, entry_tid=9646]]], state=connected] 2024-09-11T06:40:01.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_3.status 2024-09-11T06:40:01.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-09-11T06:40:01.310 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=1' 2024-09-11T06:40:01.310 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:40:01.310 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_snap_present cluster1 mirror image_3 snap 2024-09-11T06:40:01.310 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:40:01.310 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:40:01.310 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_3 2024-09-11T06:40:01.310 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:40:01.310 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:40:01.310 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T06:40:01.311 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:40:01.614 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:40:01.608+0000 7fb45318f640 -1 reset not still connected to 0x561322165790 2024-09-11T06:40:01.614 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:40:01.608+0000 7fb45318f640 -1 reset not still connected to 0x5613224a2d00 2024-09-11T06:40:01.614 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:40:01.608+0000 7fb45318f640 -1 reset not still connected to 0x561322bdf450 2024-09-11T06:40:01.614 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:40:01.608+0000 7fb45318f640 -1 reset not still connected to 0x5613237a41a0 2024-09-11T06:40:01.614 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:40:01.608+0000 7fb45318f640 -1 reset not still connected to 0x5613273cfd40 2024-09-11T06:40:01.617 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:40:01.612+0000 7fb45318f640 -1 reset not still connected to 0x561328b5c680 2024-09-11T06:40:01.617 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:40:01.612+0000 7fb45318f640 -1 reset not still connected to 0x56132e90e340 2024-09-11T06:40:01.625 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:40:01.620+0000 7fb45318f640 -1 reset not still connected to 0x561333ca9450 2024-09-11T06:40:01.625 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:40:01.620+0000 7fb45318f640 -1 reset not still connected to 0x561337a04410 2024-09-11T06:40:01.625 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:40:01.620+0000 7fb45318f640 -1 reset not still connected to 0x561337b67e10 2024-09-11T06:40:01.625 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:40:01.620+0000 7fb45318f640 -1 reset not still connected to 0x56133cabc000 2024-09-11T06:40:01.625 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:40:01.620+0000 7fb45318f640 -1 reset not still connected to 0x5613480c0a90 2024-09-11T06:40:01.625 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:40:01.620+0000 7fb45318f640 -1 reset not still connected to 0x5613480c1040 2024-09-11T06:40:01.625 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:40:01.620+0000 7fb45318f640 -1 reset not still connected to 0x56134f7f48f0 2024-09-11T06:40:01.625 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:40:01.620+0000 7fb45318f640 -1 reset not still connected to 0x5613551aac30 2024-09-11T06:40:01.902 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:40:01.896+0000 7fb2ca654640 -1 reset not still connected to 0x55ed0e550dd0 2024-09-11T06:40:01.903 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:40:01.896+0000 7fb2ca654640 -1 reset not still connected to 0x55ed12570820 2024-09-11T06:40:01.903 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:40:01.896+0000 7fb2ca654640 -1 reset not still connected to 0x55ed149fe8f0 2024-09-11T06:40:01.903 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:40:01.896+0000 7fb2ca654640 -1 reset not still connected to 0x55ed161c2000 2024-09-11T06:40:01.903 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:40:01.896+0000 7fb2ca654640 -1 reset not still connected to 0x55ed194eaea0 2024-09-11T06:40:01.906 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:40:01.904+0000 7fb2ca654640 -1 reset not still connected to 0x55ed2a1e0ea0 2024-09-11T06:40:01.906 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:40:01.904+0000 7fb2ca654640 -1 reset not still connected to 0x55ed3d399d40 2024-09-11T06:40:01.915 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:40:01.912+0000 7fb2ca654640 -1 reset not still connected to 0x55ed50ca3380 2024-09-11T06:40:02.097 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:40:02.092+0000 7fb45318f640 -1 reset not still connected to 0x561315ba4270 2024-09-11T06:40:02.097 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:40:02.092+0000 7fb45318f640 -1 reset not still connected to 0x561315ba5520 2024-09-11T06:40:02.098 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:40:02.096+0000 7fb45318f640 -1 reset not still connected to 0x56131a1ffe10 2024-09-11T06:40:02.098 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:40:02.096+0000 7fb45318f640 -1 reset not still connected to 0x561322931860 2024-09-11T06:40:02.100 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:40:02.096+0000 7fb45318f640 -1 reset not still connected to 0x561322ded930 2024-09-11T06:40:02.101 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:40:02.096+0000 7fb45318f640 -1 reset not still connected to 0x561330ed0410 2024-09-11T06:40:02.102 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:40:02.100+0000 7fb45318f640 -1 reset not still connected to 0x56133339c410 2024-09-11T06:40:02.105 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:40:02.100+0000 7fb45318f640 -1 reset not still connected to 0x56133c1909c0 2024-09-11T06:40:02.108 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:40:02.104+0000 7fb45318f640 -1 reset not still connected to 0x561341f69450 2024-09-11T06:40:02.108 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:40:02.104+0000 7fb45318f640 -1 reset not still connected to 0x561342043930 2024-09-11T06:40:02.108 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:40:02.104+0000 7fb45318f640 -1 reset not still connected to 0x5613480e41a0 2024-09-11T06:40:02.109 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:40:02.104+0000 7fb45318f640 -1 reset not still connected to 0x5613488b0820 2024-09-11T06:40:02.109 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:40:02.104+0000 7fb45318f640 -1 reset not still connected to 0x5613488b0d00 2024-09-11T06:40:02.109 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:40:02.104+0000 7fb45318f640 -1 reset not still connected to 0x56134d76fe10 2024-09-11T06:40:02.109 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:40:02.104+0000 7fb45318f640 -1 reset not still connected to 0x56134d81fd40 2024-09-11T06:40:02.109 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:40:02.104+0000 7fb45318f640 -1 reset not still connected to 0x561354d77110 2024-09-11T06:40:02.111 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:40:02.108+0000 7fb45318f640 -1 reset not still connected to 0x561355040c30 2024-09-11T06:40:02.111 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:40:02.108+0000 7fb45318f640 -1 reset not still connected to 0x5613551fd110 2024-09-11T06:40:02.310 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 info mirror/image_3@snap 2024-09-11T06:40:02.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:rbd image 'image_3': 2024-09-11T06:40:02.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: size 128 MiB in 32 objects 2024-09-11T06:40:02.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: order 22 (4 MiB objects) 2024-09-11T06:40:02.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: snapshot_count: 1 2024-09-11T06:40:02.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: id: 10a6c13e3910 2024-09-11T06:40:02.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: block_name_prefix: rbd_data.10a6c13e3910 2024-09-11T06:40:02.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: format: 2 2024-09-11T06:40:02.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: features: layering, exclusive-lock, journaling 2024-09-11T06:40:02.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: op_features: 2024-09-11T06:40:02.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: flags: 2024-09-11T06:40:02.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: create_timestamp: Wed Sep 11 06:31:19 2024 2024-09-11T06:40:02.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: access_timestamp: Wed Sep 11 06:31:19 2024 2024-09-11T06:40:02.372 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: modify_timestamp: Wed Sep 11 06:31:19 2024 2024-09-11T06:40:02.372 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: protected: False 2024-09-11T06:40:02.372 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: journal: 10a6c13e3910 2024-09-11T06:40:02.372 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring state: enabled 2024-09-11T06:40:02.372 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring mode: journal 2024-09-11T06:40:02.372 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring global id: 14e53f66-4e3b-419a-a706-dd27ffb309f2 2024-09-11T06:40:02.372 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring primary: false 2024-09-11T06:40:02.375 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:40:02.375 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ compare_image_snaps mirror image_3 snap 2024-09-11T06:40:02.375 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:40:02.375 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_3 2024-09-11T06:40:02.375 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:40:02.376 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local ret=0 2024-09-11T06:40:02.376 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local rmt_export=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_3.export 2024-09-11T06:40:02.376 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loc_export=/tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_3.export 2024-09-11T06:40:02.376 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_3.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_3.export 2024-09-11T06:40:02.376 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror export image_3@snap /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_3.export 2024-09-11T06:40:02.569 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T06:40:02.578 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 -p mirror export image_3@snap /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_3.export 2024-09-11T06:40:02.921 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T06:40:02.934 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmp /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_3.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_3.export 2024-09-11T06:40:03.013 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_3.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_3.export 2024-09-11T06:40:03.057 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:40:03.057 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:40:03.057 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_4 2024-09-11T06:40:03.057 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_snap cluster2 mirror image_4 snap 2024-09-11T06:40:03.057 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:40:03.057 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:40:03.057 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_4 2024-09-11T06:40:03.057 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:40:03.058 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror snap create image_4@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T06:40:03.681 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_started cluster1 mirror image_4 2024-09-11T06:40:03.681 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:40:03.681 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:40:03.681 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_4 2024-09-11T06:40:03.681 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_state cluster1 mirror image_4 started 2024-09-11T06:40:03.681 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:40:03.681 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:40:03.681 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_4 2024-09-11T06:40:03.681 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=started 2024-09-11T06:40:03.681 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:40:03.682 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:40:03.682 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:40:04.682 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_replay_state cluster1 mirror image_4 started 2024-09-11T06:40:04.682 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:40:04.683 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:40:04.683 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_4 2024-09-11T06:40:04.683 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=started 2024-09-11T06:40:04.683 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local status_result 2024-09-11T06:40:04.683 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=stopped 2024-09-11T06:40:04.684 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_4 2024-09-11T06:40:04.684 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_instance=cluster1 2024-09-11T06:40:04.684 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:40:04.684 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:40:04.684 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance=cluster1 2024-09-11T06:40:04.684 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local loop_instance 2024-09-11T06:40:04.684 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:40:04.684 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sleep 0 2024-09-11T06:40:04.684 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -i state 2024-09-11T06:40:04.685 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:40:04.685 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ seq 0 1 2024-09-11T06:40:04.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:40:04.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_4 2024-09-11T06:40:04.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1:0 2024-09-11T06:40:04.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:40:04.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:40:04.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:40:04.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:40:04.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=cluster 2024-09-11T06:40:04.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:40:04.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:40:04.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:40:04.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:40:04.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval cluster=cluster1 2024-09-11T06:40:04.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:40:04.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:40:04.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:40:04.690 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:40:04.690 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local local_cluster=cluster1:0 2024-09-11T06:40:04.690 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster=cluster1 2024-09-11T06:40:04.690 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance 2024-09-11T06:40:04.690 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:40:04.690 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local val=cluster1:0 2024-09-11T06:40:04.690 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T06:40:04.690 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance_var_name=instance 2024-09-11T06:40:04.690 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:40:04.690 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:40:04.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T06:40:04.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval local_cluster=cluster1 2024-09-11T06:40:04.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ local_cluster=cluster1 2024-09-11T06:40:04.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval instance=0 2024-09-11T06:40:04.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ instance=0 2024-09-11T06:40:04.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:40:04.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:40:04.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:40:04.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:40:04.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_4 2024-09-11T06:40:04.860 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:no valid command found; 5 closest matches: 2024-09-11T06:40:04.860 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd mirror status mirror/image_41 2024-09-11T06:40:04.860 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd mirror status mirror/image_43 2024-09-11T06:40:04.860 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd mirror status mirror/image_45 2024-09-11T06:40:04.860 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd mirror status mirror/image_47 2024-09-11T06:40:04.860 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd mirror status mirror/image_49 2024-09-11T06:40:04.860 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:admin_socket: invalid command 2024-09-11T06:40:04.863 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:40:04.863 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_4 2024-09-11T06:40:04.863 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1:1 2024-09-11T06:40:04.864 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:40:04.864 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:40:04.864 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-11T06:40:04.864 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:1 2024-09-11T06:40:04.864 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=cluster 2024-09-11T06:40:04.864 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:40:04.864 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:40:04.864 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=1 2024-09-11T06:40:04.864 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T06:40:04.864 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval cluster=cluster1 2024-09-11T06:40:04.864 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:40:04.864 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=1 2024-09-11T06:40:04.864 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=1 2024-09-11T06:40:04.865 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-11T06:40:04.865 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local local_cluster=cluster1:1 2024-09-11T06:40:04.865 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster=cluster1 2024-09-11T06:40:04.865 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance 2024-09-11T06:40:04.865 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T06:40:04.865 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local val=cluster1:1 2024-09-11T06:40:04.866 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T06:40:04.866 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance_var_name=instance 2024-09-11T06:40:04.866 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:40:04.866 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=1 2024-09-11T06:40:04.866 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-11T06:40:04.866 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval local_cluster=cluster1 2024-09-11T06:40:04.866 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ local_cluster=cluster1 2024-09-11T06:40:04.867 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval instance=1 2024-09-11T06:40:04.867 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ instance=1 2024-09-11T06:40:04.867 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T06:40:04.890 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:40:04.890 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:40:04.891 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:40:04.891 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_4 2024-09-11T06:40:05.031 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ return 0 2024-09-11T06:40:05.032 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T06:40:05.032 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ echo ' "state": "Replaying"' 2024-09-11T06:40:05.032 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ grep -i Replaying 2024-09-11T06:40:05.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: "state": "Replaying" 2024-09-11T06:40:05.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_state=started 2024-09-11T06:40:05.035 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test started = started 2024-09-11T06:40:05.035 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:40:05.035 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_4 2024-09-11T06:40:05.035 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:40:05.035 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:40:05.035 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:40:05.035 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_4 2024-09-11T06:40:05.035 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' journal = journal ']' 2024-09-11T06:40:05.035 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_4 2024-09-11T06:40:05.035 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:40:05.035 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:40:05.035 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:40:05.035 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_4 2024-09-11T06:40:05.035 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-11T06:40:05.036 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-11T06:40:05.036 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:40:05.036 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:40:05.036 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:40:05.236 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror image_4 2024-09-11T06:40:05.236 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:40:05.236 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:40:05.236 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_4 2024-09-11T06:40:05.237 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:40:05.237 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n image_4 ']' 2024-09-11T06:40:05.237 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/image_4' 2024-09-11T06:40:05.237 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_4 2024-09-11T06:40:05.237 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:40:05.237 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:40:05.237 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:40:05.237 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:40:05.237 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:40:05.237 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:40:05.237 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:40:05.238 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:40:05.238 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:40:05.240 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:40:05.240 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_4 2024-09-11T06:40:05.240 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:40:05.240 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:40:05.240 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:40:05.240 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:40:05.240 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:40:05.240 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:40:05.240 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:40:05.240 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:40:05.240 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:40:05.240 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:40:05.240 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:40:05.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:40:05.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:40:05.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:40:05.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:40:05.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:40:05.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:40:05.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:40:05.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:40:05.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:40:05.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:40:05.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:40:05.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:40:05.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:40:05.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:40:05.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:40:05.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:40:05.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:40:05.244 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:40:05.244 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:40:05.270 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:40:05.270 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:40:05.270 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:40:05.270 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_4 2024-09-11T06:40:05.421 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:no valid command found; 5 closest matches: 2024-09-11T06:40:05.421 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd mirror flush mirror/image_41 2024-09-11T06:40:05.422 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd mirror flush mirror/image_43 2024-09-11T06:40:05.422 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd mirror flush mirror/image_45 2024-09-11T06:40:05.422 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd mirror flush mirror/image_47 2024-09-11T06:40:05.422 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd mirror flush mirror/image_49 2024-09-11T06:40:05.422 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:admin_socket: invalid command 2024-09-11T06:40:05.424 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:40:05.424 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_4 2024-09-11T06:40:05.424 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:1 2024-09-11T06:40:05.424 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:40:05.424 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:40:05.424 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-11T06:40:05.424 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:1 2024-09-11T06:40:05.424 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:40:05.424 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:40:05.424 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:40:05.424 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=1 2024-09-11T06:40:05.425 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-11T06:40:05.425 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:40:05.425 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:40:05.425 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=1 2024-09-11T06:40:05.425 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=1 2024-09-11T06:40:05.427 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-11T06:40:05.427 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:1 2024-09-11T06:40:05.427 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:40:05.427 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:40:05.427 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T06:40:05.428 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:1 2024-09-11T06:40:05.428 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:40:05.428 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:40:05.428 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:40:05.428 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=1 2024-09-11T06:40:05.428 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T06:40:05.429 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:40:05.429 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:40:05.429 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=1 2024-09-11T06:40:05.429 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=1 2024-09-11T06:40:05.429 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T06:40:05.450 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:40:05.451 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:40:05.451 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:40:05.451 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_4 2024-09-11T06:40:05.606 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:40:05.607 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror image_4 2024-09-11T06:40:05.607 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:40:05.607 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:40:05.607 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_4 2024-09-11T06:40:05.607 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror image_4 '' 2024-09-11T06:40:05.607 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:40:05.607 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:40:05.607 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_4 2024-09-11T06:40:05.607 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:40:05.607 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-image_4.status 2024-09-11T06:40:05.607 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-image_4.status 2024-09-11T06:40:05.608 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:40:05.610 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_4.status 2024-09-11T06:40:05.610 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_4 2024-09-11T06:40:05.610 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_4.status 2024-09-11T06:40:05.660 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 2 2024-09-11T06:40:05.660 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 2 2024-09-11T06:40:05.660 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:40:05.660 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=3, entry_tid=1], [object_number=8, tag_tid=3, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=7938], [object_number=11, tag_tid=1, entry_tid=7935]]], state=connected] 2024-09-11T06:40:05.660 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=9, tag_tid=3, entry_tid=1], [object_number=8, tag_tid=3, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=7938], [object_number=11, tag_tid=1, entry_tid=7935]]], state=connected] 2024-09-11T06:40:05.663 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_4.status 2024-09-11T06:40:05.665 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=9, tag_tid=3, entry_tid=1' 2024-09-11T06:40:05.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror image_4 2024-09-11T06:40:05.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:40:05.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:40:05.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_4 2024-09-11T06:40:05.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror image_4 '..*' 2024-09-11T06:40:05.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:40:05.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:40:05.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_4 2024-09-11T06:40:05.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:40:05.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-image_4.status 2024-09-11T06:40:05.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-image_4.status 2024-09-11T06:40:05.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:40:05.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_4.status 2024-09-11T06:40:05.668 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_4 2024-09-11T06:40:05.668 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_4.status 2024-09-11T06:40:05.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 2 2024-09-11T06:40:05.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 2 2024-09-11T06:40:05.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:40:05.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=3, entry_tid=1], [object_number=8, tag_tid=3, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=7938], [object_number=11, tag_tid=1, entry_tid=7935]]], state=connected] 2024-09-11T06:40:05.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=9, tag_tid=3, entry_tid=1], [object_number=8, tag_tid=3, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=7938], [object_number=11, tag_tid=1, entry_tid=7935]]], state=connected] 2024-09-11T06:40:05.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_4.status 2024-09-11T06:40:05.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=9, tag_tid=3, entry_tid=1' 2024-09-11T06:40:05.721 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=9, tag_tid=3, entry_tid=1' -a 'object_number=9, tag_tid=3, entry_tid=1' = 'object_number=9, tag_tid=3, entry_tid=1' 2024-09-11T06:40:05.721 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:40:05.721 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_snap_present cluster1 mirror image_4 snap 2024-09-11T06:40:05.721 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:40:05.721 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:40:05.721 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_4 2024-09-11T06:40:05.721 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:40:05.721 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:40:05.721 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T06:40:05.721 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:40:06.721 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 info mirror/image_4@snap 2024-09-11T06:40:06.790 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:rbd image 'image_4': 2024-09-11T06:40:06.790 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: size 128 MiB in 32 objects 2024-09-11T06:40:06.790 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: order 22 (4 MiB objects) 2024-09-11T06:40:06.790 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: snapshot_count: 1 2024-09-11T06:40:06.790 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: id: 10a9f0312846 2024-09-11T06:40:06.790 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: block_name_prefix: rbd_data.10a9f0312846 2024-09-11T06:40:06.791 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: format: 2 2024-09-11T06:40:06.791 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: features: layering, exclusive-lock, journaling 2024-09-11T06:40:06.791 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: op_features: 2024-09-11T06:40:06.791 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: flags: 2024-09-11T06:40:06.791 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: create_timestamp: Wed Sep 11 06:31:30 2024 2024-09-11T06:40:06.791 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: access_timestamp: Wed Sep 11 06:31:30 2024 2024-09-11T06:40:06.791 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: modify_timestamp: Wed Sep 11 06:32:30 2024 2024-09-11T06:40:06.791 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: protected: False 2024-09-11T06:40:06.792 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: journal: 10a9f0312846 2024-09-11T06:40:06.792 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring state: enabled 2024-09-11T06:40:06.792 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring mode: journal 2024-09-11T06:40:06.792 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring global id: 59b0717d-d8ce-4a66-813f-344d49279024 2024-09-11T06:40:06.792 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring primary: false 2024-09-11T06:40:06.794 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:40:06.795 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ compare_image_snaps mirror image_4 snap 2024-09-11T06:40:06.795 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:40:06.795 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_4 2024-09-11T06:40:06.795 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:40:06.795 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local ret=0 2024-09-11T06:40:06.795 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local rmt_export=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_4.export 2024-09-11T06:40:06.795 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loc_export=/tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_4.export 2024-09-11T06:40:06.795 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_4.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_4.export 2024-09-11T06:40:06.796 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror export image_4@snap /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_4.export 2024-09-11T06:40:06.999 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T06:40:07.007 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 -p mirror export image_4@snap /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_4.export 2024-09-11T06:40:07.420 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T06:40:07.435 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmp /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_4.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_4.export 2024-09-11T06:40:07.515 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_4.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_4.export 2024-09-11T06:40:07.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:40:07.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:40:07.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_5 2024-09-11T06:40:07.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_snap cluster2 mirror image_5 snap 2024-09-11T06:40:07.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:40:07.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:40:07.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_5 2024-09-11T06:40:07.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:40:07.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror snap create image_5@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T06:40:07.678 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:40:07.676+0000 7fb2ca654640 -1 reset not still connected to 0x55ed1b91f450 2024-09-11T06:40:07.678 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:40:07.676+0000 7fb2ca654640 -1 reset not still connected to 0x55ed1c2d45b0 2024-09-11T06:40:07.678 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:40:07.676+0000 7fb2ca654640 -1 reset not still connected to 0x55ed288cd110 2024-09-11T06:40:07.683 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:40:07.680+0000 7fb2ca654640 -1 reset not still connected to 0x55ed2b074410 2024-09-11T06:40:07.685 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:40:07.680+0000 7fb2ca654640 -1 reset not still connected to 0x55ed32b6bba0 2024-09-11T06:40:07.685 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:40:07.680+0000 7fb2ca654640 -1 reset not still connected to 0x55ed369ccc30 2024-09-11T06:40:07.685 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:40:07.680+0000 7fb2ca654640 -1 reset not still connected to 0x55ed3a812820 2024-09-11T06:40:07.685 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:40:07.680+0000 7fb2ca654640 -1 reset not still connected to 0x55ed3ecf6b60 2024-09-11T06:40:07.685 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:40:07.680+0000 7fb2ca654640 -1 reset not still connected to 0x55ed43611450 2024-09-11T06:40:07.687 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:40:07.684+0000 7fb2ca654640 -1 reset not still connected to 0x55ed436c8750 2024-09-11T06:40:07.687 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:40:07.684+0000 7fb2ca654640 -1 reset not still connected to 0x55ed44536340 2024-09-11T06:40:07.687 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:40:07.684+0000 7fb2ca654640 -1 reset not still connected to 0x55ed44538dd0 2024-09-11T06:40:07.687 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:40:07.684+0000 7fb2ca654640 -1 reset not still connected to 0x55ed4b336c30 2024-09-11T06:40:07.687 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:40:07.684+0000 7fb2ca654640 -1 reset not still connected to 0x55ed4b37ca90 2024-09-11T06:40:09.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_started cluster1 mirror image_5 2024-09-11T06:40:09.693 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:40:09.693 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:40:09.693 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_5 2024-09-11T06:40:09.693 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_state cluster1 mirror image_5 started 2024-09-11T06:40:09.693 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:40:09.694 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:40:09.694 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_5 2024-09-11T06:40:09.694 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=started 2024-09-11T06:40:09.694 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:40:09.694 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:40:09.694 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:40:10.694 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_replay_state cluster1 mirror image_5 started 2024-09-11T06:40:10.694 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:40:10.694 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:40:10.694 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_5 2024-09-11T06:40:10.694 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=started 2024-09-11T06:40:10.694 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local status_result 2024-09-11T06:40:10.694 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=stopped 2024-09-11T06:40:10.695 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_5 2024-09-11T06:40:10.695 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -i state 2024-09-11T06:40:10.695 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_instance=cluster1 2024-09-11T06:40:10.695 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:40:10.696 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:40:10.696 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance=cluster1 2024-09-11T06:40:10.696 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local loop_instance 2024-09-11T06:40:10.696 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:40:10.696 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sleep 0 2024-09-11T06:40:10.696 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:40:10.697 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ seq 0 1 2024-09-11T06:40:10.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:40:10.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_5 2024-09-11T06:40:10.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1:0 2024-09-11T06:40:10.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:40:10.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:40:10.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:40:10.700 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:40:10.700 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=cluster 2024-09-11T06:40:10.700 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:40:10.700 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:40:10.700 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:40:10.700 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:40:10.700 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval cluster=cluster1 2024-09-11T06:40:10.700 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:40:10.700 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:40:10.700 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:40:10.700 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:40:10.702 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local local_cluster=cluster1:0 2024-09-11T06:40:10.702 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster=cluster1 2024-09-11T06:40:10.702 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance 2024-09-11T06:40:10.702 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:40:10.702 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local val=cluster1:0 2024-09-11T06:40:10.702 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T06:40:10.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance_var_name=instance 2024-09-11T06:40:10.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:40:10.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:40:10.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T06:40:10.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval local_cluster=cluster1 2024-09-11T06:40:10.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ local_cluster=cluster1 2024-09-11T06:40:10.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval instance=0 2024-09-11T06:40:10.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ instance=0 2024-09-11T06:40:10.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:40:10.723 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:40:10.723 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:40:10.723 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:40:10.723 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_5 2024-09-11T06:40:10.868 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ return 0 2024-09-11T06:40:10.868 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T06:40:10.869 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ echo ' "state": "Replaying"' 2024-09-11T06:40:10.869 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ grep -i Replaying 2024-09-11T06:40:10.871 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: "state": "Replaying" 2024-09-11T06:40:10.873 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_state=started 2024-09-11T06:40:10.873 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test started = started 2024-09-11T06:40:10.873 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:40:10.873 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_5 2024-09-11T06:40:10.873 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:40:10.873 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:40:10.873 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:40:10.873 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_5 2024-09-11T06:40:10.873 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' journal = journal ']' 2024-09-11T06:40:10.873 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_5 2024-09-11T06:40:10.873 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:40:10.874 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:40:10.874 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:40:10.874 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_5 2024-09-11T06:40:10.874 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-11T06:40:10.874 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-11T06:40:10.874 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:40:10.874 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:40:10.874 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:40:11.074 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror image_5 2024-09-11T06:40:11.074 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:40:11.074 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:40:11.074 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_5 2024-09-11T06:40:11.074 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:40:11.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n image_5 ']' 2024-09-11T06:40:11.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/image_5' 2024-09-11T06:40:11.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_5 2024-09-11T06:40:11.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:40:11.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:40:11.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:40:11.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:40:11.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:40:11.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:40:11.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:40:11.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:40:11.078 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:40:11.078 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:40:11.078 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_5 2024-09-11T06:40:11.078 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:40:11.078 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:40:11.078 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:40:11.078 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:40:11.079 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:40:11.079 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:40:11.079 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:40:11.079 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:40:11.079 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:40:11.079 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:40:11.079 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:40:11.079 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:40:11.079 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:40:11.079 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:40:11.079 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:40:11.079 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:40:11.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:40:11.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:40:11.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:40:11.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:40:11.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:40:11.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:40:11.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:40:11.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:40:11.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:40:11.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:40:11.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:40:11.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:40:11.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:40:11.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:40:11.103 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:40:11.104 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:40:11.104 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:40:11.104 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_5 2024-09-11T06:40:11.248 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:40:11.250 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror image_5 2024-09-11T06:40:11.250 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:40:11.250 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:40:11.250 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_5 2024-09-11T06:40:11.250 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror image_5 '' 2024-09-11T06:40:11.250 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:40:11.250 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:40:11.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_5 2024-09-11T06:40:11.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:40:11.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-image_5.status 2024-09-11T06:40:11.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-image_5.status 2024-09-11T06:40:11.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:40:11.252 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_5.status 2024-09-11T06:40:11.252 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_5 2024-09-11T06:40:11.252 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_5.status 2024-09-11T06:40:11.297 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 2 2024-09-11T06:40:11.297 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 2 2024-09-11T06:40:11.297 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:40:11.298 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=8735], [object_number=10, tag_tid=1, entry_tid=8734]]], state=connected] 2024-09-11T06:40:11.298 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=8735], [object_number=10, tag_tid=1, entry_tid=8734]]], state=connected] 2024-09-11T06:40:11.300 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_5.status 2024-09-11T06:40:11.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-09-11T06:40:11.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror image_5 2024-09-11T06:40:11.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:40:11.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:40:11.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_5 2024-09-11T06:40:11.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror image_5 '..*' 2024-09-11T06:40:11.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:40:11.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:40:11.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_5 2024-09-11T06:40:11.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:40:11.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-image_5.status 2024-09-11T06:40:11.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-image_5.status 2024-09-11T06:40:11.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:40:11.305 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_5.status 2024-09-11T06:40:11.305 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_5 2024-09-11T06:40:11.305 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_5.status 2024-09-11T06:40:11.351 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 2 2024-09-11T06:40:11.351 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 2 2024-09-11T06:40:11.351 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:40:11.351 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=8735], [object_number=10, tag_tid=1, entry_tid=8734]]], state=connected] 2024-09-11T06:40:11.351 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=8735], [object_number=10, tag_tid=1, entry_tid=8734]]], state=connected] 2024-09-11T06:40:11.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_5.status 2024-09-11T06:40:11.357 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-09-11T06:40:11.357 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=1' 2024-09-11T06:40:11.357 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:40:11.357 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_snap_present cluster1 mirror image_5 snap 2024-09-11T06:40:11.357 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:40:11.357 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:40:11.357 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_5 2024-09-11T06:40:11.357 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:40:11.357 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:40:11.357 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T06:40:11.358 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:40:11.566 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:40:11.559+0000 7fb45318f640 -1 reset not still connected to 0x561315ba4270 2024-09-11T06:40:11.566 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:40:11.559+0000 7fb45318f640 -1 reset not still connected to 0x56131a1ffe10 2024-09-11T06:40:11.570 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:40:11.567+0000 7fb45318f640 -1 reset not still connected to 0x561322931860 2024-09-11T06:40:11.570 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:40:11.567+0000 7fb45318f640 -1 reset not still connected to 0x561330ed0410 2024-09-11T06:40:11.570 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:40:11.567+0000 7fb45318f640 -1 reset not still connected to 0x56134d81fd40 2024-09-11T06:40:12.357 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 info mirror/image_5@snap 2024-09-11T06:40:12.420 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:rbd image 'image_5': 2024-09-11T06:40:12.420 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: size 128 MiB in 32 objects 2024-09-11T06:40:12.420 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: order 22 (4 MiB objects) 2024-09-11T06:40:12.420 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: snapshot_count: 1 2024-09-11T06:40:12.420 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: id: 10a67eebf65a 2024-09-11T06:40:12.420 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: block_name_prefix: rbd_data.10a67eebf65a 2024-09-11T06:40:12.420 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: format: 2 2024-09-11T06:40:12.420 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: features: layering, exclusive-lock, journaling 2024-09-11T06:40:12.421 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: op_features: 2024-09-11T06:40:12.421 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: flags: 2024-09-11T06:40:12.421 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: create_timestamp: Wed Sep 11 06:31:38 2024 2024-09-11T06:40:12.421 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: access_timestamp: Wed Sep 11 06:31:38 2024 2024-09-11T06:40:12.421 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: modify_timestamp: Wed Sep 11 06:31:38 2024 2024-09-11T06:40:12.421 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: protected: False 2024-09-11T06:40:12.421 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: journal: 10a67eebf65a 2024-09-11T06:40:12.421 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring state: enabled 2024-09-11T06:40:12.421 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring mode: journal 2024-09-11T06:40:12.421 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring global id: 67988ec8-7ef8-4160-ae37-1b46d8d53dc5 2024-09-11T06:40:12.421 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring primary: false 2024-09-11T06:40:12.425 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:40:12.425 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ compare_image_snaps mirror image_5 snap 2024-09-11T06:40:12.425 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:40:12.425 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_5 2024-09-11T06:40:12.425 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:40:12.425 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local ret=0 2024-09-11T06:40:12.425 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local rmt_export=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_5.export 2024-09-11T06:40:12.425 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loc_export=/tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_5.export 2024-09-11T06:40:12.425 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_5.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_5.export 2024-09-11T06:40:12.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror export image_5@snap /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_5.export 2024-09-11T06:40:12.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T06:40:12.651 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 -p mirror export image_5@snap /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_5.export 2024-09-11T06:40:12.935 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T06:40:12.948 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmp /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_5.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_5.export 2024-09-11T06:40:13.025 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_5.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_5.export 2024-09-11T06:40:13.067 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:40:13.068 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:40:13.068 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_6 2024-09-11T06:40:13.068 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_snap cluster2 mirror image_6 snap 2024-09-11T06:40:13.068 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:40:13.068 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:40:13.068 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_6 2024-09-11T06:40:13.068 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:40:13.068 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror snap create image_6@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T06:40:13.351 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:40:13.347+0000 7fb2ca654640 -1 reset not still connected to 0x55ed0d8cd5f0 2024-09-11T06:40:13.351 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:40:13.347+0000 7fb2ca654640 -1 reset not still connected to 0x55ed0e550dd0 2024-09-11T06:40:13.358 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:40:13.355+0000 7fb2ca654640 -1 reset not still connected to 0x55ed12570820 2024-09-11T06:40:13.358 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:40:13.355+0000 7fb2ca654640 -1 reset not still connected to 0x55ed149fe8f0 2024-09-11T06:40:13.358 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:40:13.355+0000 7fb2ca654640 -1 reset not still connected to 0x55ed161c2000 2024-09-11T06:40:13.359 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:40:13.355+0000 7fb2ca654640 -1 reset not still connected to 0x55ed194eaea0 2024-09-11T06:40:13.359 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:40:13.355+0000 7fb2ca654640 -1 reset not still connected to 0x55ed1bd1add0 2024-09-11T06:40:13.359 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:40:13.355+0000 7fb2ca654640 -1 reset not still connected to 0x55ed2a1e0ea0 2024-09-11T06:40:13.359 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:40:13.355+0000 7fb2ca654640 -1 reset not still connected to 0x55ed2b00a000 2024-09-11T06:40:13.359 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:40:13.355+0000 7fb2ca654640 -1 reset not still connected to 0x55ed38d48270 2024-09-11T06:40:13.359 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:40:13.355+0000 7fb2ca654640 -1 reset not still connected to 0x55ed3d399d40 2024-09-11T06:40:13.360 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:40:13.355+0000 7fb2ca654640 -1 reset not still connected to 0x55ed43824c30 2024-09-11T06:40:13.360 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:40:13.355+0000 7fb2ca654640 -1 reset not still connected to 0x55ed50ca3380 2024-09-11T06:40:13.700 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_started cluster1 mirror image_6 2024-09-11T06:40:13.700 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:40:13.700 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:40:13.701 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_6 2024-09-11T06:40:13.701 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_state cluster1 mirror image_6 started 2024-09-11T06:40:13.701 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:40:13.701 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:40:13.701 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_6 2024-09-11T06:40:13.701 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=started 2024-09-11T06:40:13.701 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:40:13.701 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:40:13.701 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:40:14.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_replay_state cluster1 mirror image_6 started 2024-09-11T06:40:14.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:40:14.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:40:14.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_6 2024-09-11T06:40:14.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=started 2024-09-11T06:40:14.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local status_result 2024-09-11T06:40:14.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=stopped 2024-09-11T06:40:14.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_6 2024-09-11T06:40:14.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_instance=cluster1 2024-09-11T06:40:14.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:40:14.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:40:14.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance=cluster1 2024-09-11T06:40:14.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local loop_instance 2024-09-11T06:40:14.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:40:14.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sleep 0 2024-09-11T06:40:14.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -i state 2024-09-11T06:40:14.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:40:14.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ seq 0 1 2024-09-11T06:40:14.708 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:40:14.708 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_6 2024-09-11T06:40:14.708 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1:0 2024-09-11T06:40:14.708 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:40:14.708 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:40:14.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:40:14.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:40:14.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=cluster 2024-09-11T06:40:14.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:40:14.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:40:14.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:40:14.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:40:14.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval cluster=cluster1 2024-09-11T06:40:14.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:40:14.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:40:14.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:40:14.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:40:14.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local local_cluster=cluster1:0 2024-09-11T06:40:14.711 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster=cluster1 2024-09-11T06:40:14.711 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance 2024-09-11T06:40:14.711 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:40:14.711 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local val=cluster1:0 2024-09-11T06:40:14.711 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T06:40:14.711 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance_var_name=instance 2024-09-11T06:40:14.712 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:40:14.712 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:40:14.712 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T06:40:14.712 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval local_cluster=cluster1 2024-09-11T06:40:14.712 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ local_cluster=cluster1 2024-09-11T06:40:14.712 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval instance=0 2024-09-11T06:40:14.712 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ instance=0 2024-09-11T06:40:14.712 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:40:14.734 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:40:14.735 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:40:14.735 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:40:14.735 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_6 2024-09-11T06:40:14.889 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:Invalid command: unused arguments: ['mirror/image_6'] 2024-09-11T06:40:14.889 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T06:40:14.889 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:admin_socket: invalid command 2024-09-11T06:40:14.893 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:40:14.893 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_6 2024-09-11T06:40:14.893 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1:1 2024-09-11T06:40:14.893 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:40:14.893 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:40:14.893 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-11T06:40:14.893 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:1 2024-09-11T06:40:14.893 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=cluster 2024-09-11T06:40:14.893 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:40:14.894 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:40:14.894 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=1 2024-09-11T06:40:14.894 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T06:40:14.894 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval cluster=cluster1 2024-09-11T06:40:14.894 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:40:14.894 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=1 2024-09-11T06:40:14.894 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=1 2024-09-11T06:40:14.894 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-11T06:40:14.894 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local local_cluster=cluster1:1 2024-09-11T06:40:14.894 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster=cluster1 2024-09-11T06:40:14.894 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance 2024-09-11T06:40:14.894 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T06:40:14.894 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local val=cluster1:1 2024-09-11T06:40:14.894 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T06:40:14.895 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance_var_name=instance 2024-09-11T06:40:14.895 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:40:14.895 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=1 2024-09-11T06:40:14.895 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-11T06:40:14.895 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval local_cluster=cluster1 2024-09-11T06:40:14.895 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ local_cluster=cluster1 2024-09-11T06:40:14.895 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval instance=1 2024-09-11T06:40:14.895 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ instance=1 2024-09-11T06:40:14.895 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T06:40:14.916 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:40:14.916 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:40:14.916 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:40:14.916 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_6 2024-09-11T06:40:15.051 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ return 0 2024-09-11T06:40:15.052 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T06:40:15.052 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ grep -i Replaying 2024-09-11T06:40:15.052 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ echo ' "state": "Replaying"' 2024-09-11T06:40:15.053 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: "state": "Replaying" 2024-09-11T06:40:15.054 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_state=started 2024-09-11T06:40:15.054 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test started = started 2024-09-11T06:40:15.054 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:40:15.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_6 2024-09-11T06:40:15.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:40:15.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:40:15.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:40:15.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_6 2024-09-11T06:40:15.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' journal = journal ']' 2024-09-11T06:40:15.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_6 2024-09-11T06:40:15.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:40:15.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:40:15.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:40:15.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_6 2024-09-11T06:40:15.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-11T06:40:15.056 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-11T06:40:15.056 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:40:15.056 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:40:15.056 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:40:15.255 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror image_6 2024-09-11T06:40:15.256 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:40:15.256 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:40:15.256 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_6 2024-09-11T06:40:15.256 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:40:15.256 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n image_6 ']' 2024-09-11T06:40:15.256 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/image_6' 2024-09-11T06:40:15.256 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_6 2024-09-11T06:40:15.256 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:40:15.256 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:40:15.256 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:40:15.256 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:40:15.256 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:40:15.257 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:40:15.257 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:40:15.257 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:40:15.257 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:40:15.261 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:40:15.261 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_6 2024-09-11T06:40:15.261 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:40:15.261 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:40:15.261 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:40:15.261 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:40:15.261 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:40:15.261 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:40:15.261 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:40:15.261 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:40:15.261 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:40:15.262 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:40:15.262 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:40:15.262 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:40:15.262 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:40:15.262 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:40:15.262 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:40:15.262 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:40:15.262 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:40:15.262 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:40:15.262 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:40:15.262 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:40:15.262 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:40:15.262 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:40:15.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:40:15.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:40:15.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:40:15.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:40:15.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:40:15.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:40:15.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:40:15.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:40:15.286 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:40:15.287 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:40:15.287 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:40:15.287 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_6 2024-09-11T06:40:15.413 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:Invalid command: unused arguments: ['mirror/image_6'] 2024-09-11T06:40:15.413 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd mirror flush : flush rbd mirror 2024-09-11T06:40:15.413 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:admin_socket: invalid command 2024-09-11T06:40:15.416 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:40:15.416 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_6 2024-09-11T06:40:15.416 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:1 2024-09-11T06:40:15.416 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:40:15.416 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:40:15.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-11T06:40:15.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:1 2024-09-11T06:40:15.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:40:15.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:40:15.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:40:15.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=1 2024-09-11T06:40:15.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-11T06:40:15.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:40:15.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:40:15.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=1 2024-09-11T06:40:15.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=1 2024-09-11T06:40:15.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-11T06:40:15.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:1 2024-09-11T06:40:15.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:40:15.418 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:40:15.418 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T06:40:15.418 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:1 2024-09-11T06:40:15.418 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:40:15.418 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:40:15.418 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:40:15.418 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=1 2024-09-11T06:40:15.418 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T06:40:15.418 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:40:15.418 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:40:15.418 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=1 2024-09-11T06:40:15.418 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=1 2024-09-11T06:40:15.418 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T06:40:15.435 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:40:15.435 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:40:15.435 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:40:15.435 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_6 2024-09-11T06:40:15.586 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:40:15.588 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror image_6 2024-09-11T06:40:15.588 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:40:15.588 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:40:15.588 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_6 2024-09-11T06:40:15.588 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror image_6 '' 2024-09-11T06:40:15.588 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:40:15.589 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:40:15.589 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_6 2024-09-11T06:40:15.589 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:40:15.589 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-image_6.status 2024-09-11T06:40:15.589 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-image_6.status 2024-09-11T06:40:15.589 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:40:15.591 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_6.status 2024-09-11T06:40:15.591 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_6 2024-09-11T06:40:15.591 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_6.status 2024-09-11T06:40:15.637 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 0 2024-09-11T06:40:15.637 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 1 2024-09-11T06:40:15.637 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:40:15.637 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=3371], [object_number=2, tag_tid=1, entry_tid=3370]]], state=connected] 2024-09-11T06:40:15.637 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=3371], [object_number=2, tag_tid=1, entry_tid=3370]]], state=connected] 2024-09-11T06:40:15.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_6.status 2024-09-11T06:40:15.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-09-11T06:40:15.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror image_6 2024-09-11T06:40:15.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:40:15.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:40:15.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_6 2024-09-11T06:40:15.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror image_6 '..*' 2024-09-11T06:40:15.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:40:15.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:40:15.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_6 2024-09-11T06:40:15.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:40:15.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-image_6.status 2024-09-11T06:40:15.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-image_6.status 2024-09-11T06:40:15.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:40:15.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_6.status 2024-09-11T06:40:15.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_6 2024-09-11T06:40:15.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_6.status 2024-09-11T06:40:15.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 0 2024-09-11T06:40:15.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 1 2024-09-11T06:40:15.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:40:15.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=3371], [object_number=2, tag_tid=1, entry_tid=3370]]], state=connected] 2024-09-11T06:40:15.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=3371], [object_number=2, tag_tid=1, entry_tid=3370]]], state=connected] 2024-09-11T06:40:15.695 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_6.status 2024-09-11T06:40:15.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-09-11T06:40:15.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=3, entry_tid=1' 2024-09-11T06:40:15.700 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:40:15.700 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_snap_present cluster1 mirror image_6 snap 2024-09-11T06:40:15.700 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:40:15.700 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:40:15.700 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_6 2024-09-11T06:40:15.700 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:40:15.701 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:40:15.701 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T06:40:15.701 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:40:16.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 info mirror/image_6@snap 2024-09-11T06:40:16.756 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:rbd image 'image_6': 2024-09-11T06:40:16.756 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: size 128 MiB in 32 objects 2024-09-11T06:40:16.756 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: order 22 (4 MiB objects) 2024-09-11T06:40:16.756 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: snapshot_count: 1 2024-09-11T06:40:16.756 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: id: 10a9ff03d408 2024-09-11T06:40:16.756 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: block_name_prefix: rbd_data.10a9ff03d408 2024-09-11T06:40:16.757 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: format: 2 2024-09-11T06:40:16.757 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: features: layering, exclusive-lock, journaling 2024-09-11T06:40:16.757 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: op_features: 2024-09-11T06:40:16.757 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: flags: 2024-09-11T06:40:16.757 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: create_timestamp: Wed Sep 11 06:31:47 2024 2024-09-11T06:40:16.757 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: access_timestamp: Wed Sep 11 06:31:47 2024 2024-09-11T06:40:16.757 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: modify_timestamp: Wed Sep 11 06:31:47 2024 2024-09-11T06:40:16.757 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: protected: False 2024-09-11T06:40:16.757 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: journal: 10a9ff03d408 2024-09-11T06:40:16.757 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring state: enabled 2024-09-11T06:40:16.757 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring mode: journal 2024-09-11T06:40:16.757 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring global id: 27e7c44e-8cb3-4e7f-b9cf-bf5961445156 2024-09-11T06:40:16.757 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring primary: false 2024-09-11T06:40:16.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:40:16.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ compare_image_snaps mirror image_6 snap 2024-09-11T06:40:16.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:40:16.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_6 2024-09-11T06:40:16.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:40:16.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local ret=0 2024-09-11T06:40:16.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local rmt_export=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_6.export 2024-09-11T06:40:16.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loc_export=/tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_6.export 2024-09-11T06:40:16.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_6.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_6.export 2024-09-11T06:40:16.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror export image_6@snap /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_6.export 2024-09-11T06:40:16.924 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T06:40:16.932 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 -p mirror export image_6@snap /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_6.export 2024-09-11T06:40:17.183 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T06:40:17.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmp /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_6.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_6.export 2024-09-11T06:40:17.274 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_6.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_6.export 2024-09-11T06:40:17.310 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:40:17.310 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:40:17.310 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_7 2024-09-11T06:40:17.310 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_snap cluster2 mirror image_7 snap 2024-09-11T06:40:17.310 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:40:17.310 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:40:17.310 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_7 2024-09-11T06:40:17.310 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:40:17.311 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror snap create image_7@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T06:40:17.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_started cluster1 mirror image_7 2024-09-11T06:40:17.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:40:17.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:40:17.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_7 2024-09-11T06:40:17.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_state cluster1 mirror image_7 started 2024-09-11T06:40:17.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:40:17.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:40:17.706 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_7 2024-09-11T06:40:17.706 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=started 2024-09-11T06:40:17.706 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:40:17.706 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:40:17.706 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:40:18.706 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_replay_state cluster1 mirror image_7 started 2024-09-11T06:40:18.707 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:40:18.707 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:40:18.707 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_7 2024-09-11T06:40:18.707 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=started 2024-09-11T06:40:18.707 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local status_result 2024-09-11T06:40:18.707 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=stopped 2024-09-11T06:40:18.707 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_7 2024-09-11T06:40:18.708 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -i state 2024-09-11T06:40:18.708 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_instance=cluster1 2024-09-11T06:40:18.708 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:40:18.708 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:40:18.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance=cluster1 2024-09-11T06:40:18.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local loop_instance 2024-09-11T06:40:18.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:40:18.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sleep 0 2024-09-11T06:40:18.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:40:18.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ seq 0 1 2024-09-11T06:40:18.712 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:40:18.712 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_7 2024-09-11T06:40:18.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1:0 2024-09-11T06:40:18.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:40:18.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:40:18.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:40:18.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:40:18.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=cluster 2024-09-11T06:40:18.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:40:18.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:40:18.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:40:18.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:40:18.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval cluster=cluster1 2024-09-11T06:40:18.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:40:18.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:40:18.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:40:18.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:40:18.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local local_cluster=cluster1:0 2024-09-11T06:40:18.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster=cluster1 2024-09-11T06:40:18.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance 2024-09-11T06:40:18.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:40:18.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local val=cluster1:0 2024-09-11T06:40:18.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T06:40:18.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance_var_name=instance 2024-09-11T06:40:18.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:40:18.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:40:18.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T06:40:18.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval local_cluster=cluster1 2024-09-11T06:40:18.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ local_cluster=cluster1 2024-09-11T06:40:18.716 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval instance=0 2024-09-11T06:40:18.716 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ instance=0 2024-09-11T06:40:18.716 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:40:18.735 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:40:18.735 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:40:18.735 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:40:18.736 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_7 2024-09-11T06:40:18.879 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:40:18.875+0000 7fc41084f640 -1 reset not still connected to 0x55eb38da3a00 2024-09-11T06:40:18.879 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:40:18.875+0000 7fc41084f640 -1 reset not still connected to 0x55eb43fe4340 2024-09-11T06:40:18.880 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:40:18.875+0000 7fc41084f640 -1 reset not still connected to 0x55eb4464a410 2024-09-11T06:40:18.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ return 0 2024-09-11T06:40:18.887 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T06:40:18.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ echo ' "state": "Replaying"' 2024-09-11T06:40:18.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ grep -i Replaying 2024-09-11T06:40:18.889 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: "state": "Replaying" 2024-09-11T06:40:18.890 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_state=started 2024-09-11T06:40:18.891 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test started = started 2024-09-11T06:40:18.891 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:40:18.891 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_7 2024-09-11T06:40:18.891 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:40:18.891 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:40:18.891 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:40:18.891 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_7 2024-09-11T06:40:18.891 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' journal = journal ']' 2024-09-11T06:40:18.891 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_7 2024-09-11T06:40:18.891 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:40:18.891 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:40:18.891 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:40:18.892 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_7 2024-09-11T06:40:18.892 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-11T06:40:18.892 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-11T06:40:18.892 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:40:18.892 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:40:18.892 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:40:19.093 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror image_7 2024-09-11T06:40:19.093 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:40:19.093 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:40:19.093 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_7 2024-09-11T06:40:19.093 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:40:19.093 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n image_7 ']' 2024-09-11T06:40:19.093 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/image_7' 2024-09-11T06:40:19.093 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_7 2024-09-11T06:40:19.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:40:19.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:40:19.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:40:19.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:40:19.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:40:19.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:40:19.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:40:19.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:40:19.095 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:40:19.098 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:40:19.098 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_7 2024-09-11T06:40:19.098 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:40:19.098 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:40:19.098 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:40:19.098 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:40:19.098 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:40:19.098 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:40:19.098 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:40:19.098 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:40:19.098 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:40:19.098 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:40:19.099 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:40:19.099 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:40:19.099 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:40:19.099 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:40:19.099 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:40:19.099 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:40:19.099 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:40:19.099 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:40:19.099 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:40:19.099 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:40:19.099 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:40:19.099 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:40:19.100 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:40:19.100 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:40:19.100 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:40:19.100 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:40:19.100 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:40:19.100 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:40:19.100 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:40:19.100 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:40:19.124 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:40:19.124 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:40:19.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:40:19.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_7 2024-09-11T06:40:19.266 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:40:19.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror image_7 2024-09-11T06:40:19.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:40:19.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:40:19.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_7 2024-09-11T06:40:19.269 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror image_7 '' 2024-09-11T06:40:19.269 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:40:19.269 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:40:19.269 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_7 2024-09-11T06:40:19.269 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:40:19.269 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-image_7.status 2024-09-11T06:40:19.270 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-image_7.status 2024-09-11T06:40:19.270 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:40:19.270 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_7.status 2024-09-11T06:40:19.271 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_7 2024-09-11T06:40:19.271 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_7.status 2024-09-11T06:40:19.327 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 1 2024-09-11T06:40:19.328 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 2 2024-09-11T06:40:19.328 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:40:19.328 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=3, entry_tid=1], [object_number=8, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6375], [object_number=6, tag_tid=1, entry_tid=6374]]], state=connected] 2024-09-11T06:40:19.328 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=9, tag_tid=3, entry_tid=1], [object_number=8, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6375], [object_number=6, tag_tid=1, entry_tid=6374]]], state=connected] 2024-09-11T06:40:19.331 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_7.status 2024-09-11T06:40:19.332 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=9, tag_tid=3, entry_tid=1' 2024-09-11T06:40:19.333 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror image_7 2024-09-11T06:40:19.334 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:40:19.334 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:40:19.334 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_7 2024-09-11T06:40:19.334 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror image_7 '..*' 2024-09-11T06:40:19.334 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:40:19.334 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:40:19.334 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_7 2024-09-11T06:40:19.334 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:40:19.334 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-image_7.status 2024-09-11T06:40:19.334 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-image_7.status 2024-09-11T06:40:19.334 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:40:19.335 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_7.status 2024-09-11T06:40:19.335 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_7 2024-09-11T06:40:19.335 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_7.status 2024-09-11T06:40:19.381 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 1 2024-09-11T06:40:19.381 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 2 2024-09-11T06:40:19.381 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:40:19.381 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=3, entry_tid=1], [object_number=8, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6375], [object_number=6, tag_tid=1, entry_tid=6374]]], state=connected] 2024-09-11T06:40:19.381 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=9, tag_tid=3, entry_tid=1], [object_number=8, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6375], [object_number=6, tag_tid=1, entry_tid=6374]]], state=connected] 2024-09-11T06:40:19.384 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_7.status 2024-09-11T06:40:19.386 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=9, tag_tid=3, entry_tid=1' 2024-09-11T06:40:19.387 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=9, tag_tid=3, entry_tid=1' -a 'object_number=9, tag_tid=3, entry_tid=1' = 'object_number=9, tag_tid=3, entry_tid=1' 2024-09-11T06:40:19.387 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:40:19.387 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_snap_present cluster1 mirror image_7 snap 2024-09-11T06:40:19.387 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:40:19.387 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:40:19.387 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_7 2024-09-11T06:40:19.387 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:40:19.387 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:40:19.387 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T06:40:19.387 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:40:20.387 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 info mirror/image_7@snap 2024-09-11T06:40:20.442 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:rbd image 'image_7': 2024-09-11T06:40:20.442 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: size 128 MiB in 32 objects 2024-09-11T06:40:20.442 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: order 22 (4 MiB objects) 2024-09-11T06:40:20.442 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: snapshot_count: 1 2024-09-11T06:40:20.442 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: id: 10a6d8e1ca02 2024-09-11T06:40:20.443 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: block_name_prefix: rbd_data.10a6d8e1ca02 2024-09-11T06:40:20.443 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: format: 2 2024-09-11T06:40:20.443 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: features: layering, exclusive-lock, journaling 2024-09-11T06:40:20.443 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: op_features: 2024-09-11T06:40:20.443 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: flags: 2024-09-11T06:40:20.443 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: create_timestamp: Wed Sep 11 06:31:53 2024 2024-09-11T06:40:20.443 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: access_timestamp: Wed Sep 11 06:31:53 2024 2024-09-11T06:40:20.443 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: modify_timestamp: Wed Sep 11 06:33:12 2024 2024-09-11T06:40:20.443 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: protected: False 2024-09-11T06:40:20.443 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: journal: 10a6d8e1ca02 2024-09-11T06:40:20.443 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring state: enabled 2024-09-11T06:40:20.444 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring mode: journal 2024-09-11T06:40:20.444 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring global id: 5ae2db0c-ea28-4d92-aaa7-30d23807fd8f 2024-09-11T06:40:20.444 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring primary: false 2024-09-11T06:40:20.447 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:40:20.447 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ compare_image_snaps mirror image_7 snap 2024-09-11T06:40:20.447 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:40:20.448 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_7 2024-09-11T06:40:20.448 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:40:20.448 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local ret=0 2024-09-11T06:40:20.448 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local rmt_export=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_7.export 2024-09-11T06:40:20.448 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loc_export=/tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_7.export 2024-09-11T06:40:20.448 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_7.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_7.export 2024-09-11T06:40:20.448 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror export image_7@snap /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_7.export 2024-09-11T06:40:20.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T06:40:20.650 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 -p mirror export image_7@snap /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_7.export 2024-09-11T06:40:20.921 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T06:40:20.933 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmp /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_7.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_7.export 2024-09-11T06:40:21.032 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_7.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_7.export 2024-09-11T06:40:21.069 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:40:21.069 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:40:21.069 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_8 2024-09-11T06:40:21.069 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_snap cluster2 mirror image_8 snap 2024-09-11T06:40:21.069 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:40:21.069 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:40:21.069 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_8 2024-09-11T06:40:21.069 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:40:21.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror snap create image_8@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T06:40:22.045 INFO:tasks.ceph.cluster2.osd.1.smithi112.stderr:2024-09-11T06:40:22.045+0000 7f2cecaac640 -1 reset not still connected to 0x55da8e8c29c0 2024-09-11T06:40:22.045 INFO:tasks.ceph.cluster2.osd.1.smithi112.stderr:2024-09-11T06:40:22.045+0000 7f2cecaac640 -1 reset not still connected to 0x55da8ee0bd40 2024-09-11T06:40:22.045 INFO:tasks.ceph.cluster2.osd.1.smithi112.stderr:2024-09-11T06:40:22.045+0000 7f2cecaac640 -1 reset not still connected to 0x55da93619110 2024-09-11T06:40:22.046 INFO:tasks.ceph.cluster2.osd.1.smithi112.stderr:2024-09-11T06:40:22.045+0000 7f2cecaac640 -1 reset not still connected to 0x55da941c60d0 2024-09-11T06:40:22.046 INFO:tasks.ceph.cluster2.osd.1.smithi112.stderr:2024-09-11T06:40:22.045+0000 7f2cecaac640 -1 reset not still connected to 0x55da97faed00 2024-09-11T06:40:22.046 INFO:tasks.ceph.cluster2.osd.1.smithi112.stderr:2024-09-11T06:40:22.045+0000 7f2cecaac640 -1 reset not still connected to 0x55daa159cf70 2024-09-11T06:40:22.046 INFO:tasks.ceph.cluster2.osd.1.smithi112.stderr:2024-09-11T06:40:22.045+0000 7f2cecaac640 -1 reset not still connected to 0x55daaabf31e0 2024-09-11T06:40:22.046 INFO:tasks.ceph.cluster2.osd.1.smithi112.stderr:2024-09-11T06:40:22.045+0000 7f2cecaac640 -1 reset not still connected to 0x55daaf92cdd0 2024-09-11T06:40:22.046 INFO:tasks.ceph.cluster2.osd.1.smithi112.stderr:2024-09-11T06:40:22.045+0000 7f2cecaac640 -1 reset not still connected to 0x55dab8b8c270 2024-09-11T06:40:23.748 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_started cluster1 mirror image_8 2024-09-11T06:40:23.748 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:40:23.748 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:40:23.748 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_8 2024-09-11T06:40:23.748 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_state cluster1 mirror image_8 started 2024-09-11T06:40:23.749 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:40:23.749 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:40:23.749 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_8 2024-09-11T06:40:23.749 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=started 2024-09-11T06:40:23.749 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:40:23.749 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:40:23.749 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:40:24.750 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_replay_state cluster1 mirror image_8 started 2024-09-11T06:40:24.750 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:40:24.751 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:40:24.751 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_8 2024-09-11T06:40:24.751 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=started 2024-09-11T06:40:24.751 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local status_result 2024-09-11T06:40:24.751 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=stopped 2024-09-11T06:40:24.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_8 2024-09-11T06:40:24.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_instance=cluster1 2024-09-11T06:40:24.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:40:24.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:40:24.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance=cluster1 2024-09-11T06:40:24.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local loop_instance 2024-09-11T06:40:24.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:40:24.753 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sleep 0 2024-09-11T06:40:24.753 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -i state 2024-09-11T06:40:24.753 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:40:24.754 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ seq 0 1 2024-09-11T06:40:24.756 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:40:24.756 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_8 2024-09-11T06:40:24.756 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1:0 2024-09-11T06:40:24.756 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:40:24.756 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:40:24.756 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:40:24.756 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:40:24.756 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=cluster 2024-09-11T06:40:24.756 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:40:24.756 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:40:24.757 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:40:24.757 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:40:24.757 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval cluster=cluster1 2024-09-11T06:40:24.757 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:40:24.757 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:40:24.757 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:40:24.757 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:40:24.757 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local local_cluster=cluster1:0 2024-09-11T06:40:24.757 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster=cluster1 2024-09-11T06:40:24.758 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance 2024-09-11T06:40:24.758 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:40:24.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local val=cluster1:0 2024-09-11T06:40:24.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T06:40:24.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance_var_name=instance 2024-09-11T06:40:24.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:40:24.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:40:24.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T06:40:24.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval local_cluster=cluster1 2024-09-11T06:40:24.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ local_cluster=cluster1 2024-09-11T06:40:24.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval instance=0 2024-09-11T06:40:24.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ instance=0 2024-09-11T06:40:24.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:40:24.781 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:40:24.782 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:40:24.782 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:40:24.782 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_8 2024-09-11T06:40:24.940 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:Invalid command: unused arguments: ['mirror/image_8'] 2024-09-11T06:40:24.940 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T06:40:24.941 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:admin_socket: invalid command 2024-09-11T06:40:24.944 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:40:24.944 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_8 2024-09-11T06:40:24.944 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1:1 2024-09-11T06:40:24.944 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:40:24.944 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:40:24.944 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-11T06:40:24.944 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:1 2024-09-11T06:40:24.944 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=cluster 2024-09-11T06:40:24.944 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:40:24.944 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:40:24.944 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=1 2024-09-11T06:40:24.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T06:40:24.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval cluster=cluster1 2024-09-11T06:40:24.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:40:24.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=1 2024-09-11T06:40:24.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=1 2024-09-11T06:40:24.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-11T06:40:24.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local local_cluster=cluster1:1 2024-09-11T06:40:24.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster=cluster1 2024-09-11T06:40:24.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance 2024-09-11T06:40:24.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T06:40:24.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local val=cluster1:1 2024-09-11T06:40:24.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T06:40:24.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance_var_name=instance 2024-09-11T06:40:24.946 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:40:24.946 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=1 2024-09-11T06:40:24.946 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-11T06:40:24.946 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval local_cluster=cluster1 2024-09-11T06:40:24.946 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ local_cluster=cluster1 2024-09-11T06:40:24.946 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval instance=1 2024-09-11T06:40:24.946 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ instance=1 2024-09-11T06:40:24.946 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T06:40:24.962 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:40:24.962 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:40:24.962 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:40:24.962 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_8 2024-09-11T06:40:25.097 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ return 0 2024-09-11T06:40:25.098 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T06:40:25.098 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ echo ' "state": "Replaying"' 2024-09-11T06:40:25.098 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ grep -i Replaying 2024-09-11T06:40:25.099 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: "state": "Replaying" 2024-09-11T06:40:25.101 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_state=started 2024-09-11T06:40:25.101 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test started = started 2024-09-11T06:40:25.101 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:40:25.101 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_8 2024-09-11T06:40:25.101 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:40:25.101 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:40:25.101 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:40:25.101 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_8 2024-09-11T06:40:25.101 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' journal = journal ']' 2024-09-11T06:40:25.101 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_8 2024-09-11T06:40:25.101 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:40:25.101 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:40:25.102 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:40:25.102 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_8 2024-09-11T06:40:25.102 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-11T06:40:25.102 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-11T06:40:25.102 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:40:25.102 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:40:25.102 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:40:25.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror image_8 2024-09-11T06:40:25.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:40:25.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:40:25.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_8 2024-09-11T06:40:25.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:40:25.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n image_8 ']' 2024-09-11T06:40:25.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/image_8' 2024-09-11T06:40:25.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_8 2024-09-11T06:40:25.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:40:25.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:40:25.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:40:25.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:40:25.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:40:25.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:40:25.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:40:25.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:40:25.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:40:25.306 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:40:25.306 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_8 2024-09-11T06:40:25.306 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:40:25.306 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:40:25.306 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:40:25.306 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:40:25.306 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:40:25.306 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:40:25.306 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:40:25.306 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:40:25.307 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:40:25.307 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:40:25.307 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:40:25.307 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:40:25.307 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:40:25.307 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:40:25.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:40:25.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:40:25.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:40:25.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:40:25.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:40:25.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:40:25.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:40:25.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:40:25.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:40:25.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:40:25.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:40:25.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:40:25.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:40:25.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:40:25.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:40:25.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:40:25.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:40:25.337 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:40:25.337 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:40:25.337 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_8 2024-09-11T06:40:25.458 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:Invalid command: unused arguments: ['mirror/image_8'] 2024-09-11T06:40:25.458 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd mirror flush : flush rbd mirror 2024-09-11T06:40:25.458 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:admin_socket: invalid command 2024-09-11T06:40:25.460 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:40:25.460 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_8 2024-09-11T06:40:25.460 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:1 2024-09-11T06:40:25.460 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:40:25.460 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:40:25.461 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-11T06:40:25.461 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:1 2024-09-11T06:40:25.461 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:40:25.461 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:40:25.461 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:40:25.461 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=1 2024-09-11T06:40:25.461 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-11T06:40:25.461 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:40:25.461 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:40:25.461 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=1 2024-09-11T06:40:25.461 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=1 2024-09-11T06:40:25.463 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-11T06:40:25.463 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:1 2024-09-11T06:40:25.463 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:40:25.463 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:40:25.463 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T06:40:25.463 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:1 2024-09-11T06:40:25.463 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:40:25.463 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:40:25.463 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:40:25.463 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=1 2024-09-11T06:40:25.463 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T06:40:25.463 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:40:25.463 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:40:25.464 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=1 2024-09-11T06:40:25.464 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=1 2024-09-11T06:40:25.464 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T06:40:25.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:40:25.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:40:25.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:40:25.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_8 2024-09-11T06:40:25.629 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:40:25.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror image_8 2024-09-11T06:40:25.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:40:25.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:40:25.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_8 2024-09-11T06:40:25.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror image_8 '' 2024-09-11T06:40:25.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:40:25.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:40:25.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_8 2024-09-11T06:40:25.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:40:25.631 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-image_8.status 2024-09-11T06:40:25.631 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-image_8.status 2024-09-11T06:40:25.631 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:40:25.633 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_8.status 2024-09-11T06:40:25.633 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_8 2024-09-11T06:40:25.633 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_8.status 2024-09-11T06:40:25.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 3 2024-09-11T06:40:25.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 3 2024-09-11T06:40:25.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:40:25.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=14, tag_tid=1, entry_tid=12818], [object_number=15, tag_tid=1, entry_tid=12815]]], state=connected] 2024-09-11T06:40:25.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=14, tag_tid=1, entry_tid=12818], [object_number=15, tag_tid=1, entry_tid=12815]]], state=connected] 2024-09-11T06:40:25.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_8.status 2024-09-11T06:40:25.693 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-09-11T06:40:25.694 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror image_8 2024-09-11T06:40:25.694 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:40:25.694 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:40:25.694 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_8 2024-09-11T06:40:25.694 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror image_8 '..*' 2024-09-11T06:40:25.694 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:40:25.695 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:40:25.695 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_8 2024-09-11T06:40:25.695 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:40:25.695 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-image_8.status 2024-09-11T06:40:25.695 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-image_8.status 2024-09-11T06:40:25.695 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:40:25.695 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_8.status 2024-09-11T06:40:25.696 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_8 2024-09-11T06:40:25.696 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_8.status 2024-09-11T06:40:25.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 3 2024-09-11T06:40:25.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 3 2024-09-11T06:40:25.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:40:25.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=14, tag_tid=1, entry_tid=12818], [object_number=15, tag_tid=1, entry_tid=12815]]], state=connected] 2024-09-11T06:40:25.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=13, tag_tid=2, entry_tid=1], [object_number=12, tag_tid=2, entry_tid=0], [object_number=14, tag_tid=1, entry_tid=12818], [object_number=15, tag_tid=1, entry_tid=12815]]], state=connected] 2024-09-11T06:40:25.746 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_8.status 2024-09-11T06:40:25.748 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=13, tag_tid=2, entry_tid=1' 2024-09-11T06:40:25.748 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=13, tag_tid=2, entry_tid=1' -a 'object_number=13, tag_tid=2, entry_tid=1' = 'object_number=13, tag_tid=2, entry_tid=1' 2024-09-11T06:40:25.748 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:40:25.748 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_snap_present cluster1 mirror image_8 snap 2024-09-11T06:40:25.748 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:40:25.748 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:40:25.749 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_8 2024-09-11T06:40:25.749 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:40:25.749 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:40:25.749 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T06:40:25.749 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:40:26.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 info mirror/image_8@snap 2024-09-11T06:40:26.818 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:rbd image 'image_8': 2024-09-11T06:40:26.818 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: size 128 MiB in 32 objects 2024-09-11T06:40:26.818 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: order 22 (4 MiB objects) 2024-09-11T06:40:26.819 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: snapshot_count: 1 2024-09-11T06:40:26.819 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: id: 10a9a2384935 2024-09-11T06:40:26.819 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: block_name_prefix: rbd_data.10a9a2384935 2024-09-11T06:40:26.819 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: format: 2 2024-09-11T06:40:26.819 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: features: layering, exclusive-lock, journaling 2024-09-11T06:40:26.819 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: op_features: 2024-09-11T06:40:26.819 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: flags: 2024-09-11T06:40:26.819 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: create_timestamp: Wed Sep 11 06:31:58 2024 2024-09-11T06:40:26.819 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: access_timestamp: Wed Sep 11 06:31:58 2024 2024-09-11T06:40:26.819 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: modify_timestamp: Wed Sep 11 06:32:58 2024 2024-09-11T06:40:26.819 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: protected: False 2024-09-11T06:40:26.819 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: journal: 10a9a2384935 2024-09-11T06:40:26.819 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring state: enabled 2024-09-11T06:40:26.819 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring mode: journal 2024-09-11T06:40:26.820 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring global id: 21e812ba-de23-490a-9ea6-7dc719838f35 2024-09-11T06:40:26.820 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring primary: false 2024-09-11T06:40:26.823 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:40:26.823 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ compare_image_snaps mirror image_8 snap 2024-09-11T06:40:26.824 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:40:26.824 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_8 2024-09-11T06:40:26.824 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:40:26.824 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local ret=0 2024-09-11T06:40:26.824 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local rmt_export=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_8.export 2024-09-11T06:40:26.824 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loc_export=/tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_8.export 2024-09-11T06:40:26.824 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_8.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_8.export 2024-09-11T06:40:26.824 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror export image_8@snap /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_8.export 2024-09-11T06:40:26.968 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T06:40:26.975 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 -p mirror export image_8@snap /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_8.export 2024-09-11T06:40:27.272 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T06:40:27.289 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmp /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_8.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_8.export 2024-09-11T06:40:27.418 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_8.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_8.export 2024-09-11T06:40:27.441 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:40:27.441 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:40:27.441 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_9 2024-09-11T06:40:27.441 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_snap cluster2 mirror image_9 snap 2024-09-11T06:40:27.441 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:40:27.441 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:40:27.441 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_9 2024-09-11T06:40:27.441 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:40:27.441 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror snap create image_9@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T06:40:27.775 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_started cluster1 mirror image_9 2024-09-11T06:40:27.775 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:40:27.775 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:40:27.775 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_9 2024-09-11T06:40:27.775 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_state cluster1 mirror image_9 started 2024-09-11T06:40:27.775 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:40:27.775 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:40:27.775 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_9 2024-09-11T06:40:27.776 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=started 2024-09-11T06:40:27.776 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:40:27.776 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:40:27.776 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:40:28.776 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_replay_state cluster1 mirror image_9 started 2024-09-11T06:40:28.776 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:40:28.776 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:40:28.777 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_9 2024-09-11T06:40:28.777 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=started 2024-09-11T06:40:28.777 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local status_result 2024-09-11T06:40:28.777 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=stopped 2024-09-11T06:40:28.778 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_9 2024-09-11T06:40:28.778 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_instance=cluster1 2024-09-11T06:40:28.778 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -i state 2024-09-11T06:40:28.778 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:40:28.778 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:40:28.778 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance=cluster1 2024-09-11T06:40:28.778 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local loop_instance 2024-09-11T06:40:28.778 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:40:28.778 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sleep 0 2024-09-11T06:40:28.779 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:40:28.780 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ seq 0 1 2024-09-11T06:40:28.783 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:40:28.783 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_9 2024-09-11T06:40:28.783 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1:0 2024-09-11T06:40:28.783 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:40:28.783 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:40:28.783 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:40:28.783 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:40:28.783 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=cluster 2024-09-11T06:40:28.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:40:28.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:40:28.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:40:28.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:40:28.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval cluster=cluster1 2024-09-11T06:40:28.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:40:28.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:40:28.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:40:28.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:40:28.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local local_cluster=cluster1:0 2024-09-11T06:40:28.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster=cluster1 2024-09-11T06:40:28.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance 2024-09-11T06:40:28.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:40:28.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local val=cluster1:0 2024-09-11T06:40:28.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T06:40:28.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance_var_name=instance 2024-09-11T06:40:28.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:40:28.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:40:28.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T06:40:28.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval local_cluster=cluster1 2024-09-11T06:40:28.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ local_cluster=cluster1 2024-09-11T06:40:28.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval instance=0 2024-09-11T06:40:28.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ instance=0 2024-09-11T06:40:28.787 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:40:28.807 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:40:28.807 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:40:28.807 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:40:28.807 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_9 2024-09-11T06:40:28.941 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ return 0 2024-09-11T06:40:28.941 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T06:40:28.942 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ echo ' "state": "Replaying"' 2024-09-11T06:40:28.942 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ grep -i Replaying 2024-09-11T06:40:28.943 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: "state": "Replaying" 2024-09-11T06:40:28.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_state=started 2024-09-11T06:40:28.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test started = started 2024-09-11T06:40:28.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:40:28.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_9 2024-09-11T06:40:28.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:40:28.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:40:28.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:40:28.946 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_9 2024-09-11T06:40:28.946 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' journal = journal ']' 2024-09-11T06:40:28.946 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_9 2024-09-11T06:40:28.946 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:40:28.946 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:40:28.946 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:40:28.946 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_9 2024-09-11T06:40:28.946 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-11T06:40:28.946 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-11T06:40:28.946 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:40:28.946 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:40:28.946 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:40:29.146 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror image_9 2024-09-11T06:40:29.147 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:40:29.147 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:40:29.147 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_9 2024-09-11T06:40:29.147 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:40:29.147 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n image_9 ']' 2024-09-11T06:40:29.147 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/image_9' 2024-09-11T06:40:29.147 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_9 2024-09-11T06:40:29.147 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:40:29.147 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:40:29.147 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:40:29.147 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:40:29.147 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:40:29.148 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:40:29.148 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:40:29.148 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:40:29.148 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:40:29.152 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:40:29.152 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_9 2024-09-11T06:40:29.152 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:40:29.152 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:40:29.152 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:40:29.152 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:40:29.152 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:40:29.152 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:40:29.152 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:40:29.152 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:40:29.153 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:40:29.153 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:40:29.153 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:40:29.153 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:40:29.153 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:40:29.153 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:40:29.153 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:40:29.153 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:40:29.153 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:40:29.153 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:40:29.153 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:40:29.153 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:40:29.153 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:40:29.154 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:40:29.154 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:40:29.154 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:40:29.154 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:40:29.154 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:40:29.154 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:40:29.154 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:40:29.154 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:40:29.154 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:40:29.176 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:40:29.177 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:40:29.177 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:40:29.177 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_9 2024-09-11T06:40:29.321 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:40:29.322 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror image_9 2024-09-11T06:40:29.322 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:40:29.322 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:40:29.322 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_9 2024-09-11T06:40:29.322 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror image_9 '' 2024-09-11T06:40:29.322 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:40:29.323 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:40:29.323 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_9 2024-09-11T06:40:29.323 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:40:29.323 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-image_9.status 2024-09-11T06:40:29.323 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-image_9.status 2024-09-11T06:40:29.323 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:40:29.324 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_9.status 2024-09-11T06:40:29.325 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_9 2024-09-11T06:40:29.325 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_9.status 2024-09-11T06:40:29.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 2 2024-09-11T06:40:29.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 2 2024-09-11T06:40:29.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:40:29.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=3, entry_tid=1], [object_number=8, tag_tid=3, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=9234], [object_number=11, tag_tid=1, entry_tid=9231]]], state=connected] 2024-09-11T06:40:29.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=9, tag_tid=3, entry_tid=1], [object_number=8, tag_tid=3, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=9234], [object_number=11, tag_tid=1, entry_tid=9231]]], state=connected] 2024-09-11T06:40:29.373 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_9.status 2024-09-11T06:40:29.374 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=9, tag_tid=3, entry_tid=1' 2024-09-11T06:40:29.376 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror image_9 2024-09-11T06:40:29.376 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:40:29.376 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:40:29.376 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_9 2024-09-11T06:40:29.376 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror image_9 '..*' 2024-09-11T06:40:29.376 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:40:29.376 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:40:29.376 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_9 2024-09-11T06:40:29.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:40:29.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-image_9.status 2024-09-11T06:40:29.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-image_9.status 2024-09-11T06:40:29.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:40:29.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_9.status 2024-09-11T06:40:29.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_9 2024-09-11T06:40:29.378 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_9.status 2024-09-11T06:40:29.421 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 2 2024-09-11T06:40:29.422 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 2 2024-09-11T06:40:29.422 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:40:29.422 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=3, entry_tid=1], [object_number=8, tag_tid=3, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=9234], [object_number=11, tag_tid=1, entry_tid=9231]]], state=connected] 2024-09-11T06:40:29.422 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=9, tag_tid=3, entry_tid=1], [object_number=8, tag_tid=3, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=9234], [object_number=11, tag_tid=1, entry_tid=9231]]], state=connected] 2024-09-11T06:40:29.425 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_9.status 2024-09-11T06:40:29.427 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=9, tag_tid=3, entry_tid=1' 2024-09-11T06:40:29.427 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=9, tag_tid=3, entry_tid=1' -a 'object_number=9, tag_tid=3, entry_tid=1' = 'object_number=9, tag_tid=3, entry_tid=1' 2024-09-11T06:40:29.427 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:40:29.427 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_snap_present cluster1 mirror image_9 snap 2024-09-11T06:40:29.427 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:40:29.427 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:40:29.427 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_9 2024-09-11T06:40:29.427 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:40:29.427 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:40:29.427 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T06:40:29.427 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:40:30.427 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 info mirror/image_9@snap 2024-09-11T06:40:30.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:rbd image 'image_9': 2024-09-11T06:40:30.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: size 128 MiB in 32 objects 2024-09-11T06:40:30.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: order 22 (4 MiB objects) 2024-09-11T06:40:30.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: snapshot_count: 1 2024-09-11T06:40:30.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: id: 10a68b2c8d10 2024-09-11T06:40:30.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: block_name_prefix: rbd_data.10a68b2c8d10 2024-09-11T06:40:30.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: format: 2 2024-09-11T06:40:30.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: features: layering, exclusive-lock, journaling 2024-09-11T06:40:30.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: op_features: 2024-09-11T06:40:30.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: flags: 2024-09-11T06:40:30.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: create_timestamp: Wed Sep 11 06:32:12 2024 2024-09-11T06:40:30.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: access_timestamp: Wed Sep 11 06:32:12 2024 2024-09-11T06:40:30.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: modify_timestamp: Wed Sep 11 06:33:48 2024 2024-09-11T06:40:30.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: protected: False 2024-09-11T06:40:30.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: journal: 10a68b2c8d10 2024-09-11T06:40:30.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring state: enabled 2024-09-11T06:40:30.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring mode: journal 2024-09-11T06:40:30.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring global id: 1b552b14-7335-41aa-9e5f-1ff228316905 2024-09-11T06:40:30.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring primary: false 2024-09-11T06:40:30.502 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:40:30.502 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ compare_image_snaps mirror image_9 snap 2024-09-11T06:40:30.502 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:40:30.502 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_9 2024-09-11T06:40:30.502 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:40:30.502 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local ret=0 2024-09-11T06:40:30.502 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local rmt_export=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_9.export 2024-09-11T06:40:30.502 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loc_export=/tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_9.export 2024-09-11T06:40:30.502 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_9.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_9.export 2024-09-11T06:40:30.502 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror export image_9@snap /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_9.export 2024-09-11T06:40:30.712 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T06:40:30.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 -p mirror export image_9@snap /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_9.export 2024-09-11T06:40:31.310 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T06:40:31.324 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmp /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_9.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_9.export 2024-09-11T06:40:31.404 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_9.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_9.export 2024-09-11T06:40:31.446 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:40:31.446 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:40:31.446 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_10 2024-09-11T06:40:31.446 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_snap cluster2 mirror image_10 snap 2024-09-11T06:40:31.446 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:40:31.447 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:40:31.447 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_10 2024-09-11T06:40:31.447 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:40:31.447 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror snap create image_10@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T06:40:35.412 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_started cluster1 mirror image_10 2024-09-11T06:40:35.412 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:40:35.412 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:40:35.413 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_10 2024-09-11T06:40:35.413 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_state cluster1 mirror image_10 started 2024-09-11T06:40:35.413 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:40:35.413 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:40:35.413 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_10 2024-09-11T06:40:35.413 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=started 2024-09-11T06:40:35.413 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:40:35.413 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:40:35.413 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:40:36.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_replay_state cluster1 mirror image_10 started 2024-09-11T06:40:36.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:40:36.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:40:36.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_10 2024-09-11T06:40:36.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=started 2024-09-11T06:40:36.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local status_result 2024-09-11T06:40:36.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=stopped 2024-09-11T06:40:36.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_10 2024-09-11T06:40:36.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -i state 2024-09-11T06:40:36.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_instance=cluster1 2024-09-11T06:40:36.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:40:36.416 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:40:36.416 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance=cluster1 2024-09-11T06:40:36.416 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local loop_instance 2024-09-11T06:40:36.416 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:40:36.416 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sleep 0 2024-09-11T06:40:36.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:40:36.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ seq 0 1 2024-09-11T06:40:36.420 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:40:36.420 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_10 2024-09-11T06:40:36.420 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1:0 2024-09-11T06:40:36.420 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:40:36.420 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:40:36.420 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:40:36.420 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:40:36.420 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=cluster 2024-09-11T06:40:36.420 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:40:36.420 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:40:36.421 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:40:36.421 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:40:36.421 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval cluster=cluster1 2024-09-11T06:40:36.421 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:40:36.421 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:40:36.421 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:40:36.421 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:40:36.421 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local local_cluster=cluster1:0 2024-09-11T06:40:36.421 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster=cluster1 2024-09-11T06:40:36.421 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance 2024-09-11T06:40:36.421 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:40:36.421 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local val=cluster1:0 2024-09-11T06:40:36.421 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T06:40:36.421 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance_var_name=instance 2024-09-11T06:40:36.422 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:40:36.422 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:40:36.422 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T06:40:36.422 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval local_cluster=cluster1 2024-09-11T06:40:36.422 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ local_cluster=cluster1 2024-09-11T06:40:36.423 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval instance=0 2024-09-11T06:40:36.423 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ instance=0 2024-09-11T06:40:36.423 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:40:36.441 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:40:36.442 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:40:36.442 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:40:36.442 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_10 2024-09-11T06:40:36.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2024-09-11T06:40:36.572 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T06:40:36.572 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:admin_socket: invalid command 2024-09-11T06:40:36.578 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:40:36.578 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_10 2024-09-11T06:40:36.578 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1:1 2024-09-11T06:40:36.578 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:40:36.578 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:40:36.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-11T06:40:36.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:1 2024-09-11T06:40:36.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=cluster 2024-09-11T06:40:36.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:40:36.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:40:36.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=1 2024-09-11T06:40:36.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T06:40:36.580 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval cluster=cluster1 2024-09-11T06:40:36.580 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:40:36.580 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=1 2024-09-11T06:40:36.580 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=1 2024-09-11T06:40:36.580 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-11T06:40:36.580 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local local_cluster=cluster1:1 2024-09-11T06:40:36.580 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster=cluster1 2024-09-11T06:40:36.580 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance 2024-09-11T06:40:36.580 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T06:40:36.580 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local val=cluster1:1 2024-09-11T06:40:36.580 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T06:40:36.580 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance_var_name=instance 2024-09-11T06:40:36.580 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:40:36.580 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=1 2024-09-11T06:40:36.581 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-11T06:40:36.581 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval local_cluster=cluster1 2024-09-11T06:40:36.581 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ local_cluster=cluster1 2024-09-11T06:40:36.581 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval instance=1 2024-09-11T06:40:36.581 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ instance=1 2024-09-11T06:40:36.581 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T06:40:36.598 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:40:36.598 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:40:36.598 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:40:36.598 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_10 2024-09-11T06:40:36.730 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ return 0 2024-09-11T06:40:36.731 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T06:40:36.732 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ echo ' "state": "Replaying"' 2024-09-11T06:40:36.732 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ grep -i Replaying 2024-09-11T06:40:36.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: "state": "Replaying" 2024-09-11T06:40:36.735 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_state=started 2024-09-11T06:40:36.735 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test started = started 2024-09-11T06:40:36.735 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:40:36.735 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_10 2024-09-11T06:40:36.735 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:40:36.735 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:40:36.735 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:40:36.735 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_10 2024-09-11T06:40:36.736 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' journal = journal ']' 2024-09-11T06:40:36.736 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_10 2024-09-11T06:40:36.736 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:40:36.736 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:40:36.736 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:40:36.736 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_10 2024-09-11T06:40:36.736 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-11T06:40:36.736 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-11T06:40:36.736 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:40:36.736 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:40:36.736 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:40:36.937 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror image_10 2024-09-11T06:40:36.937 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:40:36.937 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:40:36.937 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_10 2024-09-11T06:40:36.937 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:40:36.938 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n image_10 ']' 2024-09-11T06:40:36.938 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/image_10' 2024-09-11T06:40:36.938 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_10 2024-09-11T06:40:36.938 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:40:36.938 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:40:36.939 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:40:36.939 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:40:36.939 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:40:36.939 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:40:36.939 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:40:36.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:40:36.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:40:36.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:40:36.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_10 2024-09-11T06:40:36.946 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:40:36.946 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:40:36.946 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:40:36.946 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:40:36.946 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:40:36.947 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:40:36.947 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:40:36.947 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:40:36.947 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:40:36.947 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:40:36.947 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:40:36.948 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:40:36.948 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:40:36.948 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:40:36.948 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:40:36.948 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:40:36.948 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:40:36.948 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:40:36.949 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:40:36.949 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:40:36.949 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:40:36.949 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:40:36.949 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:40:36.949 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:40:36.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:40:36.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:40:36.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:40:36.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:40:36.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:40:36.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:40:36.966 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:40:36.967 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:40:36.967 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:40:36.967 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_10 2024-09-11T06:40:37.091 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2024-09-11T06:40:37.091 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd mirror flush : flush rbd mirror 2024-09-11T06:40:37.092 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:admin_socket: invalid command 2024-09-11T06:40:37.095 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:40:37.095 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_10 2024-09-11T06:40:37.095 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:1 2024-09-11T06:40:37.095 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:40:37.095 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:40:37.095 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-11T06:40:37.095 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:1 2024-09-11T06:40:37.095 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:40:37.095 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:40:37.095 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:40:37.095 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=1 2024-09-11T06:40:37.095 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-11T06:40:37.095 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:40:37.096 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:40:37.096 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=1 2024-09-11T06:40:37.096 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=1 2024-09-11T06:40:37.096 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-11T06:40:37.096 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:1 2024-09-11T06:40:37.096 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:40:37.096 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:40:37.096 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T06:40:37.096 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:1 2024-09-11T06:40:37.096 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:40:37.096 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:40:37.096 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:40:37.097 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=1 2024-09-11T06:40:37.097 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T06:40:37.097 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:40:37.097 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:40:37.097 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=1 2024-09-11T06:40:37.097 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=1 2024-09-11T06:40:37.097 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T06:40:37.115 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:40:37.115 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:40:37.115 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:40:37.115 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_10 2024-09-11T06:40:37.238 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:40:37.239 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror image_10 2024-09-11T06:40:37.239 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:40:37.239 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:40:37.239 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_10 2024-09-11T06:40:37.239 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror image_10 '' 2024-09-11T06:40:37.240 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:40:37.240 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:40:37.240 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_10 2024-09-11T06:40:37.240 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:40:37.240 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-image_10.status 2024-09-11T06:40:37.240 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-image_10.status 2024-09-11T06:40:37.240 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:40:37.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_10.status 2024-09-11T06:40:37.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_10 2024-09-11T06:40:37.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_10.status 2024-09-11T06:40:37.293 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 2 2024-09-11T06:40:37.293 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 2 2024-09-11T06:40:37.293 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:40:37.293 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=8007], [object_number=10, tag_tid=1, entry_tid=8006]]], state=connected] 2024-09-11T06:40:37.294 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=8, tag_tid=1, entry_tid=8008], [object_number=11, tag_tid=1, entry_tid=8007], [object_number=10, tag_tid=1, entry_tid=8006], [object_number=9, tag_tid=1, entry_tid=8005]]], state=connected] 2024-09-11T06:40:37.297 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_10.status 2024-09-11T06:40:37.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-09-11T06:40:37.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror image_10 2024-09-11T06:40:37.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:40:37.300 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:40:37.300 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_10 2024-09-11T06:40:37.300 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror image_10 '..*' 2024-09-11T06:40:37.300 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:40:37.300 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:40:37.300 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_10 2024-09-11T06:40:37.300 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:40:37.300 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-image_10.status 2024-09-11T06:40:37.300 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-image_10.status 2024-09-11T06:40:37.300 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:40:37.301 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_10.status 2024-09-11T06:40:37.301 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_10 2024-09-11T06:40:37.301 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_10.status 2024-09-11T06:40:37.346 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 2 2024-09-11T06:40:37.346 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 2 2024-09-11T06:40:37.346 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:40:37.346 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=8007], [object_number=10, tag_tid=1, entry_tid=8006]]], state=connected] 2024-09-11T06:40:37.346 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=8, tag_tid=1, entry_tid=8008], [object_number=11, tag_tid=1, entry_tid=8007], [object_number=10, tag_tid=1, entry_tid=8006], [object_number=9, tag_tid=1, entry_tid=8005]]], state=connected] 2024-09-11T06:40:37.349 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_10.status 2024-09-11T06:40:37.352 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=8, tag_tid=1, entry_tid=8008' 2024-09-11T06:40:37.352 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=8, tag_tid=1, entry_tid=8008' 2024-09-11T06:40:37.352 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=8, tag_tid=1, entry_tid=8008' '!=' '' 2024-09-11T06:40:37.352 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:40:37.352 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=8, tag_tid=1, entry_tid=8008' = '' 2024-09-11T06:40:37.352 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=8, tag_tid=1, entry_tid=8008' 2024-09-11T06:40:37.352 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=1' 2024-09-11T06:40:37.352 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:40:37.352 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:40:37.353 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=2 2024-09-11T06:40:37.353 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=8, tag_tid=1, entry_tid=8008' 2024-09-11T06:40:37.353 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:40:37.353 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:40:37.354 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=1 2024-09-11T06:40:37.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=1' 2024-09-11T06:40:37.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:40:37.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:40:37.356 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:40:37.357 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=8, tag_tid=1, entry_tid=8008' 2024-09-11T06:40:37.357 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:40:37.357 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:40:37.359 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=8008 2024-09-11T06:40:37.359 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 2 = 1 -a 1 -le 8008 2024-09-11T06:40:37.359 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:40:37.359 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:40:37.359 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:40:37.560 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror image_10 2024-09-11T06:40:37.560 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:40:37.560 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:40:37.560 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_10 2024-09-11T06:40:37.561 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:40:37.561 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n image_10 ']' 2024-09-11T06:40:37.561 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/image_10' 2024-09-11T06:40:37.561 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_10 2024-09-11T06:40:37.561 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:40:37.561 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:40:37.561 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:40:37.561 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:40:37.561 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:40:37.561 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:40:37.561 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:40:37.562 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:40:37.562 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:40:37.565 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:40:37.565 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_10 2024-09-11T06:40:37.565 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:40:37.565 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:40:37.565 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:40:37.566 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:40:37.566 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:40:37.566 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:40:37.566 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:40:37.566 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:40:37.566 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:40:37.566 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:40:37.566 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:40:37.566 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:40:37.566 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:40:37.566 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:40:37.566 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:40:37.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:40:37.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:40:37.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:40:37.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:40:37.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:40:37.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:40:37.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:40:37.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:40:37.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:40:37.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:40:37.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:40:37.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:40:37.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:40:37.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:40:37.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:40:37.590 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:40:37.590 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:40:37.591 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:40:37.591 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_10 2024-09-11T06:40:37.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2024-09-11T06:40:37.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd mirror flush : flush rbd mirror 2024-09-11T06:40:37.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:admin_socket: invalid command 2024-09-11T06:40:37.712 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:40:37.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_10 2024-09-11T06:40:37.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:1 2024-09-11T06:40:37.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:40:37.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:40:37.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-11T06:40:37.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:1 2024-09-11T06:40:37.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:40:37.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:40:37.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:40:37.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=1 2024-09-11T06:40:37.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-11T06:40:37.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:40:37.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:40:37.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=1 2024-09-11T06:40:37.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=1 2024-09-11T06:40:37.716 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-11T06:40:37.716 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:1 2024-09-11T06:40:37.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:40:37.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:40:37.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T06:40:37.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:1 2024-09-11T06:40:37.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:40:37.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:40:37.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:40:37.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=1 2024-09-11T06:40:37.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T06:40:37.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:40:37.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:40:37.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=1 2024-09-11T06:40:37.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=1 2024-09-11T06:40:37.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T06:40:37.734 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:40:37.734 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:40:37.734 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:40:37.734 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_10 2024-09-11T06:40:37.883 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:40:37.884 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror image_10 2024-09-11T06:40:37.884 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:40:37.884 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:40:37.884 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_10 2024-09-11T06:40:37.884 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror image_10 '' 2024-09-11T06:40:37.884 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:40:37.885 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:40:37.885 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_10 2024-09-11T06:40:37.885 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:40:37.885 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-image_10.status 2024-09-11T06:40:37.885 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-image_10.status 2024-09-11T06:40:37.885 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:40:37.887 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_10.status 2024-09-11T06:40:37.887 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_10 2024-09-11T06:40:37.887 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_10.status 2024-09-11T06:40:37.938 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 2 2024-09-11T06:40:37.939 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 2 2024-09-11T06:40:37.939 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:40:37.939 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=8007], [object_number=10, tag_tid=1, entry_tid=8006]]], state=connected] 2024-09-11T06:40:37.939 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=8007], [object_number=10, tag_tid=1, entry_tid=8006]]], state=connected] 2024-09-11T06:40:37.942 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_10.status 2024-09-11T06:40:37.943 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-09-11T06:40:37.944 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror image_10 2024-09-11T06:40:37.944 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:40:37.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:40:37.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_10 2024-09-11T06:40:37.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror image_10 '..*' 2024-09-11T06:40:37.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:40:37.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:40:37.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_10 2024-09-11T06:40:37.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:40:37.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-image_10.status 2024-09-11T06:40:37.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-image_10.status 2024-09-11T06:40:37.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:40:37.946 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_10.status 2024-09-11T06:40:37.946 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_10 2024-09-11T06:40:37.946 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_10.status 2024-09-11T06:40:37.992 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 2 2024-09-11T06:40:37.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 2 2024-09-11T06:40:37.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:40:37.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=8007], [object_number=10, tag_tid=1, entry_tid=8006]]], state=connected] 2024-09-11T06:40:37.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=11, tag_tid=1, entry_tid=8007], [object_number=10, tag_tid=1, entry_tid=8006]]], state=connected] 2024-09-11T06:40:37.996 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_10.status 2024-09-11T06:40:37.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-09-11T06:40:37.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=1' 2024-09-11T06:40:37.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:40:37.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_snap_present cluster1 mirror image_10 snap 2024-09-11T06:40:37.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:40:37.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:40:37.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_10 2024-09-11T06:40:37.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:40:37.999 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:40:37.999 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T06:40:37.999 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:40:38.999 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 info mirror/image_10@snap 2024-09-11T06:40:39.065 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:rbd image 'image_10': 2024-09-11T06:40:39.065 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: size 128 MiB in 32 objects 2024-09-11T06:40:39.065 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: order 22 (4 MiB objects) 2024-09-11T06:40:39.065 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: snapshot_count: 1 2024-09-11T06:40:39.065 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: id: 10a995cc8acc 2024-09-11T06:40:39.065 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: block_name_prefix: rbd_data.10a995cc8acc 2024-09-11T06:40:39.065 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: format: 2 2024-09-11T06:40:39.065 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: features: layering, exclusive-lock, journaling 2024-09-11T06:40:39.065 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: op_features: 2024-09-11T06:40:39.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: flags: 2024-09-11T06:40:39.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: create_timestamp: Wed Sep 11 06:32:20 2024 2024-09-11T06:40:39.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: access_timestamp: Wed Sep 11 06:32:20 2024 2024-09-11T06:40:39.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: modify_timestamp: Wed Sep 11 06:32:20 2024 2024-09-11T06:40:39.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: protected: False 2024-09-11T06:40:39.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: journal: 10a995cc8acc 2024-09-11T06:40:39.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring state: enabled 2024-09-11T06:40:39.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring mode: journal 2024-09-11T06:40:39.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring global id: 71d26011-5f7f-414e-b39f-06fc5bc462a5 2024-09-11T06:40:39.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring primary: false 2024-09-11T06:40:39.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:40:39.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ compare_image_snaps mirror image_10 snap 2024-09-11T06:40:39.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:40:39.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_10 2024-09-11T06:40:39.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:40:39.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local ret=0 2024-09-11T06:40:39.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local rmt_export=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_10.export 2024-09-11T06:40:39.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loc_export=/tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_10.export 2024-09-11T06:40:39.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_10.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_10.export 2024-09-11T06:40:39.071 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror export image_10@snap /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_10.export 2024-09-11T06:40:39.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T06:40:39.258 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 -p mirror export image_10@snap /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_10.export 2024-09-11T06:40:39.741 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T06:40:39.758 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmp /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_10.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_10.export 2024-09-11T06:40:39.839 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_10.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_10.export 2024-09-11T06:40:39.890 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:40:39.890 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:40:39.890 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_11 2024-09-11T06:40:39.890 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_snap cluster2 mirror image_11 snap 2024-09-11T06:40:39.890 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:40:39.890 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:40:39.891 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_11 2024-09-11T06:40:39.891 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:40:39.891 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror snap create image_11@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T06:40:40.592 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_started cluster1 mirror image_11 2024-09-11T06:40:40.592 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:40:40.592 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:40:40.592 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_11 2024-09-11T06:40:40.593 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_state cluster1 mirror image_11 started 2024-09-11T06:40:40.593 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:40:40.593 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:40:40.593 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_11 2024-09-11T06:40:40.593 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=started 2024-09-11T06:40:40.593 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:40:40.593 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:40:40.593 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:40:41.594 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_replay_state cluster1 mirror image_11 started 2024-09-11T06:40:41.594 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:40:41.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:40:41.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_11 2024-09-11T06:40:41.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=started 2024-09-11T06:40:41.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local status_result 2024-09-11T06:40:41.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=stopped 2024-09-11T06:40:41.596 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_11 2024-09-11T06:40:41.596 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -i state 2024-09-11T06:40:41.596 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_instance=cluster1 2024-09-11T06:40:41.596 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:40:41.597 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:40:41.597 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance=cluster1 2024-09-11T06:40:41.597 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local loop_instance 2024-09-11T06:40:41.597 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:40:41.597 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sleep 0 2024-09-11T06:40:41.597 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:40:41.597 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ seq 0 1 2024-09-11T06:40:41.601 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:40:41.601 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_11 2024-09-11T06:40:41.601 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1:0 2024-09-11T06:40:41.601 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:40:41.601 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:40:41.601 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:40:41.601 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:40:41.601 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=cluster 2024-09-11T06:40:41.601 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:40:41.601 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:40:41.601 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:40:41.601 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:40:41.602 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval cluster=cluster1 2024-09-11T06:40:41.602 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:40:41.602 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:40:41.602 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:40:41.602 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:40:41.602 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local local_cluster=cluster1:0 2024-09-11T06:40:41.602 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster=cluster1 2024-09-11T06:40:41.602 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance 2024-09-11T06:40:41.602 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:40:41.602 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local val=cluster1:0 2024-09-11T06:40:41.602 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T06:40:41.602 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance_var_name=instance 2024-09-11T06:40:41.602 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:40:41.602 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:40:41.603 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T06:40:41.603 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval local_cluster=cluster1 2024-09-11T06:40:41.603 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ local_cluster=cluster1 2024-09-11T06:40:41.603 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval instance=0 2024-09-11T06:40:41.603 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ instance=0 2024-09-11T06:40:41.603 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:40:41.626 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:40:41.626 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:40:41.626 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:40:41.626 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_11 2024-09-11T06:40:41.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ return 0 2024-09-11T06:40:41.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T06:40:41.764 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ echo ' "state": "Replaying"' 2024-09-11T06:40:41.764 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ grep -i Replaying 2024-09-11T06:40:41.765 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: "state": "Replaying" 2024-09-11T06:40:41.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_state=started 2024-09-11T06:40:41.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test started = started 2024-09-11T06:40:41.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:40:41.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_11 2024-09-11T06:40:41.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:40:41.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:40:41.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:40:41.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_11 2024-09-11T06:40:41.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' journal = journal ']' 2024-09-11T06:40:41.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_11 2024-09-11T06:40:41.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:40:41.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:40:41.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:40:41.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_11 2024-09-11T06:40:41.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-11T06:40:41.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-11T06:40:41.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:40:41.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:40:41.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:40:41.967 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror image_11 2024-09-11T06:40:41.968 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:40:41.968 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:40:41.968 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_11 2024-09-11T06:40:41.968 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:40:41.968 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n image_11 ']' 2024-09-11T06:40:41.968 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/image_11' 2024-09-11T06:40:41.968 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_11 2024-09-11T06:40:41.968 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:40:41.968 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:40:41.968 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:40:41.968 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:40:41.968 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:40:41.968 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:40:41.969 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:40:41.969 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:40:41.969 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:40:41.972 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:40:41.972 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_11 2024-09-11T06:40:41.972 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:40:41.972 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:40:41.972 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:40:41.972 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:40:41.972 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:40:41.972 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:40:41.972 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:40:41.973 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:40:41.973 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:40:41.973 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:40:41.973 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:40:41.973 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:40:41.973 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:40:41.973 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:40:41.973 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:40:41.973 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:40:41.973 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:40:41.973 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:40:41.973 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:40:41.973 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:40:41.973 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:40:41.974 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:40:41.974 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:40:41.974 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:40:41.974 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:40:41.974 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:40:41.974 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:40:41.974 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:40:41.974 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:40:41.974 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:40:41.999 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:40:41.999 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:40:41.999 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:40:41.999 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_11 2024-09-11T06:40:42.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:40:42.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror image_11 2024-09-11T06:40:42.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:40:42.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:40:42.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_11 2024-09-11T06:40:42.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror image_11 '' 2024-09-11T06:40:42.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:40:42.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:40:42.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_11 2024-09-11T06:40:42.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:40:42.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-image_11.status 2024-09-11T06:40:42.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-image_11.status 2024-09-11T06:40:42.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:40:42.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_11.status 2024-09-11T06:40:42.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_11 2024-09-11T06:40:42.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_11.status 2024-09-11T06:40:42.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 1 2024-09-11T06:40:42.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 2 2024-09-11T06:40:42.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:40:42.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=3, entry_tid=1], [object_number=8, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6895], [object_number=6, tag_tid=1, entry_tid=6894]]], state=connected] 2024-09-11T06:40:42.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6895], [object_number=6, tag_tid=1, entry_tid=6894]]], state=connected] 2024-09-11T06:40:42.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_11.status 2024-09-11T06:40:42.196 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=9, tag_tid=3, entry_tid=1' 2024-09-11T06:40:42.197 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror image_11 2024-09-11T06:40:42.197 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:40:42.197 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:40:42.197 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_11 2024-09-11T06:40:42.197 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror image_11 '..*' 2024-09-11T06:40:42.197 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:40:42.197 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:40:42.197 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_11 2024-09-11T06:40:42.197 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:40:42.197 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-image_11.status 2024-09-11T06:40:42.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-image_11.status 2024-09-11T06:40:42.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:40:42.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_11.status 2024-09-11T06:40:42.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_11 2024-09-11T06:40:42.199 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_11.status 2024-09-11T06:40:42.245 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 1 2024-09-11T06:40:42.245 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 2 2024-09-11T06:40:42.245 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:40:42.246 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=3, entry_tid=1], [object_number=8, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6895], [object_number=6, tag_tid=1, entry_tid=6894]]], state=connected] 2024-09-11T06:40:42.246 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6895], [object_number=6, tag_tid=1, entry_tid=6894]]], state=connected] 2024-09-11T06:40:42.248 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_11.status 2024-09-11T06:40:42.250 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-09-11T06:40:42.250 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=9, tag_tid=3, entry_tid=1' -a 'object_number=9, tag_tid=3, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=1' 2024-09-11T06:40:42.250 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=9, tag_tid=2, entry_tid=1' '!=' '' 2024-09-11T06:40:42.250 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:40:42.250 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=9, tag_tid=2, entry_tid=1' = '' 2024-09-11T06:40:42.250 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-09-11T06:40:42.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=9, tag_tid=3, entry_tid=1' 2024-09-11T06:40:42.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:40:42.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:40:42.252 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=3 2024-09-11T06:40:42.252 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=1' 2024-09-11T06:40:42.252 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:40:42.252 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:40:42.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=2 2024-09-11T06:40:42.254 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=9, tag_tid=3, entry_tid=1' 2024-09-11T06:40:42.254 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:40:42.254 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:40:42.255 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:40:42.256 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=9, tag_tid=2, entry_tid=1' 2024-09-11T06:40:42.256 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:40:42.256 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:40:42.257 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=1 2024-09-11T06:40:42.257 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 3 = 2 -a 1 -le 1 2024-09-11T06:40:42.257 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:40:42.257 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:40:42.257 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:40:42.459 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror image_11 2024-09-11T06:40:42.459 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:40:42.459 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:40:42.459 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_11 2024-09-11T06:40:42.459 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:40:42.459 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n image_11 ']' 2024-09-11T06:40:42.459 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/image_11' 2024-09-11T06:40:42.459 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_11 2024-09-11T06:40:42.460 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:40:42.460 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:40:42.460 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:40:42.460 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:40:42.460 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:40:42.460 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:40:42.460 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:40:42.461 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:40:42.461 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:40:42.464 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:40:42.464 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_11 2024-09-11T06:40:42.464 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:40:42.464 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:40:42.464 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:40:42.464 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:40:42.464 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:40:42.464 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:40:42.464 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:40:42.464 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:40:42.464 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:40:42.465 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:40:42.465 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:40:42.465 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:40:42.465 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:40:42.465 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:40:42.465 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:40:42.465 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:40:42.465 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:40:42.465 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:40:42.465 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:40:42.465 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:40:42.465 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:40:42.465 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:40:42.466 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:40:42.466 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:40:42.466 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:40:42.466 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:40:42.466 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:40:42.466 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:40:42.466 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:40:42.466 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:40:42.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:40:42.490 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:40:42.490 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:40:42.490 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_11 2024-09-11T06:40:42.638 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:40:42.640 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror image_11 2024-09-11T06:40:42.640 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:40:42.640 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:40:42.640 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_11 2024-09-11T06:40:42.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror image_11 '' 2024-09-11T06:40:42.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:40:42.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:40:42.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_11 2024-09-11T06:40:42.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:40:42.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-image_11.status 2024-09-11T06:40:42.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-image_11.status 2024-09-11T06:40:42.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:40:42.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_11.status 2024-09-11T06:40:42.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_11 2024-09-11T06:40:42.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_11.status 2024-09-11T06:40:42.689 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 1 2024-09-11T06:40:42.689 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 2 2024-09-11T06:40:42.689 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:40:42.689 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=3, entry_tid=1], [object_number=8, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6895], [object_number=6, tag_tid=1, entry_tid=6894]]], state=connected] 2024-09-11T06:40:42.689 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=9, tag_tid=3, entry_tid=1], [object_number=8, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6895], [object_number=6, tag_tid=1, entry_tid=6894]]], state=connected] 2024-09-11T06:40:42.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_11.status 2024-09-11T06:40:42.693 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=9, tag_tid=3, entry_tid=1' 2024-09-11T06:40:42.694 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror image_11 2024-09-11T06:40:42.694 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:40:42.694 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:40:42.694 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_11 2024-09-11T06:40:42.694 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror image_11 '..*' 2024-09-11T06:40:42.695 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:40:42.695 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:40:42.695 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_11 2024-09-11T06:40:42.695 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:40:42.695 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-image_11.status 2024-09-11T06:40:42.695 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-image_11.status 2024-09-11T06:40:42.695 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:40:42.696 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_11.status 2024-09-11T06:40:42.696 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_11 2024-09-11T06:40:42.696 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_11.status 2024-09-11T06:40:42.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 1 2024-09-11T06:40:42.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 2 2024-09-11T06:40:42.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:40:42.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=3, entry_tid=1], [object_number=8, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6895], [object_number=6, tag_tid=1, entry_tid=6894]]], state=connected] 2024-09-11T06:40:42.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=9, tag_tid=3, entry_tid=1], [object_number=8, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=6895], [object_number=6, tag_tid=1, entry_tid=6894]]], state=connected] 2024-09-11T06:40:42.746 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_11.status 2024-09-11T06:40:42.748 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=9, tag_tid=3, entry_tid=1' 2024-09-11T06:40:42.748 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=9, tag_tid=3, entry_tid=1' -a 'object_number=9, tag_tid=3, entry_tid=1' = 'object_number=9, tag_tid=3, entry_tid=1' 2024-09-11T06:40:42.748 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:40:42.748 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_snap_present cluster1 mirror image_11 snap 2024-09-11T06:40:42.748 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:40:42.748 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:40:42.748 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_11 2024-09-11T06:40:42.748 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:40:42.748 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:40:42.748 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T06:40:42.749 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:40:43.749 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 info mirror/image_11@snap 2024-09-11T06:40:43.813 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:rbd image 'image_11': 2024-09-11T06:40:43.813 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: size 128 MiB in 32 objects 2024-09-11T06:40:43.813 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: order 22 (4 MiB objects) 2024-09-11T06:40:43.813 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: snapshot_count: 1 2024-09-11T06:40:43.813 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: id: 10a6b44818cc 2024-09-11T06:40:43.813 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: block_name_prefix: rbd_data.10a6b44818cc 2024-09-11T06:40:43.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: format: 2 2024-09-11T06:40:43.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: features: layering, exclusive-lock, journaling 2024-09-11T06:40:43.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: op_features: 2024-09-11T06:40:43.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: flags: 2024-09-11T06:40:43.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: create_timestamp: Wed Sep 11 06:32:32 2024 2024-09-11T06:40:43.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: access_timestamp: Wed Sep 11 06:32:32 2024 2024-09-11T06:40:43.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: modify_timestamp: Wed Sep 11 06:33:32 2024 2024-09-11T06:40:43.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: protected: False 2024-09-11T06:40:43.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: journal: 10a6b44818cc 2024-09-11T06:40:43.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring state: enabled 2024-09-11T06:40:43.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring mode: journal 2024-09-11T06:40:43.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring global id: 2034def0-c9ca-4709-a78f-5ce40e33bcee 2024-09-11T06:40:43.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring primary: false 2024-09-11T06:40:43.817 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:40:43.818 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ compare_image_snaps mirror image_11 snap 2024-09-11T06:40:43.818 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:40:43.818 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_11 2024-09-11T06:40:43.818 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:40:43.818 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local ret=0 2024-09-11T06:40:43.818 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local rmt_export=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_11.export 2024-09-11T06:40:43.818 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loc_export=/tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_11.export 2024-09-11T06:40:43.818 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_11.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_11.export 2024-09-11T06:40:43.818 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror export image_11@snap /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_11.export 2024-09-11T06:40:43.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T06:40:44.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 -p mirror export image_11@snap /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_11.export 2024-09-11T06:40:44.287 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T06:40:44.298 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmp /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_11.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_11.export 2024-09-11T06:40:44.375 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_11.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_11.export 2024-09-11T06:40:44.418 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:40:44.418 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:40:44.418 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_12 2024-09-11T06:40:44.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_snap cluster2 mirror image_12 snap 2024-09-11T06:40:44.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:40:44.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:40:44.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_12 2024-09-11T06:40:44.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:40:44.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror snap create image_12@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T06:40:46.628 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_started cluster1 mirror image_12 2024-09-11T06:40:46.628 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:40:46.628 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:40:46.629 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_12 2024-09-11T06:40:46.629 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_state cluster1 mirror image_12 started 2024-09-11T06:40:46.629 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:40:46.629 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:40:46.629 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_12 2024-09-11T06:40:46.629 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=started 2024-09-11T06:40:46.629 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:40:46.629 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:40:46.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:40:47.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_replay_state cluster1 mirror image_12 started 2024-09-11T06:40:47.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:40:47.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:40:47.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_12 2024-09-11T06:40:47.631 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=started 2024-09-11T06:40:47.631 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local status_result 2024-09-11T06:40:47.631 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=stopped 2024-09-11T06:40:47.632 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_12 2024-09-11T06:40:47.632 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -i state 2024-09-11T06:40:47.632 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_instance=cluster1 2024-09-11T06:40:47.632 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:40:47.632 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:40:47.632 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance=cluster1 2024-09-11T06:40:47.632 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local loop_instance 2024-09-11T06:40:47.632 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:40:47.632 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sleep 0 2024-09-11T06:40:47.633 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:40:47.633 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ seq 0 1 2024-09-11T06:40:47.634 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:40:47.635 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_12 2024-09-11T06:40:47.635 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1:0 2024-09-11T06:40:47.635 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:40:47.635 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:40:47.635 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:40:47.635 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:40:47.635 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=cluster 2024-09-11T06:40:47.635 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:40:47.635 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:40:47.635 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:40:47.635 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:40:47.637 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval cluster=cluster1 2024-09-11T06:40:47.637 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:40:47.638 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:40:47.638 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:40:47.638 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:40:47.638 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local local_cluster=cluster1:0 2024-09-11T06:40:47.638 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster=cluster1 2024-09-11T06:40:47.638 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance 2024-09-11T06:40:47.638 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:40:47.638 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local val=cluster1:0 2024-09-11T06:40:47.638 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T06:40:47.638 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance_var_name=instance 2024-09-11T06:40:47.638 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:40:47.638 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:40:47.638 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T06:40:47.638 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval local_cluster=cluster1 2024-09-11T06:40:47.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ local_cluster=cluster1 2024-09-11T06:40:47.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval instance=0 2024-09-11T06:40:47.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ instance=0 2024-09-11T06:40:47.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:40:47.659 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:40:47.659 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:40:47.660 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:40:47.660 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_12 2024-09-11T06:40:47.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2024-09-11T06:40:47.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T06:40:47.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:admin_socket: invalid command 2024-09-11T06:40:47.790 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:40:47.790 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_12 2024-09-11T06:40:47.790 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1:1 2024-09-11T06:40:47.790 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:40:47.790 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:40:47.790 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-11T06:40:47.790 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:1 2024-09-11T06:40:47.790 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=cluster 2024-09-11T06:40:47.790 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:40:47.791 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:40:47.791 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=1 2024-09-11T06:40:47.791 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T06:40:47.791 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval cluster=cluster1 2024-09-11T06:40:47.791 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:40:47.791 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=1 2024-09-11T06:40:47.791 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=1 2024-09-11T06:40:47.791 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-11T06:40:47.791 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local local_cluster=cluster1:1 2024-09-11T06:40:47.791 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster=cluster1 2024-09-11T06:40:47.791 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance 2024-09-11T06:40:47.791 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T06:40:47.791 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local val=cluster1:1 2024-09-11T06:40:47.791 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T06:40:47.792 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance_var_name=instance 2024-09-11T06:40:47.792 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:40:47.792 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=1 2024-09-11T06:40:47.792 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-11T06:40:47.792 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval local_cluster=cluster1 2024-09-11T06:40:47.792 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ local_cluster=cluster1 2024-09-11T06:40:47.792 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval instance=1 2024-09-11T06:40:47.792 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ instance=1 2024-09-11T06:40:47.792 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T06:40:47.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:40:47.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:40:47.810 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:40:47.810 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_12 2024-09-11T06:40:47.939 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ return 0 2024-09-11T06:40:47.939 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T06:40:47.940 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ echo ' "state": "Replaying"' 2024-09-11T06:40:47.941 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ grep -i Replaying 2024-09-11T06:40:47.942 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: "state": "Replaying" 2024-09-11T06:40:47.943 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_state=started 2024-09-11T06:40:47.943 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test started = started 2024-09-11T06:40:47.943 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:40:47.944 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_12 2024-09-11T06:40:47.944 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:40:47.944 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:40:47.944 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:40:47.944 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_12 2024-09-11T06:40:47.944 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' journal = journal ']' 2024-09-11T06:40:47.944 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_12 2024-09-11T06:40:47.944 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:40:47.944 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:40:47.944 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:40:47.944 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_12 2024-09-11T06:40:47.944 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-11T06:40:47.944 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-11T06:40:47.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:40:47.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:40:47.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:40:48.145 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror image_12 2024-09-11T06:40:48.145 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:40:48.145 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:40:48.145 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_12 2024-09-11T06:40:48.145 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:40:48.145 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n image_12 ']' 2024-09-11T06:40:48.145 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/image_12' 2024-09-11T06:40:48.146 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_12 2024-09-11T06:40:48.146 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:40:48.146 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:40:48.146 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:40:48.146 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:40:48.146 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:40:48.146 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:40:48.146 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:40:48.146 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:40:48.147 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:40:48.151 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:40:48.151 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_12 2024-09-11T06:40:48.151 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:40:48.151 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:40:48.151 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:40:48.151 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:40:48.151 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:40:48.151 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:40:48.152 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:40:48.152 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:40:48.152 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:40:48.152 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:40:48.152 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:40:48.152 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:40:48.152 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:40:48.152 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:40:48.152 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:40:48.152 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:40:48.152 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:40:48.152 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:40:48.152 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:40:48.152 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:40:48.153 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:40:48.153 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:40:48.153 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:40:48.153 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:40:48.153 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:40:48.153 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:40:48.153 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:40:48.153 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:40:48.153 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:40:48.153 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:40:48.175 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:40:48.175 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:40:48.175 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:40:48.175 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_12 2024-09-11T06:40:48.293 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2024-09-11T06:40:48.293 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd mirror flush : flush rbd mirror 2024-09-11T06:40:48.293 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:admin_socket: invalid command 2024-09-11T06:40:48.295 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:40:48.295 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_12 2024-09-11T06:40:48.295 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:1 2024-09-11T06:40:48.296 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:40:48.296 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:40:48.296 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-11T06:40:48.296 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:1 2024-09-11T06:40:48.296 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:40:48.296 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:40:48.296 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:40:48.296 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=1 2024-09-11T06:40:48.296 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-11T06:40:48.296 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:40:48.296 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:40:48.296 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=1 2024-09-11T06:40:48.296 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=1 2024-09-11T06:40:48.298 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-11T06:40:48.298 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:1 2024-09-11T06:40:48.298 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:40:48.298 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:40:48.298 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T06:40:48.298 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:1 2024-09-11T06:40:48.298 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:40:48.298 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:40:48.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:40:48.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=1 2024-09-11T06:40:48.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T06:40:48.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:40:48.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:40:48.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=1 2024-09-11T06:40:48.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=1 2024-09-11T06:40:48.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T06:40:48.320 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:40:48.321 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:40:48.321 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:40:48.321 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_12 2024-09-11T06:40:48.459 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:40:48.460 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror image_12 2024-09-11T06:40:48.460 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:40:48.460 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:40:48.460 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_12 2024-09-11T06:40:48.460 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror image_12 '' 2024-09-11T06:40:48.460 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:40:48.460 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:40:48.460 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_12 2024-09-11T06:40:48.460 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:40:48.461 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-image_12.status 2024-09-11T06:40:48.461 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-image_12.status 2024-09-11T06:40:48.461 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:40:48.463 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_12.status 2024-09-11T06:40:48.464 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_12 2024-09-11T06:40:48.464 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_12.status 2024-09-11T06:40:48.515 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 2 2024-09-11T06:40:48.515 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 2 2024-09-11T06:40:48.515 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:40:48.515 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=10206], [object_number=11, tag_tid=1, entry_tid=10203]]], state=connected] 2024-09-11T06:40:48.515 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=10206], [object_number=11, tag_tid=1, entry_tid=10203]]], state=connected] 2024-09-11T06:40:48.518 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_12.status 2024-09-11T06:40:48.520 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-09-11T06:40:48.521 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror image_12 2024-09-11T06:40:48.521 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:40:48.521 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:40:48.521 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_12 2024-09-11T06:40:48.521 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror image_12 '..*' 2024-09-11T06:40:48.521 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:40:48.521 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:40:48.521 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_12 2024-09-11T06:40:48.522 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:40:48.522 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-image_12.status 2024-09-11T06:40:48.522 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-image_12.status 2024-09-11T06:40:48.522 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:40:48.522 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_12.status 2024-09-11T06:40:48.522 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_12 2024-09-11T06:40:48.523 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_12.status 2024-09-11T06:40:48.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 2 2024-09-11T06:40:48.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 2 2024-09-11T06:40:48.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:40:48.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=10206], [object_number=11, tag_tid=1, entry_tid=10203]]], state=connected] 2024-09-11T06:40:48.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0], [object_number=10, tag_tid=1, entry_tid=10206], [object_number=11, tag_tid=1, entry_tid=10203]]], state=connected] 2024-09-11T06:40:48.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_12.status 2024-09-11T06:40:48.573 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=9, tag_tid=2, entry_tid=1' 2024-09-11T06:40:48.573 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=9, tag_tid=2, entry_tid=1' -a 'object_number=9, tag_tid=2, entry_tid=1' = 'object_number=9, tag_tid=2, entry_tid=1' 2024-09-11T06:40:48.573 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:40:48.573 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_snap_present cluster1 mirror image_12 snap 2024-09-11T06:40:48.573 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:40:48.574 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:40:48.574 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_12 2024-09-11T06:40:48.574 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:40:48.574 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:40:48.574 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T06:40:48.574 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:40:49.574 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 info mirror/image_12@snap 2024-09-11T06:40:49.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:rbd image 'image_12': 2024-09-11T06:40:49.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: size 128 MiB in 32 objects 2024-09-11T06:40:49.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: order 22 (4 MiB objects) 2024-09-11T06:40:49.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: snapshot_count: 1 2024-09-11T06:40:49.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: id: 10a938f2c090 2024-09-11T06:40:49.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: block_name_prefix: rbd_data.10a938f2c090 2024-09-11T06:40:49.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: format: 2 2024-09-11T06:40:49.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: features: layering, exclusive-lock, journaling 2024-09-11T06:40:49.645 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: op_features: 2024-09-11T06:40:49.645 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: flags: 2024-09-11T06:40:49.645 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: create_timestamp: Wed Sep 11 06:32:38 2024 2024-09-11T06:40:49.645 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: access_timestamp: Wed Sep 11 06:32:38 2024 2024-09-11T06:40:49.645 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: modify_timestamp: Wed Sep 11 06:32:38 2024 2024-09-11T06:40:49.645 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: protected: False 2024-09-11T06:40:49.645 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: journal: 10a938f2c090 2024-09-11T06:40:49.645 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring state: enabled 2024-09-11T06:40:49.645 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring mode: journal 2024-09-11T06:40:49.645 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring global id: 059b97f8-5c17-4f22-b5a6-37d8dce47372 2024-09-11T06:40:49.645 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring primary: false 2024-09-11T06:40:49.649 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:40:49.649 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ compare_image_snaps mirror image_12 snap 2024-09-11T06:40:49.649 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:40:49.649 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_12 2024-09-11T06:40:49.649 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:40:49.649 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local ret=0 2024-09-11T06:40:49.649 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local rmt_export=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_12.export 2024-09-11T06:40:49.649 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loc_export=/tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_12.export 2024-09-11T06:40:49.649 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_12.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_12.export 2024-09-11T06:40:49.649 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror export image_12@snap /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_12.export 2024-09-11T06:40:49.823 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T06:40:49.830 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 -p mirror export image_12@snap /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_12.export 2024-09-11T06:40:50.255 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T06:40:50.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmp /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_12.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_12.export 2024-09-11T06:40:50.347 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_12.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_12.export 2024-09-11T06:40:50.382 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:40:50.382 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:40:50.382 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_13 2024-09-11T06:40:50.382 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_snap cluster2 mirror image_13 snap 2024-09-11T06:40:50.383 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:40:50.383 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:40:50.383 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_13 2024-09-11T06:40:50.383 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:40:50.383 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror snap create image_13@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T06:40:50.646 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_started cluster1 mirror image_13 2024-09-11T06:40:50.646 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:40:50.646 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:40:50.646 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_13 2024-09-11T06:40:50.646 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_state cluster1 mirror image_13 started 2024-09-11T06:40:50.646 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:40:50.646 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:40:50.646 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_13 2024-09-11T06:40:50.646 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=started 2024-09-11T06:40:50.646 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:40:50.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:40:50.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:40:51.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_replay_state cluster1 mirror image_13 started 2024-09-11T06:40:51.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:40:51.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:40:51.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_13 2024-09-11T06:40:51.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=started 2024-09-11T06:40:51.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local status_result 2024-09-11T06:40:51.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=stopped 2024-09-11T06:40:51.649 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_13 2024-09-11T06:40:51.649 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -i state 2024-09-11T06:40:51.649 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_instance=cluster1 2024-09-11T06:40:51.649 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:40:51.649 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:40:51.649 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance=cluster1 2024-09-11T06:40:51.649 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local loop_instance 2024-09-11T06:40:51.649 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:40:51.649 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sleep 0 2024-09-11T06:40:51.650 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:40:51.651 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ seq 0 1 2024-09-11T06:40:51.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:40:51.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_13 2024-09-11T06:40:51.654 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1:0 2024-09-11T06:40:51.654 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:40:51.654 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:40:51.654 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:40:51.654 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:40:51.654 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=cluster 2024-09-11T06:40:51.654 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:40:51.654 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:40:51.654 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:40:51.654 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:40:51.654 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval cluster=cluster1 2024-09-11T06:40:51.654 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:40:51.654 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:40:51.654 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:40:51.655 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:40:51.655 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local local_cluster=cluster1:0 2024-09-11T06:40:51.655 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster=cluster1 2024-09-11T06:40:51.655 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance 2024-09-11T06:40:51.656 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:40:51.656 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local val=cluster1:0 2024-09-11T06:40:51.656 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T06:40:51.656 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance_var_name=instance 2024-09-11T06:40:51.656 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:40:51.656 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:40:51.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T06:40:51.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval local_cluster=cluster1 2024-09-11T06:40:51.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ local_cluster=cluster1 2024-09-11T06:40:51.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval instance=0 2024-09-11T06:40:51.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ instance=0 2024-09-11T06:40:51.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:40:51.681 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:40:51.681 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:40:51.681 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:40:51.681 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_13 2024-09-11T06:40:51.816 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ return 0 2024-09-11T06:40:51.817 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T06:40:51.818 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ echo ' "state": "Replaying"' 2024-09-11T06:40:51.818 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ grep -i Replaying 2024-09-11T06:40:51.819 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: "state": "Replaying" 2024-09-11T06:40:51.820 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_state=started 2024-09-11T06:40:51.820 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test started = started 2024-09-11T06:40:51.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:40:51.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_13 2024-09-11T06:40:51.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:40:51.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:40:51.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:40:51.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_13 2024-09-11T06:40:51.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' journal = journal ']' 2024-09-11T06:40:51.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_13 2024-09-11T06:40:51.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:40:51.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:40:51.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:40:51.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_13 2024-09-11T06:40:51.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-11T06:40:51.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-11T06:40:51.822 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:40:51.822 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:40:51.822 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:40:52.022 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror image_13 2024-09-11T06:40:52.022 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:40:52.022 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:40:52.022 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_13 2024-09-11T06:40:52.022 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:40:52.022 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n image_13 ']' 2024-09-11T06:40:52.022 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/image_13' 2024-09-11T06:40:52.022 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_13 2024-09-11T06:40:52.022 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:40:52.022 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:40:52.022 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:40:52.022 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:40:52.023 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:40:52.023 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:40:52.023 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:40:52.023 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:40:52.024 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:40:52.026 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:40:52.026 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_13 2024-09-11T06:40:52.026 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:40:52.026 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:40:52.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:40:52.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:40:52.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:40:52.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:40:52.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:40:52.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:40:52.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:40:52.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:40:52.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:40:52.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:40:52.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:40:52.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:40:52.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:40:52.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:40:52.028 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:40:52.028 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:40:52.028 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:40:52.028 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:40:52.028 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:40:52.028 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:40:52.028 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:40:52.028 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:40:52.028 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:40:52.028 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:40:52.028 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:40:52.028 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:40:52.028 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:40:52.029 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:40:52.053 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:40:52.053 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:40:52.054 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:40:52.054 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_13 2024-09-11T06:40:52.188 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:40:52.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror image_13 2024-09-11T06:40:52.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:40:52.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:40:52.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_13 2024-09-11T06:40:52.190 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror image_13 '' 2024-09-11T06:40:52.190 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:40:52.190 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:40:52.190 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_13 2024-09-11T06:40:52.190 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:40:52.190 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-image_13.status 2024-09-11T06:40:52.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-image_13.status 2024-09-11T06:40:52.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:40:52.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_13.status 2024-09-11T06:40:52.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_13 2024-09-11T06:40:52.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_13.status 2024-09-11T06:40:52.238 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 1 2024-09-11T06:40:52.238 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 1 2024-09-11T06:40:52.239 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:40:52.239 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=4279], [object_number=6, tag_tid=1, entry_tid=4278]]], state=connected] 2024-09-11T06:40:52.239 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=4279], [object_number=6, tag_tid=1, entry_tid=4278]]], state=connected] 2024-09-11T06:40:52.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_13.status 2024-09-11T06:40:52.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-09-11T06:40:52.244 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror image_13 2024-09-11T06:40:52.244 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:40:52.245 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:40:52.245 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_13 2024-09-11T06:40:52.245 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror image_13 '..*' 2024-09-11T06:40:52.245 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:40:52.245 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:40:52.245 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_13 2024-09-11T06:40:52.245 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:40:52.245 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-image_13.status 2024-09-11T06:40:52.245 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-image_13.status 2024-09-11T06:40:52.245 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:40:52.246 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_13.status 2024-09-11T06:40:52.246 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_13 2024-09-11T06:40:52.246 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_13.status 2024-09-11T06:40:52.291 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 1 2024-09-11T06:40:52.291 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 1 2024-09-11T06:40:52.291 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:40:52.291 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=4279], [object_number=6, tag_tid=1, entry_tid=4278]]], state=connected] 2024-09-11T06:40:52.291 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=5, tag_tid=2, entry_tid=1], [object_number=4, tag_tid=2, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=4279], [object_number=6, tag_tid=1, entry_tid=4278]]], state=connected] 2024-09-11T06:40:52.294 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_13.status 2024-09-11T06:40:52.296 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-09-11T06:40:52.296 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=2, entry_tid=1' 2024-09-11T06:40:52.296 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=5, tag_tid=2, entry_tid=1' '!=' '' 2024-09-11T06:40:52.296 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:40:52.296 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=5, tag_tid=2, entry_tid=1' = '' 2024-09-11T06:40:52.296 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=5, tag_tid=2, entry_tid=1' 2024-09-11T06:40:52.297 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=5, tag_tid=3, entry_tid=1' 2024-09-11T06:40:52.297 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:40:52.297 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:40:52.297 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=3 2024-09-11T06:40:52.298 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=5, tag_tid=2, entry_tid=1' 2024-09-11T06:40:52.298 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:40:52.298 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:40:52.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=2 2024-09-11T06:40:52.300 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=5, tag_tid=3, entry_tid=1' 2024-09-11T06:40:52.300 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:40:52.300 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:40:52.301 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:40:52.301 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=5, tag_tid=2, entry_tid=1' 2024-09-11T06:40:52.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:40:52.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:40:52.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=1 2024-09-11T06:40:52.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 3 = 2 -a 1 -le 1 2024-09-11T06:40:52.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:40:52.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:40:52.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:40:52.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror image_13 2024-09-11T06:40:52.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:40:52.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:40:52.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_13 2024-09-11T06:40:52.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:40:52.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n image_13 ']' 2024-09-11T06:40:52.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/image_13' 2024-09-11T06:40:52.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_13 2024-09-11T06:40:52.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:40:52.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:40:52.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:40:52.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:40:52.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:40:52.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:40:52.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:40:52.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:40:52.507 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:40:52.511 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:40:52.511 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_13 2024-09-11T06:40:52.511 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:40:52.511 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:40:52.511 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:40:52.511 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:40:52.511 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:40:52.511 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:40:52.511 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:40:52.511 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:40:52.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:40:52.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:40:52.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:40:52.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:40:52.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:40:52.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:40:52.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:40:52.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:40:52.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:40:52.513 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:40:52.513 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:40:52.513 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:40:52.513 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:40:52.513 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:40:52.513 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:40:52.513 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:40:52.513 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:40:52.513 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:40:52.513 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:40:52.513 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:40:52.513 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:40:52.513 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:40:52.537 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:40:52.537 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:40:52.537 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:40:52.537 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_13 2024-09-11T06:40:52.676 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:40:52.677 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror image_13 2024-09-11T06:40:52.677 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:40:52.677 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:40:52.677 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_13 2024-09-11T06:40:52.677 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror image_13 '' 2024-09-11T06:40:52.677 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:40:52.677 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:40:52.678 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_13 2024-09-11T06:40:52.678 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:40:52.678 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-image_13.status 2024-09-11T06:40:52.678 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-image_13.status 2024-09-11T06:40:52.678 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:40:52.679 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_13.status 2024-09-11T06:40:52.680 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_13 2024-09-11T06:40:52.680 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_13.status 2024-09-11T06:40:52.731 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 1 2024-09-11T06:40:52.731 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 1 2024-09-11T06:40:52.732 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:40:52.732 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=4279], [object_number=6, tag_tid=1, entry_tid=4278]]], state=connected] 2024-09-11T06:40:52.732 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=4279], [object_number=6, tag_tid=1, entry_tid=4278]]], state=connected] 2024-09-11T06:40:52.735 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_13.status 2024-09-11T06:40:52.737 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-09-11T06:40:52.737 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror image_13 2024-09-11T06:40:52.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:40:52.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:40:52.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_13 2024-09-11T06:40:52.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror image_13 '..*' 2024-09-11T06:40:52.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:40:52.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:40:52.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_13 2024-09-11T06:40:52.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:40:52.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-image_13.status 2024-09-11T06:40:52.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-image_13.status 2024-09-11T06:40:52.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:40:52.739 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_13.status 2024-09-11T06:40:52.739 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_13 2024-09-11T06:40:52.739 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_13.status 2024-09-11T06:40:52.785 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 1 2024-09-11T06:40:52.785 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 1 2024-09-11T06:40:52.785 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:40:52.785 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=4279], [object_number=6, tag_tid=1, entry_tid=4278]]], state=connected] 2024-09-11T06:40:52.785 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=1, entry_tid=4279], [object_number=6, tag_tid=1, entry_tid=4278]]], state=connected] 2024-09-11T06:40:52.788 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_13.status 2024-09-11T06:40:52.790 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-09-11T06:40:52.791 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=3, entry_tid=1' 2024-09-11T06:40:52.791 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:40:52.791 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_snap_present cluster1 mirror image_13 snap 2024-09-11T06:40:52.791 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:40:52.791 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:40:52.791 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_13 2024-09-11T06:40:52.791 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:40:52.791 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:40:52.791 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T06:40:52.791 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:40:53.791 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 info mirror/image_13@snap 2024-09-11T06:40:53.862 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:rbd image 'image_13': 2024-09-11T06:40:53.862 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: size 128 MiB in 32 objects 2024-09-11T06:40:53.862 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: order 22 (4 MiB objects) 2024-09-11T06:40:53.862 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: snapshot_count: 1 2024-09-11T06:40:53.862 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: id: 10a611cf924f 2024-09-11T06:40:53.862 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: block_name_prefix: rbd_data.10a611cf924f 2024-09-11T06:40:53.862 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: format: 2 2024-09-11T06:40:53.863 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: features: layering, exclusive-lock, journaling 2024-09-11T06:40:53.863 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: op_features: 2024-09-11T06:40:53.863 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: flags: 2024-09-11T06:40:53.863 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: create_timestamp: Wed Sep 11 06:32:51 2024 2024-09-11T06:40:53.863 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: access_timestamp: Wed Sep 11 06:32:51 2024 2024-09-11T06:40:53.863 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: modify_timestamp: Wed Sep 11 06:33:52 2024 2024-09-11T06:40:53.863 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: protected: False 2024-09-11T06:40:53.863 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: journal: 10a611cf924f 2024-09-11T06:40:53.863 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring state: enabled 2024-09-11T06:40:53.863 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring mode: journal 2024-09-11T06:40:53.863 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring global id: 62e4cbae-4ae4-4a2a-9d46-d1a350f4f338 2024-09-11T06:40:53.863 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring primary: false 2024-09-11T06:40:53.867 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:40:53.867 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ compare_image_snaps mirror image_13 snap 2024-09-11T06:40:53.867 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:40:53.867 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_13 2024-09-11T06:40:53.867 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:40:53.867 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local ret=0 2024-09-11T06:40:53.867 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local rmt_export=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_13.export 2024-09-11T06:40:53.867 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loc_export=/tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_13.export 2024-09-11T06:40:53.867 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_13.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_13.export 2024-09-11T06:40:53.868 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror export image_13@snap /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_13.export 2024-09-11T06:40:54.026 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T06:40:54.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 -p mirror export image_13@snap /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_13.export 2024-09-11T06:40:54.351 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T06:40:54.362 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmp /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_13.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_13.export 2024-09-11T06:40:54.442 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_13.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_13.export 2024-09-11T06:40:54.483 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:40:54.484 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:40:54.484 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_14 2024-09-11T06:40:54.484 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_snap cluster2 mirror image_14 snap 2024-09-11T06:40:54.484 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:40:54.484 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:40:54.484 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_14 2024-09-11T06:40:54.484 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:40:54.484 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror snap create image_14@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T06:40:55.665 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_started cluster1 mirror image_14 2024-09-11T06:40:55.665 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:40:55.665 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:40:55.665 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_14 2024-09-11T06:40:55.665 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_state cluster1 mirror image_14 started 2024-09-11T06:40:55.665 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:40:55.665 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:40:55.665 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_14 2024-09-11T06:40:55.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=started 2024-09-11T06:40:55.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:40:55.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:40:55.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:40:56.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_replay_state cluster1 mirror image_14 started 2024-09-11T06:40:56.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:40:56.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:40:56.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_14 2024-09-11T06:40:56.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=started 2024-09-11T06:40:56.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local status_result 2024-09-11T06:40:56.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=stopped 2024-09-11T06:40:56.668 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -i state 2024-09-11T06:40:56.668 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_14 2024-09-11T06:40:56.668 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_instance=cluster1 2024-09-11T06:40:56.668 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:40:56.668 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:40:56.668 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance=cluster1 2024-09-11T06:40:56.668 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local loop_instance 2024-09-11T06:40:56.668 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:40:56.669 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sleep 0 2024-09-11T06:40:56.670 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:40:56.670 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ seq 0 1 2024-09-11T06:40:56.672 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:40:56.672 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_14 2024-09-11T06:40:56.672 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1:0 2024-09-11T06:40:56.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:40:56.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:40:56.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:40:56.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:40:56.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=cluster 2024-09-11T06:40:56.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:40:56.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:40:56.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:40:56.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:40:56.674 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval cluster=cluster1 2024-09-11T06:40:56.674 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:40:56.674 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:40:56.674 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:40:56.674 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:40:56.674 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local local_cluster=cluster1:0 2024-09-11T06:40:56.674 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster=cluster1 2024-09-11T06:40:56.674 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance 2024-09-11T06:40:56.674 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:40:56.675 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local val=cluster1:0 2024-09-11T06:40:56.675 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T06:40:56.675 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance_var_name=instance 2024-09-11T06:40:56.675 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:40:56.676 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:40:56.676 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T06:40:56.676 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval local_cluster=cluster1 2024-09-11T06:40:56.676 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ local_cluster=cluster1 2024-09-11T06:40:56.676 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval instance=0 2024-09-11T06:40:56.676 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ instance=0 2024-09-11T06:40:56.676 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:40:56.696 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:40:56.696 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:40:56.696 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:40:56.696 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_14 2024-09-11T06:40:56.824 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:Invalid command: unused arguments: ['mirror/image_14'] 2024-09-11T06:40:56.824 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T06:40:56.824 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:admin_socket: invalid command 2024-09-11T06:40:56.826 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:40:56.826 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_14 2024-09-11T06:40:56.827 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1:1 2024-09-11T06:40:56.827 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:40:56.827 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:40:56.827 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-11T06:40:56.827 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:1 2024-09-11T06:40:56.827 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=cluster 2024-09-11T06:40:56.827 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:40:56.827 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:40:56.827 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=1 2024-09-11T06:40:56.827 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T06:40:56.828 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval cluster=cluster1 2024-09-11T06:40:56.828 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:40:56.828 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=1 2024-09-11T06:40:56.828 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=1 2024-09-11T06:40:56.828 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-11T06:40:56.828 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local local_cluster=cluster1:1 2024-09-11T06:40:56.828 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster=cluster1 2024-09-11T06:40:56.828 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance 2024-09-11T06:40:56.828 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T06:40:56.828 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local val=cluster1:1 2024-09-11T06:40:56.828 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T06:40:56.828 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance_var_name=instance 2024-09-11T06:40:56.828 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:40:56.829 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=1 2024-09-11T06:40:56.829 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-11T06:40:56.829 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval local_cluster=cluster1 2024-09-11T06:40:56.829 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ local_cluster=cluster1 2024-09-11T06:40:56.829 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval instance=1 2024-09-11T06:40:56.829 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ instance=1 2024-09-11T06:40:56.829 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T06:40:56.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:40:56.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:40:56.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:40:56.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_14 2024-09-11T06:40:56.979 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ return 0 2024-09-11T06:40:56.980 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T06:40:56.981 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ echo ' "state": "Replaying"' 2024-09-11T06:40:56.981 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ grep -i Replaying 2024-09-11T06:40:56.982 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: "state": "Replaying" 2024-09-11T06:40:56.983 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_state=started 2024-09-11T06:40:56.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test started = started 2024-09-11T06:40:56.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:40:56.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_14 2024-09-11T06:40:56.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:40:56.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:40:56.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:40:56.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_14 2024-09-11T06:40:56.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' journal = journal ']' 2024-09-11T06:40:56.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_14 2024-09-11T06:40:56.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:40:56.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:40:56.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:40:56.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_14 2024-09-11T06:40:56.985 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-11T06:40:56.985 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-11T06:40:56.985 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:40:56.985 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:40:56.985 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:40:57.185 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror image_14 2024-09-11T06:40:57.185 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:40:57.185 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:40:57.185 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_14 2024-09-11T06:40:57.186 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:40:57.186 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n image_14 ']' 2024-09-11T06:40:57.186 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/image_14' 2024-09-11T06:40:57.186 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_14 2024-09-11T06:40:57.186 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:40:57.186 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:40:57.186 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:40:57.186 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:40:57.186 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:40:57.186 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:40:57.186 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:40:57.187 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:40:57.187 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:40:57.190 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:40:57.190 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_14 2024-09-11T06:40:57.190 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:40:57.190 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:40:57.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:40:57.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:40:57.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:40:57.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:40:57.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:40:57.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:40:57.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:40:57.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:40:57.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:40:57.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:40:57.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:40:57.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:40:57.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:40:57.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:40:57.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:40:57.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:40:57.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:40:57.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:40:57.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:40:57.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:40:57.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:40:57.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:40:57.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:40:57.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:40:57.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:40:57.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:40:57.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:40:57.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:40:57.215 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:40:57.216 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:40:57.216 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:40:57.216 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_14 2024-09-11T06:40:57.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:Invalid command: unused arguments: ['mirror/image_14'] 2024-09-11T06:40:57.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd mirror flush : flush rbd mirror 2024-09-11T06:40:57.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:admin_socket: invalid command 2024-09-11T06:40:57.338 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:40:57.338 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_14 2024-09-11T06:40:57.338 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:1 2024-09-11T06:40:57.338 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:40:57.339 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:40:57.339 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-11T06:40:57.339 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:1 2024-09-11T06:40:57.339 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:40:57.339 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:40:57.339 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:40:57.339 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=1 2024-09-11T06:40:57.339 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-11T06:40:57.339 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:40:57.339 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:40:57.339 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=1 2024-09-11T06:40:57.339 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=1 2024-09-11T06:40:57.341 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-11T06:40:57.341 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:1 2024-09-11T06:40:57.341 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:40:57.341 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:40:57.341 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T06:40:57.341 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:1 2024-09-11T06:40:57.341 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:40:57.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:40:57.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:40:57.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=1 2024-09-11T06:40:57.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T06:40:57.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:40:57.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:40:57.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=1 2024-09-11T06:40:57.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=1 2024-09-11T06:40:57.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T06:40:57.359 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:40:57.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:40:57.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:40:57.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_14 2024-09-11T06:40:57.498 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:40:57.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror image_14 2024-09-11T06:40:57.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:40:57.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:40:57.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_14 2024-09-11T06:40:57.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror image_14 '' 2024-09-11T06:40:57.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:40:57.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:40:57.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_14 2024-09-11T06:40:57.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:40:57.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-image_14.status 2024-09-11T06:40:57.501 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-image_14.status 2024-09-11T06:40:57.501 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:40:57.503 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_14.status 2024-09-11T06:40:57.503 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_14 2024-09-11T06:40:57.503 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_14.status 2024-09-11T06:40:57.550 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 0 2024-09-11T06:40:57.551 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 0 2024-09-11T06:40:57.551 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:40:57.551 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=2, tag_tid=1, entry_tid=2594], [object_number=3, tag_tid=1, entry_tid=2591]]], state=connected] 2024-09-11T06:40:57.551 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=2, tag_tid=1, entry_tid=2594], [object_number=3, tag_tid=1, entry_tid=2591]]], state=connected] 2024-09-11T06:40:57.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_14.status 2024-09-11T06:40:57.555 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=1, tag_tid=3, entry_tid=1' 2024-09-11T06:40:57.556 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror image_14 2024-09-11T06:40:57.557 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:40:57.557 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:40:57.557 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_14 2024-09-11T06:40:57.557 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror image_14 '..*' 2024-09-11T06:40:57.557 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:40:57.557 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:40:57.557 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_14 2024-09-11T06:40:57.557 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:40:57.557 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-image_14.status 2024-09-11T06:40:57.557 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-image_14.status 2024-09-11T06:40:57.557 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:40:57.558 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_14.status 2024-09-11T06:40:57.558 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_14 2024-09-11T06:40:57.558 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_14.status 2024-09-11T06:40:57.604 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 0 2024-09-11T06:40:57.604 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 0 2024-09-11T06:40:57.604 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:40:57.604 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=2, tag_tid=1, entry_tid=2594], [object_number=3, tag_tid=1, entry_tid=2591]]], state=connected] 2024-09-11T06:40:57.605 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=2, tag_tid=1, entry_tid=2594], [object_number=3, tag_tid=1, entry_tid=2591]]], state=connected] 2024-09-11T06:40:57.608 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_14.status 2024-09-11T06:40:57.610 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=1, tag_tid=3, entry_tid=1' 2024-09-11T06:40:57.610 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=1, tag_tid=3, entry_tid=1' -a 'object_number=1, tag_tid=3, entry_tid=1' = 'object_number=1, tag_tid=3, entry_tid=1' 2024-09-11T06:40:57.610 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:40:57.610 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_snap_present cluster1 mirror image_14 snap 2024-09-11T06:40:57.610 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:40:57.610 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:40:57.610 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_14 2024-09-11T06:40:57.610 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:40:57.610 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:40:57.610 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T06:40:57.610 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:40:58.611 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 info mirror/image_14@snap 2024-09-11T06:40:58.678 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:rbd image 'image_14': 2024-09-11T06:40:58.678 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: size 128 MiB in 32 objects 2024-09-11T06:40:58.678 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: order 22 (4 MiB objects) 2024-09-11T06:40:58.678 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: snapshot_count: 1 2024-09-11T06:40:58.678 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: id: 10a91d3b5d1c 2024-09-11T06:40:58.678 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: block_name_prefix: rbd_data.10a91d3b5d1c 2024-09-11T06:40:58.678 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: format: 2 2024-09-11T06:40:58.678 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: features: layering, exclusive-lock, journaling 2024-09-11T06:40:58.678 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: op_features: 2024-09-11T06:40:58.678 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: flags: 2024-09-11T06:40:58.679 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: create_timestamp: Wed Sep 11 06:32:56 2024 2024-09-11T06:40:58.679 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: access_timestamp: Wed Sep 11 06:32:56 2024 2024-09-11T06:40:58.679 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: modify_timestamp: Wed Sep 11 06:32:56 2024 2024-09-11T06:40:58.679 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: protected: False 2024-09-11T06:40:58.679 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: journal: 10a91d3b5d1c 2024-09-11T06:40:58.679 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring state: enabled 2024-09-11T06:40:58.679 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring mode: journal 2024-09-11T06:40:58.679 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring global id: 8e9a0d2a-6c15-4fe9-9cff-bc704b297f31 2024-09-11T06:40:58.679 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring primary: false 2024-09-11T06:40:58.682 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:40:58.682 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ compare_image_snaps mirror image_14 snap 2024-09-11T06:40:58.682 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:40:58.682 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_14 2024-09-11T06:40:58.682 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:40:58.683 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local ret=0 2024-09-11T06:40:58.683 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local rmt_export=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_14.export 2024-09-11T06:40:58.683 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loc_export=/tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_14.export 2024-09-11T06:40:58.683 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_14.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_14.export 2024-09-11T06:40:58.683 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror export image_14@snap /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_14.export 2024-09-11T06:40:58.842 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T06:40:58.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 -p mirror export image_14@snap /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_14.export 2024-09-11T06:40:59.166 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T06:40:59.179 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmp /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_14.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_14.export 2024-09-11T06:40:59.258 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_14.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_14.export 2024-09-11T06:40:59.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:40:59.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:40:59.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_15 2024-09-11T06:40:59.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_snap cluster2 mirror image_15 snap 2024-09-11T06:40:59.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:40:59.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:40:59.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_15 2024-09-11T06:40:59.300 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:40:59.300 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror snap create image_15@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T06:40:59.739 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_started cluster1 mirror image_15 2024-09-11T06:40:59.739 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:40:59.739 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:40:59.740 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_15 2024-09-11T06:40:59.740 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_state cluster1 mirror image_15 started 2024-09-11T06:40:59.740 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:40:59.740 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:40:59.740 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_15 2024-09-11T06:40:59.740 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=started 2024-09-11T06:40:59.740 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:40:59.740 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:40:59.740 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:41:00.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_replay_state cluster1 mirror image_15 started 2024-09-11T06:41:00.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:41:00.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:00.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_15 2024-09-11T06:41:00.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=started 2024-09-11T06:41:00.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local status_result 2024-09-11T06:41:00.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=stopped 2024-09-11T06:41:00.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_15 2024-09-11T06:41:00.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -i state 2024-09-11T06:41:00.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_instance=cluster1 2024-09-11T06:41:00.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:41:00.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:41:00.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance=cluster1 2024-09-11T06:41:00.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local loop_instance 2024-09-11T06:41:00.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:41:00.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sleep 0 2024-09-11T06:41:00.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:41:00.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ seq 0 1 2024-09-11T06:41:00.746 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:41:00.746 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_15 2024-09-11T06:41:00.747 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1:0 2024-09-11T06:41:00.747 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:41:00.747 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:41:00.747 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:41:00.747 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:41:00.747 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=cluster 2024-09-11T06:41:00.747 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:41:00.747 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:41:00.747 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:41:00.747 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:41:00.747 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval cluster=cluster1 2024-09-11T06:41:00.747 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:41:00.747 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:41:00.747 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:41:00.747 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:41:00.749 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local local_cluster=cluster1:0 2024-09-11T06:41:00.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster=cluster1 2024-09-11T06:41:00.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance 2024-09-11T06:41:00.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:41:00.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local val=cluster1:0 2024-09-11T06:41:00.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T06:41:00.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance_var_name=instance 2024-09-11T06:41:00.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:41:00.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:41:00.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T06:41:00.753 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval local_cluster=cluster1 2024-09-11T06:41:00.753 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ local_cluster=cluster1 2024-09-11T06:41:00.753 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval instance=0 2024-09-11T06:41:00.753 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ instance=0 2024-09-11T06:41:00.753 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:41:00.768 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:41:00.768 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:41:00.768 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:41:00.768 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_15 2024-09-11T06:41:00.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ return 0 2024-09-11T06:41:00.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T06:41:00.915 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ echo ' "state": "Replaying"' 2024-09-11T06:41:00.915 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ grep -i Replaying 2024-09-11T06:41:00.916 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: "state": "Replaying" 2024-09-11T06:41:00.918 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_state=started 2024-09-11T06:41:00.918 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test started = started 2024-09-11T06:41:00.918 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:41:00.918 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_15 2024-09-11T06:41:00.918 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:41:00.918 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:41:00.918 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:00.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_15 2024-09-11T06:41:00.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' journal = journal ']' 2024-09-11T06:41:00.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_15 2024-09-11T06:41:00.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:41:00.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:41:00.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:00.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_15 2024-09-11T06:41:00.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-11T06:41:00.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-11T06:41:00.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:41:00.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:41:00.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:41:01.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror image_15 2024-09-11T06:41:01.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:41:01.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:01.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_15 2024-09-11T06:41:01.120 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:41:01.120 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n image_15 ']' 2024-09-11T06:41:01.120 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/image_15' 2024-09-11T06:41:01.120 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_15 2024-09-11T06:41:01.120 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:41:01.120 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:41:01.120 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:41:01.120 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:41:01.120 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:41:01.120 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:41:01.120 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:41:01.121 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:41:01.121 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:41:01.124 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:41:01.124 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_15 2024-09-11T06:41:01.124 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:41:01.124 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:41:01.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:41:01.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:41:01.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:41:01.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:41:01.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:41:01.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:41:01.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:41:01.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:41:01.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:41:01.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:41:01.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:41:01.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:41:01.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:41:01.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:41:01.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:41:01.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:41:01.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:41:01.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:41:01.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:41:01.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:41:01.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:41:01.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:41:01.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:41:01.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:41:01.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:41:01.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:41:01.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:41:01.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:41:01.152 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:41:01.152 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:41:01.152 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:41:01.152 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_15 2024-09-11T06:41:01.293 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:41:01.294 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror image_15 2024-09-11T06:41:01.294 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:41:01.295 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:41:01.295 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_15 2024-09-11T06:41:01.295 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror image_15 '' 2024-09-11T06:41:01.295 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:41:01.295 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:41:01.295 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_15 2024-09-11T06:41:01.295 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:41:01.295 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-image_15.status 2024-09-11T06:41:01.295 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-image_15.status 2024-09-11T06:41:01.296 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:41:01.297 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_15.status 2024-09-11T06:41:01.297 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_15 2024-09-11T06:41:01.297 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_15.status 2024-09-11T06:41:01.353 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 3 2024-09-11T06:41:01.353 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 3 2024-09-11T06:41:01.353 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:41:01.353 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=3], [object_number=14, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:41:01.353 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=3], [object_number=14, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:41:01.356 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_15.status 2024-09-11T06:41:01.358 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=13, tag_tid=3, entry_tid=1' 2024-09-11T06:41:01.359 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror image_15 2024-09-11T06:41:01.359 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:41:01.359 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:41:01.359 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_15 2024-09-11T06:41:01.359 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror image_15 '..*' 2024-09-11T06:41:01.359 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:41:01.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:41:01.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_15 2024-09-11T06:41:01.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:41:01.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-image_15.status 2024-09-11T06:41:01.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-image_15.status 2024-09-11T06:41:01.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:41:01.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_15.status 2024-09-11T06:41:01.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_15 2024-09-11T06:41:01.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_15.status 2024-09-11T06:41:01.407 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 3 2024-09-11T06:41:01.407 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 3 2024-09-11T06:41:01.407 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:41:01.407 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=3], [object_number=14, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:41:01.407 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=3], [object_number=14, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:41:01.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_15.status 2024-09-11T06:41:01.412 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=13, tag_tid=3, entry_tid=1' 2024-09-11T06:41:01.412 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=13, tag_tid=3, entry_tid=1' -a 'object_number=13, tag_tid=3, entry_tid=1' = 'object_number=13, tag_tid=3, entry_tid=1' 2024-09-11T06:41:01.412 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:41:01.412 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_snap_present cluster1 mirror image_15 snap 2024-09-11T06:41:01.412 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:41:01.413 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:01.413 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_15 2024-09-11T06:41:01.413 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:41:01.413 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:41:01.413 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T06:41:01.413 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:41:02.413 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 info mirror/image_15@snap 2024-09-11T06:41:02.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:rbd image 'image_15': 2024-09-11T06:41:02.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: size 128 MiB in 32 objects 2024-09-11T06:41:02.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: order 22 (4 MiB objects) 2024-09-11T06:41:02.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: snapshot_count: 1 2024-09-11T06:41:02.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: id: 10a6d039f110 2024-09-11T06:41:02.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: block_name_prefix: rbd_data.10a6d039f110 2024-09-11T06:41:02.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: format: 2 2024-09-11T06:41:02.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: features: layering, exclusive-lock, journaling 2024-09-11T06:41:02.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: op_features: 2024-09-11T06:41:02.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: flags: 2024-09-11T06:41:02.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: create_timestamp: Wed Sep 11 06:33:01 2024 2024-09-11T06:41:02.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: access_timestamp: Wed Sep 11 06:33:01 2024 2024-09-11T06:41:02.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: modify_timestamp: Wed Sep 11 06:34:25 2024 2024-09-11T06:41:02.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: protected: False 2024-09-11T06:41:02.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: journal: 10a6d039f110 2024-09-11T06:41:02.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring state: enabled 2024-09-11T06:41:02.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring mode: journal 2024-09-11T06:41:02.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring global id: a5cc53c2-8326-4bef-9ea1-43b4cf9e2f3d 2024-09-11T06:41:02.483 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring primary: false 2024-09-11T06:41:02.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:41:02.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ compare_image_snaps mirror image_15 snap 2024-09-11T06:41:02.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:02.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_15 2024-09-11T06:41:02.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:41:02.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local ret=0 2024-09-11T06:41:02.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local rmt_export=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_15.export 2024-09-11T06:41:02.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loc_export=/tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_15.export 2024-09-11T06:41:02.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_15.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_15.export 2024-09-11T06:41:02.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror export image_15@snap /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_15.export 2024-09-11T06:41:02.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T06:41:02.696 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 -p mirror export image_15@snap /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_15.export 2024-09-11T06:41:03.130 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T06:41:03.146 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmp /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_15.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_15.export 2024-09-11T06:41:03.225 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_15.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_15.export 2024-09-11T06:41:03.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:41:03.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:41:03.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_16 2024-09-11T06:41:03.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_snap cluster2 mirror image_16 snap 2024-09-11T06:41:03.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:41:03.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:03.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_16 2024-09-11T06:41:03.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:41:03.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror snap create image_16@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T06:41:03.684 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_started cluster1 mirror image_16 2024-09-11T06:41:03.684 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:41:03.684 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:03.685 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_16 2024-09-11T06:41:03.685 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_state cluster1 mirror image_16 started 2024-09-11T06:41:03.685 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:41:03.685 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:03.685 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_16 2024-09-11T06:41:03.685 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=started 2024-09-11T06:41:03.685 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:41:03.685 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:41:03.685 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:41:04.686 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_replay_state cluster1 mirror image_16 started 2024-09-11T06:41:04.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:41:04.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:04.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_16 2024-09-11T06:41:04.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=started 2024-09-11T06:41:04.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local status_result 2024-09-11T06:41:04.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=stopped 2024-09-11T06:41:04.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_16 2024-09-11T06:41:04.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -i state 2024-09-11T06:41:04.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_instance=cluster1 2024-09-11T06:41:04.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:41:04.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:41:04.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance=cluster1 2024-09-11T06:41:04.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local loop_instance 2024-09-11T06:41:04.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:41:04.689 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sleep 0 2024-09-11T06:41:04.689 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:41:04.689 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ seq 0 1 2024-09-11T06:41:04.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:41:04.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_16 2024-09-11T06:41:04.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1:0 2024-09-11T06:41:04.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:41:04.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:41:04.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:41:04.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:41:04.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=cluster 2024-09-11T06:41:04.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:41:04.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:41:04.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:41:04.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:41:04.693 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval cluster=cluster1 2024-09-11T06:41:04.693 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:41:04.693 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:41:04.693 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:41:04.693 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:41:04.693 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local local_cluster=cluster1:0 2024-09-11T06:41:04.693 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster=cluster1 2024-09-11T06:41:04.693 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance 2024-09-11T06:41:04.693 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:41:04.694 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local val=cluster1:0 2024-09-11T06:41:04.694 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T06:41:04.694 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance_var_name=instance 2024-09-11T06:41:04.695 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:41:04.695 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:41:04.695 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T06:41:04.695 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval local_cluster=cluster1 2024-09-11T06:41:04.695 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ local_cluster=cluster1 2024-09-11T06:41:04.695 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval instance=0 2024-09-11T06:41:04.695 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ instance=0 2024-09-11T06:41:04.695 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:41:04.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:41:04.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:41:04.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:41:04.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_16 2024-09-11T06:41:04.839 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:Invalid command: unused arguments: ['mirror/image_16'] 2024-09-11T06:41:04.839 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T06:41:04.839 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:admin_socket: invalid command 2024-09-11T06:41:04.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:41:04.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_16 2024-09-11T06:41:04.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1:1 2024-09-11T06:41:04.848 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:41:04.848 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:41:04.848 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-11T06:41:04.848 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:1 2024-09-11T06:41:04.848 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=cluster 2024-09-11T06:41:04.848 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:41:04.848 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:41:04.848 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=1 2024-09-11T06:41:04.848 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T06:41:04.848 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval cluster=cluster1 2024-09-11T06:41:04.848 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:41:04.848 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=1 2024-09-11T06:41:04.848 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=1 2024-09-11T06:41:04.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-11T06:41:04.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local local_cluster=cluster1:1 2024-09-11T06:41:04.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster=cluster1 2024-09-11T06:41:04.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance 2024-09-11T06:41:04.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T06:41:04.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local val=cluster1:1 2024-09-11T06:41:04.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T06:41:04.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance_var_name=instance 2024-09-11T06:41:04.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:41:04.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=1 2024-09-11T06:41:04.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-11T06:41:04.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval local_cluster=cluster1 2024-09-11T06:41:04.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ local_cluster=cluster1 2024-09-11T06:41:04.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval instance=1 2024-09-11T06:41:04.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ instance=1 2024-09-11T06:41:04.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T06:41:04.868 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:41:04.868 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:41:04.868 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:41:04.868 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_16 2024-09-11T06:41:05.013 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ return 0 2024-09-11T06:41:05.013 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T06:41:05.014 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ echo ' "state": "Replaying"' 2024-09-11T06:41:05.014 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ grep -i Replaying 2024-09-11T06:41:05.016 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: "state": "Replaying" 2024-09-11T06:41:05.017 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_state=started 2024-09-11T06:41:05.017 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test started = started 2024-09-11T06:41:05.017 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:41:05.017 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_16 2024-09-11T06:41:05.017 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:41:05.017 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:41:05.017 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:05.017 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_16 2024-09-11T06:41:05.017 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' journal = journal ']' 2024-09-11T06:41:05.018 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_16 2024-09-11T06:41:05.018 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:41:05.018 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:41:05.018 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:05.018 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_16 2024-09-11T06:41:05.018 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-11T06:41:05.018 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-11T06:41:05.018 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:41:05.018 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:41:05.018 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:41:05.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror image_16 2024-09-11T06:41:05.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:41:05.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:05.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_16 2024-09-11T06:41:05.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:41:05.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n image_16 ']' 2024-09-11T06:41:05.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/image_16' 2024-09-11T06:41:05.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_16 2024-09-11T06:41:05.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:41:05.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:41:05.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:41:05.221 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:41:05.221 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:41:05.221 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:41:05.221 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:41:05.226 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:41:05.226 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:41:05.226 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:41:05.226 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_16 2024-09-11T06:41:05.226 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:41:05.226 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:41:05.226 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:41:05.226 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:41:05.226 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:41:05.227 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:41:05.227 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:41:05.227 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:41:05.227 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:41:05.227 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:41:05.227 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:41:05.227 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:41:05.227 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:41:05.228 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:41:05.228 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:41:05.228 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:41:05.228 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:41:05.228 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:41:05.228 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:41:05.228 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:41:05.228 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:41:05.229 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:41:05.229 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:41:05.229 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:41:05.229 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:41:05.229 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:41:05.229 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:41:05.229 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:41:05.229 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:41:05.229 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:41:05.250 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:41:05.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:41:05.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:41:05.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_16 2024-09-11T06:41:05.372 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:Invalid command: unused arguments: ['mirror/image_16'] 2024-09-11T06:41:05.372 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd mirror flush : flush rbd mirror 2024-09-11T06:41:05.372 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:admin_socket: invalid command 2024-09-11T06:41:05.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:41:05.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_16 2024-09-11T06:41:05.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:1 2024-09-11T06:41:05.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:41:05.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:41:05.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-11T06:41:05.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:1 2024-09-11T06:41:05.378 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:41:05.378 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:41:05.378 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:41:05.378 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=1 2024-09-11T06:41:05.378 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-11T06:41:05.378 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:41:05.378 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:41:05.378 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=1 2024-09-11T06:41:05.378 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=1 2024-09-11T06:41:05.378 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-11T06:41:05.378 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:1 2024-09-11T06:41:05.378 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:41:05.378 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:41:05.379 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T06:41:05.379 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:1 2024-09-11T06:41:05.379 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:41:05.379 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:41:05.379 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:41:05.379 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=1 2024-09-11T06:41:05.379 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T06:41:05.379 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:41:05.379 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:41:05.379 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=1 2024-09-11T06:41:05.379 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=1 2024-09-11T06:41:05.379 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T06:41:05.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:41:05.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:41:05.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:41:05.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_16 2024-09-11T06:41:05.533 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:41:05.533 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror image_16 2024-09-11T06:41:05.534 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:41:05.534 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:41:05.534 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_16 2024-09-11T06:41:05.534 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror image_16 '' 2024-09-11T06:41:05.534 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:41:05.534 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:41:05.534 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_16 2024-09-11T06:41:05.534 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:41:05.535 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-image_16.status 2024-09-11T06:41:05.535 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-image_16.status 2024-09-11T06:41:05.535 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:41:05.536 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_16.status 2024-09-11T06:41:05.537 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_16 2024-09-11T06:41:05.537 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_16.status 2024-09-11T06:41:05.587 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 1 2024-09-11T06:41:05.588 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 1 2024-09-11T06:41:05.588 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:41:05.588 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:41:05.588 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:41:05.591 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_16.status 2024-09-11T06:41:05.592 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-09-11T06:41:05.593 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror image_16 2024-09-11T06:41:05.593 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:41:05.594 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:41:05.594 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_16 2024-09-11T06:41:05.594 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror image_16 '..*' 2024-09-11T06:41:05.594 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:41:05.594 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:41:05.594 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_16 2024-09-11T06:41:05.594 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:41:05.594 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-image_16.status 2024-09-11T06:41:05.594 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-image_16.status 2024-09-11T06:41:05.594 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:41:05.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_16.status 2024-09-11T06:41:05.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_16 2024-09-11T06:41:05.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_16.status 2024-09-11T06:41:05.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 1 2024-09-11T06:41:05.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 1 2024-09-11T06:41:05.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:41:05.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:41:05.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:41:05.675 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_16.status 2024-09-11T06:41:05.678 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-09-11T06:41:05.678 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=3, entry_tid=1' 2024-09-11T06:41:05.678 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:41:05.678 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_snap_present cluster1 mirror image_16 snap 2024-09-11T06:41:05.678 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:41:05.678 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:05.678 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_16 2024-09-11T06:41:05.678 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:41:05.678 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:41:05.678 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T06:41:05.679 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:41:06.679 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 info mirror/image_16@snap 2024-09-11T06:41:06.747 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:rbd image 'image_16': 2024-09-11T06:41:06.747 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: size 128 MiB in 32 objects 2024-09-11T06:41:06.747 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: order 22 (4 MiB objects) 2024-09-11T06:41:06.747 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: snapshot_count: 1 2024-09-11T06:41:06.747 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: id: 10a9dfda92a2 2024-09-11T06:41:06.747 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: block_name_prefix: rbd_data.10a9dfda92a2 2024-09-11T06:41:06.747 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: format: 2 2024-09-11T06:41:06.747 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: features: layering, exclusive-lock, journaling 2024-09-11T06:41:06.747 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: op_features: 2024-09-11T06:41:06.747 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: flags: 2024-09-11T06:41:06.747 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: create_timestamp: Wed Sep 11 06:33:14 2024 2024-09-11T06:41:06.748 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: access_timestamp: Wed Sep 11 06:33:14 2024 2024-09-11T06:41:06.748 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: modify_timestamp: Wed Sep 11 06:34:14 2024 2024-09-11T06:41:06.748 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: protected: False 2024-09-11T06:41:06.748 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: journal: 10a9dfda92a2 2024-09-11T06:41:06.748 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring state: enabled 2024-09-11T06:41:06.748 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring mode: journal 2024-09-11T06:41:06.748 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring global id: 2295004c-e7b7-4c66-aa2b-c834dbfe76e8 2024-09-11T06:41:06.748 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring primary: false 2024-09-11T06:41:06.751 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:41:06.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ compare_image_snaps mirror image_16 snap 2024-09-11T06:41:06.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:06.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_16 2024-09-11T06:41:06.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:41:06.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local ret=0 2024-09-11T06:41:06.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local rmt_export=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_16.export 2024-09-11T06:41:06.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loc_export=/tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_16.export 2024-09-11T06:41:06.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_16.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_16.export 2024-09-11T06:41:06.753 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror export image_16@snap /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_16.export 2024-09-11T06:41:06.918 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T06:41:06.925 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 -p mirror export image_16@snap /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_16.export 2024-09-11T06:41:07.230 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T06:41:07.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmp /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_16.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_16.export 2024-09-11T06:41:07.319 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_16.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_16.export 2024-09-11T06:41:07.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:41:07.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:41:07.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_17 2024-09-11T06:41:07.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_snap cluster2 mirror image_17 snap 2024-09-11T06:41:07.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:41:07.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:07.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_17 2024-09-11T06:41:07.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:41:07.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror snap create image_17@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T06:41:07.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_started cluster1 mirror image_17 2024-09-11T06:41:07.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:41:07.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:07.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_17 2024-09-11T06:41:07.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_state cluster1 mirror image_17 started 2024-09-11T06:41:07.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:41:07.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:07.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_17 2024-09-11T06:41:07.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=started 2024-09-11T06:41:07.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:41:07.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:41:07.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:41:08.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_replay_state cluster1 mirror image_17 started 2024-09-11T06:41:08.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:41:08.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:08.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_17 2024-09-11T06:41:08.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=started 2024-09-11T06:41:08.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local status_result 2024-09-11T06:41:08.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=stopped 2024-09-11T06:41:08.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_17 2024-09-11T06:41:08.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_instance=cluster1 2024-09-11T06:41:08.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:41:08.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:41:08.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance=cluster1 2024-09-11T06:41:08.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local loop_instance 2024-09-11T06:41:08.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:41:08.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sleep 0 2024-09-11T06:41:08.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -i state 2024-09-11T06:41:08.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:41:08.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ seq 0 1 2024-09-11T06:41:08.721 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:41:08.721 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_17 2024-09-11T06:41:08.722 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1:0 2024-09-11T06:41:08.722 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:41:08.722 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:41:08.722 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:41:08.722 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:41:08.722 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=cluster 2024-09-11T06:41:08.722 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:41:08.722 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:41:08.722 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:41:08.722 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:41:08.722 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval cluster=cluster1 2024-09-11T06:41:08.722 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:41:08.722 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:41:08.722 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:41:08.723 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:41:08.723 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local local_cluster=cluster1:0 2024-09-11T06:41:08.723 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster=cluster1 2024-09-11T06:41:08.723 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance 2024-09-11T06:41:08.723 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:41:08.723 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local val=cluster1:0 2024-09-11T06:41:08.723 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T06:41:08.723 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance_var_name=instance 2024-09-11T06:41:08.723 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:41:08.723 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:41:08.723 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T06:41:08.723 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval local_cluster=cluster1 2024-09-11T06:41:08.723 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ local_cluster=cluster1 2024-09-11T06:41:08.724 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval instance=0 2024-09-11T06:41:08.724 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ instance=0 2024-09-11T06:41:08.724 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:41:08.746 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:41:08.746 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:41:08.746 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:41:08.746 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_17 2024-09-11T06:41:08.881 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ return 0 2024-09-11T06:41:08.882 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T06:41:08.882 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ echo ' "state": "Replaying"' 2024-09-11T06:41:08.882 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ grep -i Replaying 2024-09-11T06:41:08.884 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: "state": "Replaying" 2024-09-11T06:41:08.885 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_state=started 2024-09-11T06:41:08.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test started = started 2024-09-11T06:41:08.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:41:08.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_17 2024-09-11T06:41:08.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:41:08.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:41:08.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:08.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_17 2024-09-11T06:41:08.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' journal = journal ']' 2024-09-11T06:41:08.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_17 2024-09-11T06:41:08.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:41:08.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:41:08.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:08.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_17 2024-09-11T06:41:08.887 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-11T06:41:08.887 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-11T06:41:08.887 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:41:08.887 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:41:08.887 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:41:09.087 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror image_17 2024-09-11T06:41:09.087 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:41:09.087 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:09.087 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_17 2024-09-11T06:41:09.087 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:41:09.087 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n image_17 ']' 2024-09-11T06:41:09.087 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/image_17' 2024-09-11T06:41:09.088 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_17 2024-09-11T06:41:09.088 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:41:09.088 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:41:09.088 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:41:09.088 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:41:09.088 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:41:09.088 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:41:09.088 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:41:09.089 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:41:09.089 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:41:09.092 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:41:09.092 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_17 2024-09-11T06:41:09.092 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:41:09.092 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:41:09.092 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:41:09.093 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:41:09.093 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:41:09.093 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:41:09.093 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:41:09.093 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:41:09.093 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:41:09.093 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:41:09.093 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:41:09.093 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:41:09.093 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:41:09.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:41:09.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:41:09.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:41:09.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:41:09.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:41:09.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:41:09.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:41:09.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:41:09.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:41:09.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:41:09.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:41:09.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:41:09.095 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:41:09.095 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:41:09.095 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:41:09.095 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:41:09.095 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:41:09.118 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:41:09.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:41:09.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:41:09.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_17 2024-09-11T06:41:09.259 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:41:09.261 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror image_17 2024-09-11T06:41:09.261 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:41:09.261 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:41:09.261 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_17 2024-09-11T06:41:09.261 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror image_17 '' 2024-09-11T06:41:09.261 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:41:09.261 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:41:09.261 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_17 2024-09-11T06:41:09.261 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:41:09.261 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-image_17.status 2024-09-11T06:41:09.262 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-image_17.status 2024-09-11T06:41:09.262 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:41:09.264 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_17.status 2024-09-11T06:41:09.264 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_17 2024-09-11T06:41:09.264 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_17.status 2024-09-11T06:41:09.311 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 1 2024-09-11T06:41:09.311 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 1 2024-09-11T06:41:09.312 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:41:09.312 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:41:09.312 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:41:09.315 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_17.status 2024-09-11T06:41:09.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-09-11T06:41:09.317 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror image_17 2024-09-11T06:41:09.317 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:41:09.317 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:41:09.317 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_17 2024-09-11T06:41:09.317 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror image_17 '..*' 2024-09-11T06:41:09.317 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:41:09.318 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:41:09.318 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_17 2024-09-11T06:41:09.318 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:41:09.318 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-image_17.status 2024-09-11T06:41:09.318 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-image_17.status 2024-09-11T06:41:09.318 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:41:09.318 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_17.status 2024-09-11T06:41:09.319 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_17 2024-09-11T06:41:09.319 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_17.status 2024-09-11T06:41:09.364 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 1 2024-09-11T06:41:09.365 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 1 2024-09-11T06:41:09.365 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:41:09.365 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:41:09.365 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:41:09.368 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_17.status 2024-09-11T06:41:09.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-09-11T06:41:09.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=3, entry_tid=1' 2024-09-11T06:41:09.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:41:09.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_snap_present cluster1 mirror image_17 snap 2024-09-11T06:41:09.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:41:09.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:09.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_17 2024-09-11T06:41:09.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:41:09.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:41:09.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T06:41:09.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:41:10.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 info mirror/image_17@snap 2024-09-11T06:41:10.438 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:rbd image 'image_17': 2024-09-11T06:41:10.438 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: size 128 MiB in 32 objects 2024-09-11T06:41:10.438 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: order 22 (4 MiB objects) 2024-09-11T06:41:10.438 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: snapshot_count: 1 2024-09-11T06:41:10.438 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: id: 10a671a1c98f 2024-09-11T06:41:10.438 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: block_name_prefix: rbd_data.10a671a1c98f 2024-09-11T06:41:10.439 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: format: 2 2024-09-11T06:41:10.439 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: features: layering, exclusive-lock, journaling 2024-09-11T06:41:10.439 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: op_features: 2024-09-11T06:41:10.439 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: flags: 2024-09-11T06:41:10.439 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: create_timestamp: Wed Sep 11 06:33:20 2024 2024-09-11T06:41:10.439 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: access_timestamp: Wed Sep 11 06:33:20 2024 2024-09-11T06:41:10.439 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: modify_timestamp: Wed Sep 11 06:34:30 2024 2024-09-11T06:41:10.439 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: protected: False 2024-09-11T06:41:10.439 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: journal: 10a671a1c98f 2024-09-11T06:41:10.439 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring state: enabled 2024-09-11T06:41:10.439 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring mode: journal 2024-09-11T06:41:10.439 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring global id: 149c48f1-9bc7-4c75-a58c-ec7f2b84f333 2024-09-11T06:41:10.439 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring primary: false 2024-09-11T06:41:10.443 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:41:10.443 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ compare_image_snaps mirror image_17 snap 2024-09-11T06:41:10.443 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:10.443 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_17 2024-09-11T06:41:10.443 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:41:10.443 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local ret=0 2024-09-11T06:41:10.443 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local rmt_export=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_17.export 2024-09-11T06:41:10.444 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loc_export=/tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_17.export 2024-09-11T06:41:10.444 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_17.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_17.export 2024-09-11T06:41:10.444 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror export image_17@snap /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_17.export 2024-09-11T06:41:10.603 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T06:41:10.609 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 -p mirror export image_17@snap /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_17.export 2024-09-11T06:41:10.871 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T06:41:10.883 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmp /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_17.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_17.export 2024-09-11T06:41:10.957 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_17.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_17.export 2024-09-11T06:41:11.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:41:11.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:41:11.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_18 2024-09-11T06:41:11.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_snap cluster2 mirror image_18 snap 2024-09-11T06:41:11.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:41:11.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:11.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_18 2024-09-11T06:41:11.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:41:11.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror snap create image_18@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T06:41:11.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_started cluster1 mirror image_18 2024-09-11T06:41:11.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:41:11.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:11.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_18 2024-09-11T06:41:11.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_state cluster1 mirror image_18 started 2024-09-11T06:41:11.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:41:11.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:11.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_18 2024-09-11T06:41:11.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=started 2024-09-11T06:41:11.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:41:11.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:41:11.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:41:12.711 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_replay_state cluster1 mirror image_18 started 2024-09-11T06:41:12.711 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:41:12.711 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:12.712 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_18 2024-09-11T06:41:12.712 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=started 2024-09-11T06:41:12.712 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local status_result 2024-09-11T06:41:12.712 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=stopped 2024-09-11T06:41:12.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_18 2024-09-11T06:41:12.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -i state 2024-09-11T06:41:12.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_instance=cluster1 2024-09-11T06:41:12.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:41:12.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:41:12.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance=cluster1 2024-09-11T06:41:12.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local loop_instance 2024-09-11T06:41:12.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:41:12.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sleep 0 2024-09-11T06:41:12.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:41:12.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ seq 0 1 2024-09-11T06:41:12.716 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:41:12.716 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_18 2024-09-11T06:41:12.716 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1:0 2024-09-11T06:41:12.716 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:41:12.716 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:41:12.716 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:41:12.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:41:12.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=cluster 2024-09-11T06:41:12.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:41:12.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:41:12.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:41:12.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:41:12.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval cluster=cluster1 2024-09-11T06:41:12.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:41:12.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:41:12.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:41:12.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:41:12.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local local_cluster=cluster1:0 2024-09-11T06:41:12.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster=cluster1 2024-09-11T06:41:12.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance 2024-09-11T06:41:12.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:41:12.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local val=cluster1:0 2024-09-11T06:41:12.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T06:41:12.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance_var_name=instance 2024-09-11T06:41:12.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:41:12.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:41:12.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T06:41:12.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval local_cluster=cluster1 2024-09-11T06:41:12.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ local_cluster=cluster1 2024-09-11T06:41:12.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval instance=0 2024-09-11T06:41:12.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ instance=0 2024-09-11T06:41:12.721 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:41:12.740 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:41:12.741 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:41:12.741 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:41:12.741 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_18 2024-09-11T06:41:12.872 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:Invalid command: unused arguments: ['mirror/image_18'] 2024-09-11T06:41:12.872 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T06:41:12.872 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:admin_socket: invalid command 2024-09-11T06:41:12.876 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:41:12.876 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_18 2024-09-11T06:41:12.876 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1:1 2024-09-11T06:41:12.876 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:41:12.876 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:41:12.876 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-11T06:41:12.876 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:1 2024-09-11T06:41:12.876 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=cluster 2024-09-11T06:41:12.877 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:41:12.877 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:41:12.877 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=1 2024-09-11T06:41:12.877 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T06:41:12.877 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval cluster=cluster1 2024-09-11T06:41:12.877 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:41:12.877 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=1 2024-09-11T06:41:12.877 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=1 2024-09-11T06:41:12.877 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-11T06:41:12.878 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local local_cluster=cluster1:1 2024-09-11T06:41:12.878 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster=cluster1 2024-09-11T06:41:12.878 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance 2024-09-11T06:41:12.878 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T06:41:12.878 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local val=cluster1:1 2024-09-11T06:41:12.878 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T06:41:12.878 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance_var_name=instance 2024-09-11T06:41:12.878 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:41:12.878 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=1 2024-09-11T06:41:12.878 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-11T06:41:12.878 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval local_cluster=cluster1 2024-09-11T06:41:12.878 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ local_cluster=cluster1 2024-09-11T06:41:12.879 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval instance=1 2024-09-11T06:41:12.879 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ instance=1 2024-09-11T06:41:12.879 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T06:41:12.894 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:41:12.895 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:41:12.895 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:41:12.895 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_18 2024-09-11T06:41:13.029 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ return 0 2024-09-11T06:41:13.030 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T06:41:13.030 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ echo ' "state": "Replaying"' 2024-09-11T06:41:13.030 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ grep -i Replaying 2024-09-11T06:41:13.032 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: "state": "Replaying" 2024-09-11T06:41:13.032 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_state=started 2024-09-11T06:41:13.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test started = started 2024-09-11T06:41:13.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:41:13.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_18 2024-09-11T06:41:13.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:41:13.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:41:13.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:13.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_18 2024-09-11T06:41:13.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' journal = journal ']' 2024-09-11T06:41:13.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_18 2024-09-11T06:41:13.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:41:13.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:41:13.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:13.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_18 2024-09-11T06:41:13.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-11T06:41:13.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-11T06:41:13.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:41:13.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:41:13.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:41:13.234 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror image_18 2024-09-11T06:41:13.234 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:41:13.234 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:13.234 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_18 2024-09-11T06:41:13.234 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:41:13.234 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n image_18 ']' 2024-09-11T06:41:13.235 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/image_18' 2024-09-11T06:41:13.235 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_18 2024-09-11T06:41:13.235 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:41:13.235 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:41:13.235 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:41:13.235 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:41:13.235 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:41:13.235 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:41:13.235 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:41:13.236 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:41:13.236 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:41:13.238 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:41:13.238 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_18 2024-09-11T06:41:13.238 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:41:13.238 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:41:13.239 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:41:13.239 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:41:13.239 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:41:13.239 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:41:13.239 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:41:13.239 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:41:13.239 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:41:13.239 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:41:13.239 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:41:13.239 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:41:13.239 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:41:13.239 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:41:13.239 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:41:13.239 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:41:13.240 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:41:13.240 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:41:13.240 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:41:13.240 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:41:13.240 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:41:13.240 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:41:13.240 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:41:13.240 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:41:13.240 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:41:13.240 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:41:13.240 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:41:13.240 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:41:13.240 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:41:13.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:41:13.264 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:41:13.265 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:41:13.265 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:41:13.265 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_18 2024-09-11T06:41:13.386 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:Invalid command: unused arguments: ['mirror/image_18'] 2024-09-11T06:41:13.386 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd mirror flush : flush rbd mirror 2024-09-11T06:41:13.386 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:admin_socket: invalid command 2024-09-11T06:41:13.388 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:41:13.388 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_18 2024-09-11T06:41:13.388 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:1 2024-09-11T06:41:13.388 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:41:13.388 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:41:13.388 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-11T06:41:13.389 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:1 2024-09-11T06:41:13.389 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:41:13.389 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:41:13.389 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:41:13.389 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=1 2024-09-11T06:41:13.389 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-11T06:41:13.389 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:41:13.389 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:41:13.389 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=1 2024-09-11T06:41:13.389 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=1 2024-09-11T06:41:13.391 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-11T06:41:13.391 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:1 2024-09-11T06:41:13.391 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:41:13.391 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:41:13.391 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T06:41:13.391 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:1 2024-09-11T06:41:13.391 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:41:13.391 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:41:13.391 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:41:13.392 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=1 2024-09-11T06:41:13.392 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T06:41:13.392 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:41:13.392 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:41:13.392 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=1 2024-09-11T06:41:13.392 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=1 2024-09-11T06:41:13.392 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T06:41:13.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:41:13.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:41:13.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:41:13.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_18 2024-09-11T06:41:13.548 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:41:13.550 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror image_18 2024-09-11T06:41:13.550 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:41:13.550 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:41:13.550 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_18 2024-09-11T06:41:13.550 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror image_18 '' 2024-09-11T06:41:13.550 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:41:13.550 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:41:13.551 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_18 2024-09-11T06:41:13.551 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:41:13.551 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-image_18.status 2024-09-11T06:41:13.551 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-image_18.status 2024-09-11T06:41:13.551 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:41:13.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_18.status 2024-09-11T06:41:13.553 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_18.status 2024-09-11T06:41:13.553 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_18 2024-09-11T06:41:13.604 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 1 2024-09-11T06:41:13.605 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 1 2024-09-11T06:41:13.605 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:41:13.605 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:41:13.605 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:41:13.608 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_18.status 2024-09-11T06:41:13.609 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-09-11T06:41:13.610 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror image_18 2024-09-11T06:41:13.610 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:41:13.611 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:41:13.611 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_18 2024-09-11T06:41:13.611 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror image_18 '..*' 2024-09-11T06:41:13.611 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:41:13.611 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:41:13.611 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_18 2024-09-11T06:41:13.611 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:41:13.611 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-image_18.status 2024-09-11T06:41:13.611 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-image_18.status 2024-09-11T06:41:13.611 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:41:13.612 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_18.status 2024-09-11T06:41:13.612 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_18 2024-09-11T06:41:13.612 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_18.status 2024-09-11T06:41:13.659 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 1 2024-09-11T06:41:13.660 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 1 2024-09-11T06:41:13.660 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:41:13.660 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:41:13.660 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:41:13.663 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_18.status 2024-09-11T06:41:13.665 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-09-11T06:41:13.665 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=3, entry_tid=1' 2024-09-11T06:41:13.665 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:41:13.665 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_snap_present cluster1 mirror image_18 snap 2024-09-11T06:41:13.665 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:41:13.665 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:13.665 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_18 2024-09-11T06:41:13.665 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:41:13.665 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:41:13.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T06:41:13.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:41:14.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 info mirror/image_18@snap 2024-09-11T06:41:14.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:rbd image 'image_18': 2024-09-11T06:41:14.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: size 128 MiB in 32 objects 2024-09-11T06:41:14.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: order 22 (4 MiB objects) 2024-09-11T06:41:14.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: snapshot_count: 1 2024-09-11T06:41:14.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: id: 10a9c124dfb7 2024-09-11T06:41:14.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: block_name_prefix: rbd_data.10a9c124dfb7 2024-09-11T06:41:14.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: format: 2 2024-09-11T06:41:14.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: features: layering, exclusive-lock, journaling 2024-09-11T06:41:14.734 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: op_features: 2024-09-11T06:41:14.734 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: flags: 2024-09-11T06:41:14.734 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: create_timestamp: Wed Sep 11 06:33:25 2024 2024-09-11T06:41:14.734 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: access_timestamp: Wed Sep 11 06:33:25 2024 2024-09-11T06:41:14.734 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: modify_timestamp: Wed Sep 11 06:34:52 2024 2024-09-11T06:41:14.734 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: protected: False 2024-09-11T06:41:14.734 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: journal: 10a9c124dfb7 2024-09-11T06:41:14.734 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring state: enabled 2024-09-11T06:41:14.734 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring mode: journal 2024-09-11T06:41:14.734 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring global id: e2e262bd-f7d3-4907-8145-48cd0c6dec76 2024-09-11T06:41:14.734 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring primary: false 2024-09-11T06:41:14.737 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:41:14.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ compare_image_snaps mirror image_18 snap 2024-09-11T06:41:14.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:14.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_18 2024-09-11T06:41:14.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:41:14.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local ret=0 2024-09-11T06:41:14.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local rmt_export=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_18.export 2024-09-11T06:41:14.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loc_export=/tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_18.export 2024-09-11T06:41:14.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_18.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_18.export 2024-09-11T06:41:14.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror export image_18@snap /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_18.export 2024-09-11T06:41:14.942 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T06:41:14.952 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 -p mirror export image_18@snap /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_18.export 2024-09-11T06:41:15.281 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T06:41:15.293 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmp /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_18.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_18.export 2024-09-11T06:41:15.376 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_18.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_18.export 2024-09-11T06:41:15.416 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:41:15.416 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:41:15.416 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_19 2024-09-11T06:41:15.416 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_snap cluster2 mirror image_19 snap 2024-09-11T06:41:15.416 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:41:15.416 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:15.416 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_19 2024-09-11T06:41:15.416 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:41:15.416 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror snap create image_19@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T06:41:15.734 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_started cluster1 mirror image_19 2024-09-11T06:41:15.734 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:41:15.734 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:15.734 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_19 2024-09-11T06:41:15.735 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_state cluster1 mirror image_19 started 2024-09-11T06:41:15.735 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:41:15.735 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:15.735 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_19 2024-09-11T06:41:15.735 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=started 2024-09-11T06:41:15.735 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:41:15.735 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:41:15.735 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:41:16.736 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_replay_state cluster1 mirror image_19 started 2024-09-11T06:41:16.736 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:41:16.736 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:16.736 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_19 2024-09-11T06:41:16.736 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=started 2024-09-11T06:41:16.736 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local status_result 2024-09-11T06:41:16.736 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=stopped 2024-09-11T06:41:16.737 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_19 2024-09-11T06:41:16.737 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -i state 2024-09-11T06:41:16.737 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_instance=cluster1 2024-09-11T06:41:16.737 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:41:16.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:41:16.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance=cluster1 2024-09-11T06:41:16.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local loop_instance 2024-09-11T06:41:16.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:41:16.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sleep 0 2024-09-11T06:41:16.739 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:41:16.739 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ seq 0 1 2024-09-11T06:41:16.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:41:16.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_19 2024-09-11T06:41:16.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1:0 2024-09-11T06:41:16.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:41:16.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:41:16.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:41:16.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:41:16.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=cluster 2024-09-11T06:41:16.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:41:16.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:41:16.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:41:16.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:41:16.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval cluster=cluster1 2024-09-11T06:41:16.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:41:16.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:41:16.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:41:16.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:41:16.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local local_cluster=cluster1:0 2024-09-11T06:41:16.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster=cluster1 2024-09-11T06:41:16.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance 2024-09-11T06:41:16.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:41:16.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local val=cluster1:0 2024-09-11T06:41:16.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T06:41:16.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance_var_name=instance 2024-09-11T06:41:16.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:41:16.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:41:16.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T06:41:16.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval local_cluster=cluster1 2024-09-11T06:41:16.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ local_cluster=cluster1 2024-09-11T06:41:16.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval instance=0 2024-09-11T06:41:16.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ instance=0 2024-09-11T06:41:16.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:41:16.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:41:16.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:41:16.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:41:16.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_19 2024-09-11T06:41:16.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ return 0 2024-09-11T06:41:16.905 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T06:41:16.905 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ echo ' "state": "Replaying"' 2024-09-11T06:41:16.906 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ grep -i Replaying 2024-09-11T06:41:16.907 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: "state": "Replaying" 2024-09-11T06:41:16.909 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_state=started 2024-09-11T06:41:16.909 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test started = started 2024-09-11T06:41:16.909 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:41:16.909 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_19 2024-09-11T06:41:16.909 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:41:16.909 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:41:16.909 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:16.909 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_19 2024-09-11T06:41:16.909 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' journal = journal ']' 2024-09-11T06:41:16.909 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_19 2024-09-11T06:41:16.910 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:41:16.910 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:41:16.910 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:16.910 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_19 2024-09-11T06:41:16.910 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-11T06:41:16.910 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-11T06:41:16.910 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:41:16.910 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:41:16.910 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:41:17.110 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror image_19 2024-09-11T06:41:17.110 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:41:17.110 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:17.110 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_19 2024-09-11T06:41:17.110 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:41:17.110 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n image_19 ']' 2024-09-11T06:41:17.110 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/image_19' 2024-09-11T06:41:17.110 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_19 2024-09-11T06:41:17.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:41:17.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:41:17.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:41:17.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:41:17.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:41:17.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:41:17.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:41:17.112 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:41:17.112 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:41:17.114 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:41:17.114 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_19 2024-09-11T06:41:17.114 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:41:17.114 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:41:17.114 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:41:17.115 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:41:17.115 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:41:17.115 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:41:17.115 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:41:17.115 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:41:17.115 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:41:17.115 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:41:17.115 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:41:17.115 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:41:17.115 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:41:17.115 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:41:17.115 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:41:17.115 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:41:17.116 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:41:17.116 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:41:17.116 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:41:17.116 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:41:17.116 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:41:17.116 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:41:17.116 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:41:17.116 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:41:17.116 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:41:17.116 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:41:17.116 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:41:17.116 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:41:17.116 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:41:17.117 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:41:17.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:41:17.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:41:17.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:41:17.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_19 2024-09-11T06:41:17.282 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:41:17.284 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror image_19 2024-09-11T06:41:17.284 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:41:17.284 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:41:17.284 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_19 2024-09-11T06:41:17.284 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror image_19 '' 2024-09-11T06:41:17.284 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:41:17.284 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:41:17.284 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_19 2024-09-11T06:41:17.285 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:41:17.285 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-image_19.status 2024-09-11T06:41:17.285 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-image_19.status 2024-09-11T06:41:17.285 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:41:17.288 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_19.status 2024-09-11T06:41:17.288 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_19 2024-09-11T06:41:17.288 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_19.status 2024-09-11T06:41:17.339 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 1 2024-09-11T06:41:17.340 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 1 2024-09-11T06:41:17.340 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:41:17.340 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:41:17.340 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:41:17.343 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_19.status 2024-09-11T06:41:17.344 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-09-11T06:41:17.346 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror image_19 2024-09-11T06:41:17.346 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:41:17.346 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:41:17.346 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_19 2024-09-11T06:41:17.346 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror image_19 '..*' 2024-09-11T06:41:17.347 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:41:17.347 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:41:17.347 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_19 2024-09-11T06:41:17.347 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:41:17.347 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-image_19.status 2024-09-11T06:41:17.347 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-image_19.status 2024-09-11T06:41:17.347 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:41:17.348 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_19.status 2024-09-11T06:41:17.348 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_19 2024-09-11T06:41:17.348 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_19.status 2024-09-11T06:41:17.394 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 1 2024-09-11T06:41:17.394 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 1 2024-09-11T06:41:17.394 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:41:17.395 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:41:17.395 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:41:17.397 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_19.status 2024-09-11T06:41:17.399 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-09-11T06:41:17.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=3, entry_tid=1' 2024-09-11T06:41:17.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:41:17.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_snap_present cluster1 mirror image_19 snap 2024-09-11T06:41:17.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:41:17.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:17.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_19 2024-09-11T06:41:17.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:41:17.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:41:17.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T06:41:17.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:41:18.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 info mirror/image_19@snap 2024-09-11T06:41:18.468 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:rbd image 'image_19': 2024-09-11T06:41:18.468 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: size 128 MiB in 32 objects 2024-09-11T06:41:18.468 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: order 22 (4 MiB objects) 2024-09-11T06:41:18.468 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: snapshot_count: 1 2024-09-11T06:41:18.468 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: id: 10a6142f7a1c 2024-09-11T06:41:18.468 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: block_name_prefix: rbd_data.10a6142f7a1c 2024-09-11T06:41:18.468 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: format: 2 2024-09-11T06:41:18.468 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: features: layering, exclusive-lock, journaling 2024-09-11T06:41:18.468 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: op_features: 2024-09-11T06:41:18.468 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: flags: 2024-09-11T06:41:18.469 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: create_timestamp: Wed Sep 11 06:33:32 2024 2024-09-11T06:41:18.469 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: access_timestamp: Wed Sep 11 06:33:32 2024 2024-09-11T06:41:18.469 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: modify_timestamp: Wed Sep 11 06:35:26 2024 2024-09-11T06:41:18.469 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: protected: False 2024-09-11T06:41:18.469 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: journal: 10a6142f7a1c 2024-09-11T06:41:18.469 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring state: enabled 2024-09-11T06:41:18.469 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring mode: journal 2024-09-11T06:41:18.469 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring global id: 92dc2a15-3ceb-4dd0-bcca-7245cfd7de86 2024-09-11T06:41:18.469 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring primary: false 2024-09-11T06:41:18.472 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:41:18.473 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ compare_image_snaps mirror image_19 snap 2024-09-11T06:41:18.473 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:18.473 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_19 2024-09-11T06:41:18.473 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:41:18.473 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local ret=0 2024-09-11T06:41:18.473 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local rmt_export=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_19.export 2024-09-11T06:41:18.473 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loc_export=/tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_19.export 2024-09-11T06:41:18.473 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_19.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_19.export 2024-09-11T06:41:18.473 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror export image_19@snap /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_19.export 2024-09-11T06:41:18.652 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T06:41:18.659 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 -p mirror export image_19@snap /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_19.export 2024-09-11T06:41:18.963 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T06:41:18.974 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmp /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_19.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_19.export 2024-09-11T06:41:19.052 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_19.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_19.export 2024-09-11T06:41:19.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:41:19.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:41:19.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_20 2024-09-11T06:41:19.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_snap cluster2 mirror image_20 snap 2024-09-11T06:41:19.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:41:19.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:19.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_20 2024-09-11T06:41:19.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:41:19.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror snap create image_20@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T06:41:19.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_started cluster1 mirror image_20 2024-09-11T06:41:19.739 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:41:19.739 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:19.739 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_20 2024-09-11T06:41:19.739 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_state cluster1 mirror image_20 started 2024-09-11T06:41:19.739 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:41:19.739 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:19.739 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_20 2024-09-11T06:41:19.739 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=started 2024-09-11T06:41:19.739 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:41:19.739 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:41:19.739 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:41:20.740 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_replay_state cluster1 mirror image_20 started 2024-09-11T06:41:20.740 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:41:20.740 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:20.740 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_20 2024-09-11T06:41:20.740 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=started 2024-09-11T06:41:20.741 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local status_result 2024-09-11T06:41:20.741 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=stopped 2024-09-11T06:41:20.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_20 2024-09-11T06:41:20.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -i state 2024-09-11T06:41:20.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_instance=cluster1 2024-09-11T06:41:20.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:41:20.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:41:20.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance=cluster1 2024-09-11T06:41:20.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local loop_instance 2024-09-11T06:41:20.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:41:20.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sleep 0 2024-09-11T06:41:20.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:41:20.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ seq 0 1 2024-09-11T06:41:20.746 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:41:20.746 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_20 2024-09-11T06:41:20.746 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1:0 2024-09-11T06:41:20.746 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:41:20.746 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:41:20.746 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:41:20.747 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:41:20.747 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=cluster 2024-09-11T06:41:20.747 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:41:20.747 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:41:20.747 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:41:20.747 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:41:20.747 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval cluster=cluster1 2024-09-11T06:41:20.747 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:41:20.747 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:41:20.747 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:41:20.749 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:41:20.749 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local local_cluster=cluster1:0 2024-09-11T06:41:20.749 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster=cluster1 2024-09-11T06:41:20.749 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance 2024-09-11T06:41:20.749 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:41:20.750 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local val=cluster1:0 2024-09-11T06:41:20.750 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T06:41:20.750 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance_var_name=instance 2024-09-11T06:41:20.750 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:41:20.750 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:41:20.750 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T06:41:20.750 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval local_cluster=cluster1 2024-09-11T06:41:20.750 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ local_cluster=cluster1 2024-09-11T06:41:20.750 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval instance=0 2024-09-11T06:41:20.750 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ instance=0 2024-09-11T06:41:20.750 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:41:20.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:41:20.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:41:20.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:41:20.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_20 2024-09-11T06:41:20.908 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2024-09-11T06:41:20.908 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T06:41:20.908 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:admin_socket: invalid command 2024-09-11T06:41:20.910 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:41:20.910 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_20 2024-09-11T06:41:20.910 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1:1 2024-09-11T06:41:20.911 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:41:20.911 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:41:20.911 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-11T06:41:20.911 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:1 2024-09-11T06:41:20.911 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=cluster 2024-09-11T06:41:20.911 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:41:20.911 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:41:20.911 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=1 2024-09-11T06:41:20.911 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T06:41:20.911 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval cluster=cluster1 2024-09-11T06:41:20.911 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:41:20.911 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=1 2024-09-11T06:41:20.911 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=1 2024-09-11T06:41:20.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-11T06:41:20.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local local_cluster=cluster1:1 2024-09-11T06:41:20.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster=cluster1 2024-09-11T06:41:20.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance 2024-09-11T06:41:20.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T06:41:20.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local val=cluster1:1 2024-09-11T06:41:20.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T06:41:20.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance_var_name=instance 2024-09-11T06:41:20.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:41:20.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=1 2024-09-11T06:41:20.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-11T06:41:20.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval local_cluster=cluster1 2024-09-11T06:41:20.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ local_cluster=cluster1 2024-09-11T06:41:20.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval instance=1 2024-09-11T06:41:20.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ instance=1 2024-09-11T06:41:20.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T06:41:20.936 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:41:20.936 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:41:20.936 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:41:20.936 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_20 2024-09-11T06:41:21.074 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ return 0 2024-09-11T06:41:21.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T06:41:21.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ echo ' "state": "Replaying"' 2024-09-11T06:41:21.076 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ grep -i Replaying 2024-09-11T06:41:21.076 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: "state": "Replaying" 2024-09-11T06:41:21.078 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_state=started 2024-09-11T06:41:21.078 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test started = started 2024-09-11T06:41:21.078 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:41:21.078 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_20 2024-09-11T06:41:21.078 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:41:21.078 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:41:21.078 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:21.078 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_20 2024-09-11T06:41:21.079 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' journal = journal ']' 2024-09-11T06:41:21.079 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_20 2024-09-11T06:41:21.079 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:41:21.079 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:41:21.079 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:21.079 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_20 2024-09-11T06:41:21.079 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-11T06:41:21.079 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-11T06:41:21.079 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:41:21.079 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:41:21.079 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:41:21.279 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror image_20 2024-09-11T06:41:21.280 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:41:21.280 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:21.280 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_20 2024-09-11T06:41:21.280 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:41:21.280 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n image_20 ']' 2024-09-11T06:41:21.280 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/image_20' 2024-09-11T06:41:21.280 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_20 2024-09-11T06:41:21.280 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:41:21.280 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:41:21.280 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:41:21.280 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:41:21.281 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:41:21.281 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:41:21.281 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:41:21.281 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:41:21.281 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:41:21.283 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:41:21.283 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_20 2024-09-11T06:41:21.283 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:41:21.283 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:41:21.283 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:41:21.283 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:41:21.284 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:41:21.284 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:41:21.284 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:41:21.284 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:41:21.284 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:41:21.284 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:41:21.284 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:41:21.284 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:41:21.284 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:41:21.284 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:41:21.286 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:41:21.286 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:41:21.286 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:41:21.286 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:41:21.286 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:41:21.286 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:41:21.286 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:41:21.286 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:41:21.286 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:41:21.286 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:41:21.286 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:41:21.286 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:41:21.286 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:41:21.287 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:41:21.287 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:41:21.287 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:41:21.312 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:41:21.313 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:41:21.313 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:41:21.313 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_20 2024-09-11T06:41:21.435 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2024-09-11T06:41:21.435 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd mirror flush : flush rbd mirror 2024-09-11T06:41:21.435 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:admin_socket: invalid command 2024-09-11T06:41:21.440 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:41:21.440 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_20 2024-09-11T06:41:21.440 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:1 2024-09-11T06:41:21.440 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:41:21.440 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:41:21.440 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-11T06:41:21.440 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:1 2024-09-11T06:41:21.440 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:41:21.440 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:41:21.440 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:41:21.440 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=1 2024-09-11T06:41:21.440 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-11T06:41:21.441 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:41:21.441 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:41:21.441 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=1 2024-09-11T06:41:21.441 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=1 2024-09-11T06:41:21.441 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-11T06:41:21.441 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:1 2024-09-11T06:41:21.441 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:41:21.441 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:41:21.441 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T06:41:21.441 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:1 2024-09-11T06:41:21.441 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:41:21.441 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:41:21.441 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:41:21.441 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=1 2024-09-11T06:41:21.442 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T06:41:21.442 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:41:21.442 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:41:21.442 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=1 2024-09-11T06:41:21.442 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=1 2024-09-11T06:41:21.442 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T06:41:21.462 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:41:21.463 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:41:21.463 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:41:21.463 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_20 2024-09-11T06:41:21.607 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:41:21.609 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror image_20 2024-09-11T06:41:21.609 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:41:21.609 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:41:21.609 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_20 2024-09-11T06:41:21.609 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror image_20 '' 2024-09-11T06:41:21.609 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:41:21.609 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:41:21.609 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_20 2024-09-11T06:41:21.609 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:41:21.610 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-image_20.status 2024-09-11T06:41:21.610 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-image_20.status 2024-09-11T06:41:21.610 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:41:21.611 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_20.status 2024-09-11T06:41:21.612 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_20 2024-09-11T06:41:21.612 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_20.status 2024-09-11T06:41:21.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 3 2024-09-11T06:41:21.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 3 2024-09-11T06:41:21.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:41:21.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=3], [object_number=14, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:41:21.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=3], [object_number=14, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:41:21.660 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_20.status 2024-09-11T06:41:21.661 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=13, tag_tid=3, entry_tid=1' 2024-09-11T06:41:21.663 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror image_20 2024-09-11T06:41:21.663 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:41:21.663 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:41:21.663 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_20 2024-09-11T06:41:21.663 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror image_20 '..*' 2024-09-11T06:41:21.663 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:41:21.663 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:41:21.663 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_20 2024-09-11T06:41:21.663 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:41:21.663 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-image_20.status 2024-09-11T06:41:21.663 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-image_20.status 2024-09-11T06:41:21.664 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:41:21.664 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_20.status 2024-09-11T06:41:21.664 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_20 2024-09-11T06:41:21.664 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_20.status 2024-09-11T06:41:21.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 3 2024-09-11T06:41:21.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 3 2024-09-11T06:41:21.716 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:41:21.716 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=3], [object_number=14, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:41:21.716 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=3], [object_number=14, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:41:21.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_20.status 2024-09-11T06:41:21.721 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=13, tag_tid=3, entry_tid=1' 2024-09-11T06:41:21.721 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=13, tag_tid=3, entry_tid=1' -a 'object_number=13, tag_tid=3, entry_tid=1' = 'object_number=13, tag_tid=3, entry_tid=1' 2024-09-11T06:41:21.722 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:41:21.722 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_snap_present cluster1 mirror image_20 snap 2024-09-11T06:41:21.722 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:41:21.722 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:21.722 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_20 2024-09-11T06:41:21.722 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:41:21.722 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:41:21.722 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T06:41:21.722 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:41:22.722 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 info mirror/image_20@snap 2024-09-11T06:41:22.789 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:rbd image 'image_20': 2024-09-11T06:41:22.789 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: size 128 MiB in 32 objects 2024-09-11T06:41:22.789 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: order 22 (4 MiB objects) 2024-09-11T06:41:22.789 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: snapshot_count: 1 2024-09-11T06:41:22.789 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: id: 10a9cd9cef14 2024-09-11T06:41:22.789 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: block_name_prefix: rbd_data.10a9cd9cef14 2024-09-11T06:41:22.789 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: format: 2 2024-09-11T06:41:22.789 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: features: layering, exclusive-lock, journaling 2024-09-11T06:41:22.789 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: op_features: 2024-09-11T06:41:22.790 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: flags: 2024-09-11T06:41:22.790 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: create_timestamp: Wed Sep 11 06:33:38 2024 2024-09-11T06:41:22.790 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: access_timestamp: Wed Sep 11 06:33:38 2024 2024-09-11T06:41:22.790 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: modify_timestamp: Wed Sep 11 06:36:34 2024 2024-09-11T06:41:22.790 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: protected: False 2024-09-11T06:41:22.790 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: journal: 10a9cd9cef14 2024-09-11T06:41:22.790 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring state: enabled 2024-09-11T06:41:22.790 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring mode: journal 2024-09-11T06:41:22.790 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring global id: 8008bd49-5529-4450-9df0-497e13c43559 2024-09-11T06:41:22.790 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring primary: false 2024-09-11T06:41:22.794 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:41:22.794 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ compare_image_snaps mirror image_20 snap 2024-09-11T06:41:22.794 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:22.794 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_20 2024-09-11T06:41:22.794 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:41:22.794 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local ret=0 2024-09-11T06:41:22.794 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local rmt_export=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_20.export 2024-09-11T06:41:22.795 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loc_export=/tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_20.export 2024-09-11T06:41:22.795 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_20.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_20.export 2024-09-11T06:41:22.795 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror export image_20@snap /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_20.export 2024-09-11T06:41:23.019 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T06:41:23.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 -p mirror export image_20@snap /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_20.export 2024-09-11T06:41:23.553 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T06:41:23.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmp /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_20.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_20.export 2024-09-11T06:41:23.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_20.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_20.export 2024-09-11T06:41:23.683 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:41:23.683 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:41:23.683 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_21 2024-09-11T06:41:23.683 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_snap cluster2 mirror image_21 snap 2024-09-11T06:41:23.684 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:41:23.684 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:23.684 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_21 2024-09-11T06:41:23.684 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:41:23.684 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror snap create image_21@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T06:41:24.750 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_started cluster1 mirror image_21 2024-09-11T06:41:24.750 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:41:24.750 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:24.750 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_21 2024-09-11T06:41:24.750 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_state cluster1 mirror image_21 started 2024-09-11T06:41:24.750 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:41:24.751 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:24.751 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_21 2024-09-11T06:41:24.751 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=started 2024-09-11T06:41:24.751 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:41:24.751 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:41:24.751 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:41:25.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_replay_state cluster1 mirror image_21 started 2024-09-11T06:41:25.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:41:25.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:25.753 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_21 2024-09-11T06:41:25.753 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=started 2024-09-11T06:41:25.753 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local status_result 2024-09-11T06:41:25.753 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=stopped 2024-09-11T06:41:25.754 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_21 2024-09-11T06:41:25.754 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -i state 2024-09-11T06:41:25.754 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_instance=cluster1 2024-09-11T06:41:25.754 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:41:25.754 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:41:25.754 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance=cluster1 2024-09-11T06:41:25.754 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local loop_instance 2024-09-11T06:41:25.754 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:41:25.754 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sleep 0 2024-09-11T06:41:25.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:41:25.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ seq 0 1 2024-09-11T06:41:25.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:41:25.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_21 2024-09-11T06:41:25.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1:0 2024-09-11T06:41:25.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:41:25.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:41:25.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:41:25.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:41:25.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=cluster 2024-09-11T06:41:25.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:41:25.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:41:25.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:41:25.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:41:25.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval cluster=cluster1 2024-09-11T06:41:25.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:41:25.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:41:25.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:41:25.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:41:25.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local local_cluster=cluster1:0 2024-09-11T06:41:25.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster=cluster1 2024-09-11T06:41:25.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance 2024-09-11T06:41:25.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:41:25.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local val=cluster1:0 2024-09-11T06:41:25.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T06:41:25.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance_var_name=instance 2024-09-11T06:41:25.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:41:25.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:41:25.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T06:41:25.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval local_cluster=cluster1 2024-09-11T06:41:25.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ local_cluster=cluster1 2024-09-11T06:41:25.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval instance=0 2024-09-11T06:41:25.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ instance=0 2024-09-11T06:41:25.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:41:25.779 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:41:25.780 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:41:25.780 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:41:25.780 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_21 2024-09-11T06:41:25.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ return 0 2024-09-11T06:41:25.921 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T06:41:25.921 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ echo ' "state": "Replaying"' 2024-09-11T06:41:25.921 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ grep -i Replaying 2024-09-11T06:41:25.922 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: "state": "Replaying" 2024-09-11T06:41:25.924 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_state=started 2024-09-11T06:41:25.924 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test started = started 2024-09-11T06:41:25.924 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:41:25.924 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_21 2024-09-11T06:41:25.924 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:41:25.925 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:41:25.925 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:25.925 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_21 2024-09-11T06:41:25.925 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' journal = journal ']' 2024-09-11T06:41:25.925 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_21 2024-09-11T06:41:25.925 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:41:25.925 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:41:25.925 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:25.925 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_21 2024-09-11T06:41:25.925 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-11T06:41:25.925 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-11T06:41:25.925 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:41:25.925 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:41:25.926 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:41:26.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror image_21 2024-09-11T06:41:26.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:41:26.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:26.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_21 2024-09-11T06:41:26.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:41:26.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n image_21 ']' 2024-09-11T06:41:26.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/image_21' 2024-09-11T06:41:26.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_21 2024-09-11T06:41:26.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:41:26.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:41:26.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:41:26.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:41:26.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:41:26.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:41:26.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:41:26.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:41:26.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:41:26.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:41:26.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_21 2024-09-11T06:41:26.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:41:26.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:41:26.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:41:26.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:41:26.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:41:26.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:41:26.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:41:26.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:41:26.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:41:26.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:41:26.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:41:26.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:41:26.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:41:26.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:41:26.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:41:26.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:41:26.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:41:26.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:41:26.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:41:26.133 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:41:26.133 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:41:26.133 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:41:26.133 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:41:26.133 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:41:26.133 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:41:26.133 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:41:26.133 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:41:26.133 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:41:26.133 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:41:26.133 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:41:26.156 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:41:26.156 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:41:26.156 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:41:26.156 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_21 2024-09-11T06:41:26.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:41:26.300 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror image_21 2024-09-11T06:41:26.301 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:41:26.301 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:41:26.301 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_21 2024-09-11T06:41:26.301 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror image_21 '' 2024-09-11T06:41:26.301 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:41:26.301 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:41:26.301 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_21 2024-09-11T06:41:26.301 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:41:26.301 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-image_21.status 2024-09-11T06:41:26.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:41:26.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-image_21.status 2024-09-11T06:41:26.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_21.status 2024-09-11T06:41:26.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_21 2024-09-11T06:41:26.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_21.status 2024-09-11T06:41:26.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 3 2024-09-11T06:41:26.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 3 2024-09-11T06:41:26.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:41:26.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=3], [object_number=14, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:41:26.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=3], [object_number=14, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:41:26.358 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_21.status 2024-09-11T06:41:26.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=13, tag_tid=3, entry_tid=1' 2024-09-11T06:41:26.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror image_21 2024-09-11T06:41:26.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:41:26.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:41:26.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_21 2024-09-11T06:41:26.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror image_21 '..*' 2024-09-11T06:41:26.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:41:26.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:41:26.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_21 2024-09-11T06:41:26.362 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:41:26.362 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-image_21.status 2024-09-11T06:41:26.362 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-image_21.status 2024-09-11T06:41:26.362 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:41:26.362 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_21.status 2024-09-11T06:41:26.362 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_21 2024-09-11T06:41:26.362 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_21.status 2024-09-11T06:41:26.408 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 3 2024-09-11T06:41:26.408 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 3 2024-09-11T06:41:26.409 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:41:26.409 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=3], [object_number=14, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:41:26.409 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=13, tag_tid=3, entry_tid=1], [object_number=12, tag_tid=3, entry_tid=0], [object_number=15, tag_tid=2, entry_tid=3], [object_number=14, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:41:26.412 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_21.status 2024-09-11T06:41:26.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=13, tag_tid=3, entry_tid=1' 2024-09-11T06:41:26.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=13, tag_tid=3, entry_tid=1' -a 'object_number=13, tag_tid=3, entry_tid=1' = 'object_number=13, tag_tid=3, entry_tid=1' 2024-09-11T06:41:26.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:41:26.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_snap_present cluster1 mirror image_21 snap 2024-09-11T06:41:26.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:41:26.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:26.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_21 2024-09-11T06:41:26.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:41:26.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:41:26.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T06:41:26.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:41:27.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 info mirror/image_21@snap 2024-09-11T06:41:27.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:rbd image 'image_21': 2024-09-11T06:41:27.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: size 128 MiB in 32 objects 2024-09-11T06:41:27.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: order 22 (4 MiB objects) 2024-09-11T06:41:27.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: snapshot_count: 1 2024-09-11T06:41:27.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: id: 10a61db7278 2024-09-11T06:41:27.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: block_name_prefix: rbd_data.10a61db7278 2024-09-11T06:41:27.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: format: 2 2024-09-11T06:41:27.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: features: layering, exclusive-lock, journaling 2024-09-11T06:41:27.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: op_features: 2024-09-11T06:41:27.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: flags: 2024-09-11T06:41:27.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: create_timestamp: Wed Sep 11 06:33:54 2024 2024-09-11T06:41:27.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: access_timestamp: Wed Sep 11 06:33:54 2024 2024-09-11T06:41:27.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: modify_timestamp: Wed Sep 11 06:36:32 2024 2024-09-11T06:41:27.483 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: protected: False 2024-09-11T06:41:27.483 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: journal: 10a61db7278 2024-09-11T06:41:27.483 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring state: enabled 2024-09-11T06:41:27.483 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring mode: journal 2024-09-11T06:41:27.483 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring global id: a4e650a9-7c5b-4739-b403-56d2185b448f 2024-09-11T06:41:27.483 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring primary: false 2024-09-11T06:41:27.486 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:41:27.486 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ compare_image_snaps mirror image_21 snap 2024-09-11T06:41:27.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:27.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_21 2024-09-11T06:41:27.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:41:27.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local ret=0 2024-09-11T06:41:27.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local rmt_export=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_21.export 2024-09-11T06:41:27.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loc_export=/tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_21.export 2024-09-11T06:41:27.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_21.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_21.export 2024-09-11T06:41:27.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror export image_21@snap /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_21.export 2024-09-11T06:41:27.686 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T06:41:27.693 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 -p mirror export image_21@snap /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_21.export 2024-09-11T06:41:28.068 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T06:41:28.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmp /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_21.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_21.export 2024-09-11T06:41:28.156 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_21.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_21.export 2024-09-11T06:41:28.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:41:28.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:41:28.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_22 2024-09-11T06:41:28.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_snap cluster2 mirror image_22 snap 2024-09-11T06:41:28.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:41:28.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:28.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_22 2024-09-11T06:41:28.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:41:28.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror snap create image_22@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T06:41:28.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_started cluster1 mirror image_22 2024-09-11T06:41:28.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:41:28.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:28.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_22 2024-09-11T06:41:28.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_state cluster1 mirror image_22 started 2024-09-11T06:41:28.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:41:28.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:28.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_22 2024-09-11T06:41:28.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=started 2024-09-11T06:41:28.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:41:28.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:41:28.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:41:29.772 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_replay_state cluster1 mirror image_22 started 2024-09-11T06:41:29.772 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:41:29.772 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:29.772 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_22 2024-09-11T06:41:29.772 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=started 2024-09-11T06:41:29.772 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local status_result 2024-09-11T06:41:29.772 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=stopped 2024-09-11T06:41:29.773 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_22 2024-09-11T06:41:29.773 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -i state 2024-09-11T06:41:29.773 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_instance=cluster1 2024-09-11T06:41:29.774 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:41:29.774 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:41:29.774 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance=cluster1 2024-09-11T06:41:29.774 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local loop_instance 2024-09-11T06:41:29.774 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:41:29.774 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sleep 0 2024-09-11T06:41:29.774 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:41:29.775 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ seq 0 1 2024-09-11T06:41:29.777 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:41:29.777 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_22 2024-09-11T06:41:29.777 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1:0 2024-09-11T06:41:29.777 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:41:29.777 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:41:29.777 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:41:29.777 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:41:29.777 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=cluster 2024-09-11T06:41:29.778 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:41:29.778 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:41:29.778 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:41:29.778 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:41:29.778 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval cluster=cluster1 2024-09-11T06:41:29.778 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:41:29.778 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:41:29.778 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:41:29.778 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:41:29.778 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local local_cluster=cluster1:0 2024-09-11T06:41:29.779 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster=cluster1 2024-09-11T06:41:29.780 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance 2024-09-11T06:41:29.780 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:41:29.780 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local val=cluster1:0 2024-09-11T06:41:29.780 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T06:41:29.780 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance_var_name=instance 2024-09-11T06:41:29.780 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:41:29.780 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:41:29.780 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T06:41:29.780 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval local_cluster=cluster1 2024-09-11T06:41:29.780 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ local_cluster=cluster1 2024-09-11T06:41:29.780 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval instance=0 2024-09-11T06:41:29.780 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ instance=0 2024-09-11T06:41:29.781 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:41:29.799 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:41:29.799 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:41:29.800 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:41:29.800 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_22 2024-09-11T06:41:29.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:Invalid command: unused arguments: ['mirror/image_22'] 2024-09-11T06:41:29.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T06:41:29.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:admin_socket: invalid command 2024-09-11T06:41:29.953 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:41:29.953 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_22 2024-09-11T06:41:29.953 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1:1 2024-09-11T06:41:29.953 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:41:29.953 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:41:29.954 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-11T06:41:29.954 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:1 2024-09-11T06:41:29.954 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=cluster 2024-09-11T06:41:29.954 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:41:29.954 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:41:29.954 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=1 2024-09-11T06:41:29.954 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T06:41:29.954 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval cluster=cluster1 2024-09-11T06:41:29.954 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:41:29.954 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=1 2024-09-11T06:41:29.954 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=1 2024-09-11T06:41:29.954 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-11T06:41:29.954 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local local_cluster=cluster1:1 2024-09-11T06:41:29.954 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster=cluster1 2024-09-11T06:41:29.955 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance 2024-09-11T06:41:29.955 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T06:41:29.955 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local val=cluster1:1 2024-09-11T06:41:29.955 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T06:41:29.955 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance_var_name=instance 2024-09-11T06:41:29.955 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:41:29.955 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=1 2024-09-11T06:41:29.955 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-11T06:41:29.955 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval local_cluster=cluster1 2024-09-11T06:41:29.955 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ local_cluster=cluster1 2024-09-11T06:41:29.955 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval instance=1 2024-09-11T06:41:29.955 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ instance=1 2024-09-11T06:41:29.955 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T06:41:29.971 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:41:29.972 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:41:29.972 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:41:29.972 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_22 2024-09-11T06:41:30.102 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ return 0 2024-09-11T06:41:30.103 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T06:41:30.104 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ echo ' "state": "Replaying"' 2024-09-11T06:41:30.104 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ grep -i Replaying 2024-09-11T06:41:30.106 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: "state": "Replaying" 2024-09-11T06:41:30.108 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_state=started 2024-09-11T06:41:30.108 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test started = started 2024-09-11T06:41:30.108 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:41:30.108 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_22 2024-09-11T06:41:30.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:41:30.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:41:30.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:30.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_22 2024-09-11T06:41:30.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' journal = journal ']' 2024-09-11T06:41:30.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_22 2024-09-11T06:41:30.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:41:30.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:41:30.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:30.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_22 2024-09-11T06:41:30.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-11T06:41:30.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-11T06:41:30.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:41:30.110 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:41:30.110 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:41:30.310 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror image_22 2024-09-11T06:41:30.310 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:41:30.310 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:30.310 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_22 2024-09-11T06:41:30.310 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:41:30.310 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n image_22 ']' 2024-09-11T06:41:30.310 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/image_22' 2024-09-11T06:41:30.310 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_22 2024-09-11T06:41:30.310 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:41:30.311 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:41:30.311 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:41:30.311 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:41:30.311 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:41:30.311 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:41:30.311 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:41:30.311 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:41:30.311 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:41:30.314 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:41:30.315 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_22 2024-09-11T06:41:30.315 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:41:30.315 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:41:30.315 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:41:30.315 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:41:30.315 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:41:30.315 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:41:30.315 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:41:30.315 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:41:30.315 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:41:30.315 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:41:30.315 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:41:30.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:41:30.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:41:30.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:41:30.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:41:30.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:41:30.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:41:30.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:41:30.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:41:30.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:41:30.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:41:30.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:41:30.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:41:30.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:41:30.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:41:30.317 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:41:30.317 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:41:30.317 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:41:30.317 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:41:30.317 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:41:30.339 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:41:30.340 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:41:30.340 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:41:30.340 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_22 2024-09-11T06:41:30.462 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:Invalid command: unused arguments: ['mirror/image_22'] 2024-09-11T06:41:30.462 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd mirror flush : flush rbd mirror 2024-09-11T06:41:30.462 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:admin_socket: invalid command 2024-09-11T06:41:30.466 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:41:30.466 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_22 2024-09-11T06:41:30.466 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:1 2024-09-11T06:41:30.466 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:41:30.466 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:41:30.466 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-11T06:41:30.466 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:1 2024-09-11T06:41:30.467 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:41:30.467 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:41:30.467 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:41:30.467 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=1 2024-09-11T06:41:30.467 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-11T06:41:30.467 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:41:30.467 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:41:30.467 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=1 2024-09-11T06:41:30.467 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=1 2024-09-11T06:41:30.467 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-11T06:41:30.467 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:1 2024-09-11T06:41:30.467 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:41:30.467 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:41:30.467 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T06:41:30.468 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:1 2024-09-11T06:41:30.468 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:41:30.468 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:41:30.468 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:41:30.468 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=1 2024-09-11T06:41:30.468 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T06:41:30.468 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:41:30.468 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:41:30.468 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=1 2024-09-11T06:41:30.468 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=1 2024-09-11T06:41:30.468 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T06:41:30.491 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:41:30.491 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:41:30.491 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:41:30.492 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_22 2024-09-11T06:41:30.628 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:41:30.629 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror image_22 2024-09-11T06:41:30.629 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:41:30.629 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:41:30.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_22 2024-09-11T06:41:30.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror image_22 '' 2024-09-11T06:41:30.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:41:30.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:41:30.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_22 2024-09-11T06:41:30.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:41:30.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-image_22.status 2024-09-11T06:41:30.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-image_22.status 2024-09-11T06:41:30.631 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:41:30.632 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_22.status 2024-09-11T06:41:30.632 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_22 2024-09-11T06:41:30.632 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_22.status 2024-09-11T06:41:30.685 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 1 2024-09-11T06:41:30.686 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 1 2024-09-11T06:41:30.686 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:41:30.686 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:41:30.686 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:41:30.689 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_22.status 2024-09-11T06:41:30.690 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-09-11T06:41:30.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror image_22 2024-09-11T06:41:30.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:41:30.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:41:30.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_22 2024-09-11T06:41:30.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror image_22 '..*' 2024-09-11T06:41:30.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:41:30.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:41:30.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_22 2024-09-11T06:41:30.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:41:30.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-image_22.status 2024-09-11T06:41:30.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-image_22.status 2024-09-11T06:41:30.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:41:30.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_22.status 2024-09-11T06:41:30.693 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_22 2024-09-11T06:41:30.693 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_22.status 2024-09-11T06:41:30.741 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 1 2024-09-11T06:41:30.741 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 1 2024-09-11T06:41:30.741 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:41:30.741 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:41:30.741 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:41:30.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_22.status 2024-09-11T06:41:30.746 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-09-11T06:41:30.746 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=3, entry_tid=1' 2024-09-11T06:41:30.746 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:41:30.746 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_snap_present cluster1 mirror image_22 snap 2024-09-11T06:41:30.746 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:41:30.746 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:30.747 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_22 2024-09-11T06:41:30.747 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:41:30.747 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:41:30.747 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T06:41:30.747 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:41:31.747 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 info mirror/image_22@snap 2024-09-11T06:41:31.813 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:rbd image 'image_22': 2024-09-11T06:41:31.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: size 128 MiB in 32 objects 2024-09-11T06:41:31.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: order 22 (4 MiB objects) 2024-09-11T06:41:31.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: snapshot_count: 1 2024-09-11T06:41:31.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: id: 10a97ebc3aa3 2024-09-11T06:41:31.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: block_name_prefix: rbd_data.10a97ebc3aa3 2024-09-11T06:41:31.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: format: 2 2024-09-11T06:41:31.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: features: layering, exclusive-lock, journaling 2024-09-11T06:41:31.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: op_features: 2024-09-11T06:41:31.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: flags: 2024-09-11T06:41:31.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: create_timestamp: Wed Sep 11 06:34:07 2024 2024-09-11T06:41:31.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: access_timestamp: Wed Sep 11 06:34:07 2024 2024-09-11T06:41:31.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: modify_timestamp: Wed Sep 11 06:35:08 2024 2024-09-11T06:41:31.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: protected: False 2024-09-11T06:41:31.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: journal: 10a97ebc3aa3 2024-09-11T06:41:31.815 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring state: enabled 2024-09-11T06:41:31.815 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring mode: journal 2024-09-11T06:41:31.815 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring global id: 9d9b92f4-73e2-4a36-954e-2791b02f6a60 2024-09-11T06:41:31.815 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring primary: false 2024-09-11T06:41:31.818 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:41:31.818 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ compare_image_snaps mirror image_22 snap 2024-09-11T06:41:31.818 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:31.818 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_22 2024-09-11T06:41:31.819 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:41:31.819 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local ret=0 2024-09-11T06:41:31.819 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local rmt_export=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_22.export 2024-09-11T06:41:31.819 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loc_export=/tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_22.export 2024-09-11T06:41:31.819 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_22.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_22.export 2024-09-11T06:41:31.819 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror export image_22@snap /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_22.export 2024-09-11T06:41:32.005 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T06:41:32.013 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 -p mirror export image_22@snap /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_22.export 2024-09-11T06:41:32.386 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T06:41:32.398 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmp /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_22.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_22.export 2024-09-11T06:41:32.477 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_22.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_22.export 2024-09-11T06:41:32.521 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:41:32.521 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:41:32.521 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_23 2024-09-11T06:41:32.521 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_snap cluster2 mirror image_23 snap 2024-09-11T06:41:32.521 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:41:32.521 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:32.521 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_23 2024-09-11T06:41:32.521 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:41:32.521 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror snap create image_23@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T06:41:33.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_started cluster1 mirror image_23 2024-09-11T06:41:33.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:41:33.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:33.480 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_23 2024-09-11T06:41:33.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_state cluster1 mirror image_23 started 2024-09-11T06:41:33.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:41:33.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:33.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_23 2024-09-11T06:41:33.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=started 2024-09-11T06:41:33.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:41:33.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:41:33.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:41:34.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_replay_state cluster1 mirror image_23 started 2024-09-11T06:41:34.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:41:34.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:34.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_23 2024-09-11T06:41:34.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=started 2024-09-11T06:41:34.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local status_result 2024-09-11T06:41:34.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=stopped 2024-09-11T06:41:34.483 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_23 2024-09-11T06:41:34.483 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_instance=cluster1 2024-09-11T06:41:34.483 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:41:34.483 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:41:34.483 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance=cluster1 2024-09-11T06:41:34.483 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local loop_instance 2024-09-11T06:41:34.483 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:41:34.484 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sleep 0 2024-09-11T06:41:34.484 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -i state 2024-09-11T06:41:34.484 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:41:34.484 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ seq 0 1 2024-09-11T06:41:34.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:41:34.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_23 2024-09-11T06:41:34.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1:0 2024-09-11T06:41:34.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:41:34.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:41:34.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:41:34.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:41:34.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=cluster 2024-09-11T06:41:34.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:41:34.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:41:34.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:41:34.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:41:34.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval cluster=cluster1 2024-09-11T06:41:34.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:41:34.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:41:34.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:41:34.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:41:34.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local local_cluster=cluster1:0 2024-09-11T06:41:34.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster=cluster1 2024-09-11T06:41:34.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance 2024-09-11T06:41:34.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:41:34.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local val=cluster1:0 2024-09-11T06:41:34.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T06:41:34.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance_var_name=instance 2024-09-11T06:41:34.490 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:41:34.490 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:41:34.490 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T06:41:34.490 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval local_cluster=cluster1 2024-09-11T06:41:34.490 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ local_cluster=cluster1 2024-09-11T06:41:34.490 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval instance=0 2024-09-11T06:41:34.490 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ instance=0 2024-09-11T06:41:34.490 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:41:34.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:41:34.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:41:34.513 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:41:34.513 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_23 2024-09-11T06:41:34.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ return 0 2024-09-11T06:41:34.649 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T06:41:34.649 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ echo ' "state": "Replaying"' 2024-09-11T06:41:34.650 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ grep -i Replaying 2024-09-11T06:41:34.651 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: "state": "Replaying" 2024-09-11T06:41:34.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_state=started 2024-09-11T06:41:34.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test started = started 2024-09-11T06:41:34.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:41:34.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_23 2024-09-11T06:41:34.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:41:34.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:41:34.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:34.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_23 2024-09-11T06:41:34.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' journal = journal ']' 2024-09-11T06:41:34.654 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_23 2024-09-11T06:41:34.654 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:41:34.654 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:41:34.654 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:34.654 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_23 2024-09-11T06:41:34.654 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-11T06:41:34.654 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-11T06:41:34.654 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:41:34.654 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:41:34.654 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:41:34.855 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror image_23 2024-09-11T06:41:34.855 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:41:34.855 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:34.855 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_23 2024-09-11T06:41:34.855 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:41:34.855 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n image_23 ']' 2024-09-11T06:41:34.855 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/image_23' 2024-09-11T06:41:34.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_23 2024-09-11T06:41:34.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:41:34.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:41:34.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:41:34.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:41:34.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:41:34.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:41:34.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:41:34.857 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:41:34.857 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:41:34.859 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:41:34.859 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_23 2024-09-11T06:41:34.859 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:41:34.860 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:41:34.860 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:41:34.860 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:41:34.860 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:41:34.860 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:41:34.860 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:41:34.860 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:41:34.860 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:41:34.860 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:41:34.860 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:41:34.860 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:41:34.860 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:41:34.860 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:41:34.860 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:41:34.862 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:41:34.862 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:41:34.862 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:41:34.862 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:41:34.862 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:41:34.862 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:41:34.862 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:41:34.862 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:41:34.862 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:41:34.862 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:41:34.862 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:41:34.862 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:41:34.862 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:41:34.863 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:41:34.863 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:41:34.891 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:41:34.891 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:41:34.891 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:41:34.891 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_23 2024-09-11T06:41:35.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:41:35.035 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror image_23 2024-09-11T06:41:35.035 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:41:35.035 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:41:35.035 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_23 2024-09-11T06:41:35.035 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror image_23 '' 2024-09-11T06:41:35.035 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:41:35.036 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:41:35.036 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_23 2024-09-11T06:41:35.036 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:41:35.036 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-image_23.status 2024-09-11T06:41:35.036 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-image_23.status 2024-09-11T06:41:35.036 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:41:35.038 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_23.status 2024-09-11T06:41:35.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_23 2024-09-11T06:41:35.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_23.status 2024-09-11T06:41:35.088 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 2 2024-09-11T06:41:35.088 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 2 2024-09-11T06:41:35.088 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:41:35.088 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=3, entry_tid=1], [object_number=8, tag_tid=3, entry_tid=0], [object_number=11, tag_tid=2, entry_tid=3], [object_number=10, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:41:35.088 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=9, tag_tid=3, entry_tid=1], [object_number=8, tag_tid=3, entry_tid=0], [object_number=11, tag_tid=2, entry_tid=3], [object_number=10, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:41:35.091 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_23.status 2024-09-11T06:41:35.092 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=9, tag_tid=3, entry_tid=1' 2024-09-11T06:41:35.093 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror image_23 2024-09-11T06:41:35.093 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:41:35.093 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:41:35.093 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_23 2024-09-11T06:41:35.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror image_23 '..*' 2024-09-11T06:41:35.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:41:35.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:41:35.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_23 2024-09-11T06:41:35.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:41:35.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-image_23.status 2024-09-11T06:41:35.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-image_23.status 2024-09-11T06:41:35.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:41:35.095 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_23.status 2024-09-11T06:41:35.095 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_23 2024-09-11T06:41:35.095 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_23.status 2024-09-11T06:41:35.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 2 2024-09-11T06:41:35.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 2 2024-09-11T06:41:35.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:41:35.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=3, entry_tid=1], [object_number=8, tag_tid=3, entry_tid=0], [object_number=11, tag_tid=2, entry_tid=3], [object_number=10, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:41:35.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=9, tag_tid=3, entry_tid=1], [object_number=8, tag_tid=3, entry_tid=0], [object_number=11, tag_tid=2, entry_tid=3], [object_number=10, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:41:35.143 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_23.status 2024-09-11T06:41:35.146 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=9, tag_tid=3, entry_tid=1' 2024-09-11T06:41:35.146 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=9, tag_tid=3, entry_tid=1' -a 'object_number=9, tag_tid=3, entry_tid=1' = 'object_number=9, tag_tid=3, entry_tid=1' 2024-09-11T06:41:35.146 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:41:35.146 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_snap_present cluster1 mirror image_23 snap 2024-09-11T06:41:35.146 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:41:35.146 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:35.146 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_23 2024-09-11T06:41:35.146 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:41:35.146 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:41:35.146 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T06:41:35.146 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:41:36.146 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 info mirror/image_23@snap 2024-09-11T06:41:36.211 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:rbd image 'image_23': 2024-09-11T06:41:36.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: size 128 MiB in 32 objects 2024-09-11T06:41:36.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: order 22 (4 MiB objects) 2024-09-11T06:41:36.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: snapshot_count: 1 2024-09-11T06:41:36.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: id: 10a629868ddf 2024-09-11T06:41:36.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: block_name_prefix: rbd_data.10a629868ddf 2024-09-11T06:41:36.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: format: 2 2024-09-11T06:41:36.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: features: layering, exclusive-lock, journaling 2024-09-11T06:41:36.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: op_features: 2024-09-11T06:41:36.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: flags: 2024-09-11T06:41:36.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: create_timestamp: Wed Sep 11 06:34:13 2024 2024-09-11T06:41:36.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: access_timestamp: Wed Sep 11 06:34:13 2024 2024-09-11T06:41:36.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: modify_timestamp: Wed Sep 11 06:37:08 2024 2024-09-11T06:41:36.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: protected: False 2024-09-11T06:41:36.213 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: journal: 10a629868ddf 2024-09-11T06:41:36.213 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring state: enabled 2024-09-11T06:41:36.213 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring mode: journal 2024-09-11T06:41:36.213 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring global id: bb9740c2-ccf7-4d3e-b493-99fe2d4789f0 2024-09-11T06:41:36.213 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring primary: false 2024-09-11T06:41:36.216 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:41:36.216 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ compare_image_snaps mirror image_23 snap 2024-09-11T06:41:36.216 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:36.216 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_23 2024-09-11T06:41:36.216 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:41:36.216 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local ret=0 2024-09-11T06:41:36.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local rmt_export=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_23.export 2024-09-11T06:41:36.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loc_export=/tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_23.export 2024-09-11T06:41:36.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_23.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_23.export 2024-09-11T06:41:36.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror export image_23@snap /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_23.export 2024-09-11T06:41:36.424 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T06:41:36.435 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 -p mirror export image_23@snap /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_23.export 2024-09-11T06:41:36.837 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T06:41:36.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmp /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_23.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_23.export 2024-09-11T06:41:36.925 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_23.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_23.export 2024-09-11T06:41:36.967 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:41:36.967 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:41:36.967 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_24 2024-09-11T06:41:36.967 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_snap cluster2 mirror image_24 snap 2024-09-11T06:41:36.968 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:41:36.968 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:36.968 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_24 2024-09-11T06:41:36.968 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:41:36.968 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror snap create image_24@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T06:41:37.617 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_started cluster1 mirror image_24 2024-09-11T06:41:37.617 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:41:37.617 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:37.617 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_24 2024-09-11T06:41:37.617 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_state cluster1 mirror image_24 started 2024-09-11T06:41:37.617 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:41:37.617 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:37.617 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_24 2024-09-11T06:41:37.617 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=started 2024-09-11T06:41:37.618 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:41:37.618 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:41:37.618 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:41:38.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_replay_state cluster1 mirror image_24 started 2024-09-11T06:41:38.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:41:38.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:38.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_24 2024-09-11T06:41:38.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=started 2024-09-11T06:41:38.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local status_result 2024-09-11T06:41:38.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=stopped 2024-09-11T06:41:38.621 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_24 2024-09-11T06:41:38.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -i state 2024-09-11T06:41:38.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_instance=cluster1 2024-09-11T06:41:38.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:41:38.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:41:38.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance=cluster1 2024-09-11T06:41:38.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local loop_instance 2024-09-11T06:41:38.623 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:41:38.623 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sleep 0 2024-09-11T06:41:38.626 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:41:38.626 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ seq 0 1 2024-09-11T06:41:38.626 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:41:38.626 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_24 2024-09-11T06:41:38.627 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1:0 2024-09-11T06:41:38.627 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:41:38.627 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:41:38.627 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:41:38.627 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:41:38.627 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=cluster 2024-09-11T06:41:38.627 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:41:38.627 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:41:38.627 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:41:38.627 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:41:38.627 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval cluster=cluster1 2024-09-11T06:41:38.627 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:41:38.627 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:41:38.628 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:41:38.628 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:41:38.628 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local local_cluster=cluster1:0 2024-09-11T06:41:38.628 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster=cluster1 2024-09-11T06:41:38.628 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance 2024-09-11T06:41:38.628 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:41:38.628 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local val=cluster1:0 2024-09-11T06:41:38.628 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T06:41:38.628 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance_var_name=instance 2024-09-11T06:41:38.629 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:41:38.629 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:41:38.629 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T06:41:38.629 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval local_cluster=cluster1 2024-09-11T06:41:38.629 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ local_cluster=cluster1 2024-09-11T06:41:38.629 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval instance=0 2024-09-11T06:41:38.629 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ instance=0 2024-09-11T06:41:38.629 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:41:38.652 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:41:38.652 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:41:38.652 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:41:38.652 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_24 2024-09-11T06:41:38.783 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:Invalid command: unused arguments: ['mirror/image_24'] 2024-09-11T06:41:38.783 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T06:41:38.783 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:admin_socket: invalid command 2024-09-11T06:41:38.787 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:41:38.787 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_24 2024-09-11T06:41:38.787 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1:1 2024-09-11T06:41:38.788 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:41:38.788 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:41:38.788 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-11T06:41:38.788 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:1 2024-09-11T06:41:38.788 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=cluster 2024-09-11T06:41:38.788 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:41:38.788 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:41:38.788 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=1 2024-09-11T06:41:38.788 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T06:41:38.788 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval cluster=cluster1 2024-09-11T06:41:38.789 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:41:38.789 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=1 2024-09-11T06:41:38.789 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=1 2024-09-11T06:41:38.789 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-11T06:41:38.789 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local local_cluster=cluster1:1 2024-09-11T06:41:38.789 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster=cluster1 2024-09-11T06:41:38.789 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance 2024-09-11T06:41:38.789 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T06:41:38.789 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local val=cluster1:1 2024-09-11T06:41:38.789 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T06:41:38.789 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance_var_name=instance 2024-09-11T06:41:38.790 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:41:38.790 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=1 2024-09-11T06:41:38.790 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-11T06:41:38.790 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval local_cluster=cluster1 2024-09-11T06:41:38.790 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ local_cluster=cluster1 2024-09-11T06:41:38.790 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval instance=1 2024-09-11T06:41:38.790 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ instance=1 2024-09-11T06:41:38.790 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T06:41:38.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:41:38.806 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:41:38.806 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:41:38.806 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_24 2024-09-11T06:41:38.935 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ return 0 2024-09-11T06:41:38.936 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T06:41:38.937 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ echo ' "state": "Replaying"' 2024-09-11T06:41:38.937 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ grep -i Replaying 2024-09-11T06:41:38.939 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: "state": "Replaying" 2024-09-11T06:41:38.941 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_state=started 2024-09-11T06:41:38.941 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test started = started 2024-09-11T06:41:38.941 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:41:38.941 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_24 2024-09-11T06:41:38.941 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:41:38.941 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:41:38.941 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:38.941 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_24 2024-09-11T06:41:38.941 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' journal = journal ']' 2024-09-11T06:41:38.941 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_24 2024-09-11T06:41:38.942 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:41:38.942 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:41:38.942 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:38.942 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_24 2024-09-11T06:41:38.942 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-11T06:41:38.942 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-11T06:41:38.942 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:41:38.942 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:41:38.942 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:41:39.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror image_24 2024-09-11T06:41:39.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:41:39.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:39.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_24 2024-09-11T06:41:39.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:41:39.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n image_24 ']' 2024-09-11T06:41:39.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/image_24' 2024-09-11T06:41:39.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_24 2024-09-11T06:41:39.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:41:39.143 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:41:39.143 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:41:39.143 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:41:39.143 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:41:39.143 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:41:39.143 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:41:39.143 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:41:39.144 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:41:39.148 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:41:39.148 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_24 2024-09-11T06:41:39.148 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:41:39.148 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:41:39.148 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:41:39.148 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:41:39.149 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:41:39.149 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:41:39.149 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:41:39.149 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:41:39.149 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:41:39.149 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:41:39.150 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:41:39.150 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:41:39.150 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:41:39.150 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:41:39.150 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:41:39.150 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:41:39.150 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:41:39.150 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:41:39.150 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:41:39.150 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:41:39.150 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:41:39.151 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:41:39.151 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:41:39.151 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:41:39.151 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:41:39.151 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:41:39.151 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:41:39.151 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:41:39.151 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:41:39.151 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:41:39.173 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:41:39.173 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:41:39.173 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:41:39.174 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_24 2024-09-11T06:41:39.298 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:Invalid command: unused arguments: ['mirror/image_24'] 2024-09-11T06:41:39.298 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd mirror flush : flush rbd mirror 2024-09-11T06:41:39.298 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:admin_socket: invalid command 2024-09-11T06:41:39.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:41:39.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_24 2024-09-11T06:41:39.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:1 2024-09-11T06:41:39.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:41:39.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:41:39.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-11T06:41:39.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:1 2024-09-11T06:41:39.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:41:39.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:41:39.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:41:39.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=1 2024-09-11T06:41:39.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-11T06:41:39.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:41:39.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:41:39.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=1 2024-09-11T06:41:39.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=1 2024-09-11T06:41:39.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-11T06:41:39.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:1 2024-09-11T06:41:39.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:41:39.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:41:39.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T06:41:39.305 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:1 2024-09-11T06:41:39.305 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:41:39.305 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:41:39.305 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:41:39.305 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=1 2024-09-11T06:41:39.305 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T06:41:39.305 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:41:39.305 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:41:39.305 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=1 2024-09-11T06:41:39.306 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=1 2024-09-11T06:41:39.306 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T06:41:39.326 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:41:39.326 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:41:39.326 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:41:39.326 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_24 2024-09-11T06:41:39.463 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:41:39.464 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror image_24 2024-09-11T06:41:39.464 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:41:39.464 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:41:39.464 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_24 2024-09-11T06:41:39.464 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror image_24 '' 2024-09-11T06:41:39.464 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:41:39.465 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:41:39.465 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_24 2024-09-11T06:41:39.465 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:41:39.465 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-image_24.status 2024-09-11T06:41:39.466 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-image_24.status 2024-09-11T06:41:39.466 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:41:39.466 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_24.status 2024-09-11T06:41:39.466 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_24.status 2024-09-11T06:41:39.467 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_24 2024-09-11T06:41:39.518 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 1 2024-09-11T06:41:39.519 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 1 2024-09-11T06:41:39.519 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:41:39.519 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:41:39.519 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:41:39.522 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_24.status 2024-09-11T06:41:39.523 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-09-11T06:41:39.524 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror image_24 2024-09-11T06:41:39.524 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:41:39.525 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:41:39.525 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_24 2024-09-11T06:41:39.525 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror image_24 '..*' 2024-09-11T06:41:39.525 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:41:39.525 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:41:39.525 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_24 2024-09-11T06:41:39.525 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:41:39.525 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-image_24.status 2024-09-11T06:41:39.525 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-image_24.status 2024-09-11T06:41:39.525 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:41:39.526 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_24.status 2024-09-11T06:41:39.526 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_24 2024-09-11T06:41:39.526 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_24.status 2024-09-11T06:41:39.572 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 1 2024-09-11T06:41:39.572 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 1 2024-09-11T06:41:39.572 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:41:39.572 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:41:39.572 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:41:39.575 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_24.status 2024-09-11T06:41:39.577 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-09-11T06:41:39.578 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=3, entry_tid=1' 2024-09-11T06:41:39.578 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:41:39.578 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_snap_present cluster1 mirror image_24 snap 2024-09-11T06:41:39.578 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:41:39.578 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:39.578 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_24 2024-09-11T06:41:39.578 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:41:39.578 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:41:39.578 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T06:41:39.578 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:41:40.578 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 info mirror/image_24@snap 2024-09-11T06:41:40.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:rbd image 'image_24': 2024-09-11T06:41:40.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: size 128 MiB in 32 objects 2024-09-11T06:41:40.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: order 22 (4 MiB objects) 2024-09-11T06:41:40.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: snapshot_count: 1 2024-09-11T06:41:40.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: id: 10a9167b40d8 2024-09-11T06:41:40.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: block_name_prefix: rbd_data.10a9167b40d8 2024-09-11T06:41:40.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: format: 2 2024-09-11T06:41:40.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: features: layering, exclusive-lock, journaling 2024-09-11T06:41:40.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: op_features: 2024-09-11T06:41:40.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: flags: 2024-09-11T06:41:40.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: create_timestamp: Wed Sep 11 06:34:22 2024 2024-09-11T06:41:40.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: access_timestamp: Wed Sep 11 06:34:22 2024 2024-09-11T06:41:40.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: modify_timestamp: Wed Sep 11 06:35:44 2024 2024-09-11T06:41:40.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: protected: False 2024-09-11T06:41:40.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: journal: 10a9167b40d8 2024-09-11T06:41:40.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring state: enabled 2024-09-11T06:41:40.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring mode: journal 2024-09-11T06:41:40.645 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring global id: 5f0f70a8-4068-4566-9329-9c4c3f5815fc 2024-09-11T06:41:40.645 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring primary: false 2024-09-11T06:41:40.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:41:40.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ compare_image_snaps mirror image_24 snap 2024-09-11T06:41:40.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:40.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_24 2024-09-11T06:41:40.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:41:40.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local ret=0 2024-09-11T06:41:40.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local rmt_export=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_24.export 2024-09-11T06:41:40.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loc_export=/tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_24.export 2024-09-11T06:41:40.649 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_24.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_24.export 2024-09-11T06:41:40.649 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror export image_24@snap /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_24.export 2024-09-11T06:41:40.839 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T06:41:40.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 -p mirror export image_24@snap /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_24.export 2024-09-11T06:41:41.211 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T06:41:41.228 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmp /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_24.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_24.export 2024-09-11T06:41:41.305 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_24.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_24.export 2024-09-11T06:41:41.346 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:41:41.346 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:41:41.346 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_25 2024-09-11T06:41:41.347 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_snap cluster2 mirror image_25 snap 2024-09-11T06:41:41.347 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:41:41.347 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:41.347 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_25 2024-09-11T06:41:41.347 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:41:41.347 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror snap create image_25@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T06:41:42.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_started cluster1 mirror image_25 2024-09-11T06:41:42.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:41:42.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:42.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_25 2024-09-11T06:41:42.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_state cluster1 mirror image_25 started 2024-09-11T06:41:42.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:41:42.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:42.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_25 2024-09-11T06:41:42.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=started 2024-09-11T06:41:42.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:41:42.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:41:42.621 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:41:43.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_replay_state cluster1 mirror image_25 started 2024-09-11T06:41:43.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:41:43.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:43.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_25 2024-09-11T06:41:43.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=started 2024-09-11T06:41:43.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local status_result 2024-09-11T06:41:43.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=stopped 2024-09-11T06:41:43.623 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_25 2024-09-11T06:41:43.623 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_instance=cluster1 2024-09-11T06:41:43.623 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -i state 2024-09-11T06:41:43.623 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:41:43.623 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:41:43.623 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance=cluster1 2024-09-11T06:41:43.623 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local loop_instance 2024-09-11T06:41:43.624 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:41:43.624 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sleep 0 2024-09-11T06:41:43.624 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:41:43.625 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ seq 0 1 2024-09-11T06:41:43.627 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:41:43.627 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_25 2024-09-11T06:41:43.627 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1:0 2024-09-11T06:41:43.627 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:41:43.627 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:41:43.627 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:41:43.627 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:41:43.627 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=cluster 2024-09-11T06:41:43.627 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:41:43.628 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:41:43.628 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:41:43.628 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:41:43.628 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval cluster=cluster1 2024-09-11T06:41:43.628 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:41:43.628 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:41:43.628 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:41:43.628 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:41:43.628 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local local_cluster=cluster1:0 2024-09-11T06:41:43.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster=cluster1 2024-09-11T06:41:43.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance 2024-09-11T06:41:43.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:41:43.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local val=cluster1:0 2024-09-11T06:41:43.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T06:41:43.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance_var_name=instance 2024-09-11T06:41:43.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:41:43.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:41:43.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T06:41:43.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval local_cluster=cluster1 2024-09-11T06:41:43.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ local_cluster=cluster1 2024-09-11T06:41:43.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval instance=0 2024-09-11T06:41:43.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ instance=0 2024-09-11T06:41:43.631 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:41:43.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:41:43.649 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:41:43.649 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:41:43.649 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_25 2024-09-11T06:41:43.788 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ return 0 2024-09-11T06:41:43.789 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T06:41:43.789 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ echo ' "state": "Replaying"' 2024-09-11T06:41:43.789 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ grep -i Replaying 2024-09-11T06:41:43.790 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: "state": "Replaying" 2024-09-11T06:41:43.791 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_state=started 2024-09-11T06:41:43.791 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test started = started 2024-09-11T06:41:43.791 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:41:43.792 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_25 2024-09-11T06:41:43.792 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:41:43.792 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:41:43.792 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:43.792 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_25 2024-09-11T06:41:43.792 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' journal = journal ']' 2024-09-11T06:41:43.792 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_25 2024-09-11T06:41:43.792 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:41:43.792 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:41:43.792 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:43.792 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_25 2024-09-11T06:41:43.792 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-11T06:41:43.792 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-11T06:41:43.793 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:41:43.793 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:41:43.793 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:41:43.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror image_25 2024-09-11T06:41:43.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:41:43.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:43.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_25 2024-09-11T06:41:43.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:41:43.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n image_25 ']' 2024-09-11T06:41:43.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/image_25' 2024-09-11T06:41:43.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_25 2024-09-11T06:41:43.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:41:43.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:41:43.994 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:41:43.994 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:41:43.994 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:41:43.994 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:41:43.994 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:41:43.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:41:43.999 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:41:43.999 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:41:43.999 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_25 2024-09-11T06:41:43.999 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:41:43.999 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:41:43.999 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:41:43.999 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:41:43.999 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:41:43.999 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:41:43.999 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:41:43.999 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:41:43.999 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:41:43.999 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:41:43.999 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:41:43.999 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:41:44.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:41:44.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:41:44.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:41:44.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:41:44.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:41:44.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:41:44.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:41:44.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:41:44.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:41:44.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:41:44.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:41:44.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:41:44.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:41:44.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:41:44.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:41:44.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:41:44.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:41:44.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:41:44.024 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:41:44.025 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:41:44.025 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:41:44.025 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_25 2024-09-11T06:41:44.162 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:41:44.163 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror image_25 2024-09-11T06:41:44.163 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:41:44.164 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:41:44.164 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_25 2024-09-11T06:41:44.164 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror image_25 '' 2024-09-11T06:41:44.164 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:41:44.164 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:41:44.164 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_25 2024-09-11T06:41:44.164 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:41:44.164 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-image_25.status 2024-09-11T06:41:44.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-image_25.status 2024-09-11T06:41:44.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:41:44.167 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_25.status 2024-09-11T06:41:44.167 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_25 2024-09-11T06:41:44.167 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_25.status 2024-09-11T06:41:44.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 1 2024-09-11T06:41:44.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 1 2024-09-11T06:41:44.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:41:44.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:41:44.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:41:44.222 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_25.status 2024-09-11T06:41:44.224 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-09-11T06:41:44.225 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror image_25 2024-09-11T06:41:44.225 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:41:44.225 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:41:44.225 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_25 2024-09-11T06:41:44.225 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror image_25 '..*' 2024-09-11T06:41:44.225 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:41:44.225 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:41:44.226 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_25 2024-09-11T06:41:44.226 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:41:44.226 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-image_25.status 2024-09-11T06:41:44.226 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-image_25.status 2024-09-11T06:41:44.226 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:41:44.226 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_25.status 2024-09-11T06:41:44.226 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_25 2024-09-11T06:41:44.227 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_25.status 2024-09-11T06:41:44.272 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 1 2024-09-11T06:41:44.272 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 1 2024-09-11T06:41:44.272 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:41:44.272 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:41:44.272 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:41:44.275 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_25.status 2024-09-11T06:41:44.277 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-09-11T06:41:44.277 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=3, entry_tid=1' 2024-09-11T06:41:44.278 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:41:44.278 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_snap_present cluster1 mirror image_25 snap 2024-09-11T06:41:44.278 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:41:44.278 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:44.278 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_25 2024-09-11T06:41:44.278 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:41:44.278 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:41:44.278 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T06:41:44.278 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:41:45.278 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 info mirror/image_25@snap 2024-09-11T06:41:45.344 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:rbd image 'image_25': 2024-09-11T06:41:45.344 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: size 128 MiB in 32 objects 2024-09-11T06:41:45.344 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: order 22 (4 MiB objects) 2024-09-11T06:41:45.344 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: snapshot_count: 1 2024-09-11T06:41:45.345 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: id: 10a6db583b94 2024-09-11T06:41:45.345 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: block_name_prefix: rbd_data.10a6db583b94 2024-09-11T06:41:45.345 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: format: 2 2024-09-11T06:41:45.345 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: features: layering, exclusive-lock, journaling 2024-09-11T06:41:45.345 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: op_features: 2024-09-11T06:41:45.345 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: flags: 2024-09-11T06:41:45.345 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: create_timestamp: Wed Sep 11 06:34:31 2024 2024-09-11T06:41:45.345 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: access_timestamp: Wed Sep 11 06:34:31 2024 2024-09-11T06:41:45.345 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: modify_timestamp: Wed Sep 11 06:36:22 2024 2024-09-11T06:41:45.345 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: protected: False 2024-09-11T06:41:45.345 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: journal: 10a6db583b94 2024-09-11T06:41:45.345 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring state: enabled 2024-09-11T06:41:45.345 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring mode: journal 2024-09-11T06:41:45.345 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring global id: 80611705-e74b-43a1-9ab4-2cc86ae82bc8 2024-09-11T06:41:45.346 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring primary: false 2024-09-11T06:41:45.349 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:41:45.349 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ compare_image_snaps mirror image_25 snap 2024-09-11T06:41:45.349 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:45.349 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_25 2024-09-11T06:41:45.349 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:41:45.349 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local ret=0 2024-09-11T06:41:45.349 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local rmt_export=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_25.export 2024-09-11T06:41:45.349 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loc_export=/tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_25.export 2024-09-11T06:41:45.350 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_25.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_25.export 2024-09-11T06:41:45.350 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror export image_25@snap /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_25.export 2024-09-11T06:41:45.520 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T06:41:45.527 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 -p mirror export image_25@snap /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_25.export 2024-09-11T06:41:45.881 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T06:41:45.893 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmp /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_25.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_25.export 2024-09-11T06:41:45.971 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_25.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_25.export 2024-09-11T06:41:46.014 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:41:46.014 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:41:46.015 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_26 2024-09-11T06:41:46.015 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_snap cluster2 mirror image_26 snap 2024-09-11T06:41:46.015 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:41:46.015 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:46.015 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_26 2024-09-11T06:41:46.015 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:41:46.015 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror snap create image_26@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T06:41:46.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_started cluster1 mirror image_26 2024-09-11T06:41:46.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:41:46.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:46.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_26 2024-09-11T06:41:46.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_state cluster1 mirror image_26 started 2024-09-11T06:41:46.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:41:46.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:46.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_26 2024-09-11T06:41:46.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=started 2024-09-11T06:41:46.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:41:46.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:41:46.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:41:47.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_replay_state cluster1 mirror image_26 started 2024-09-11T06:41:47.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:41:47.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:47.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_26 2024-09-11T06:41:47.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=started 2024-09-11T06:41:47.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local status_result 2024-09-11T06:41:47.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=stopped 2024-09-11T06:41:47.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_26 2024-09-11T06:41:47.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -i state 2024-09-11T06:41:47.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_instance=cluster1 2024-09-11T06:41:47.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:41:47.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:41:47.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance=cluster1 2024-09-11T06:41:47.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local loop_instance 2024-09-11T06:41:47.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:41:47.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sleep 0 2024-09-11T06:41:47.645 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:41:47.645 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ seq 0 1 2024-09-11T06:41:47.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:41:47.649 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_26 2024-09-11T06:41:47.649 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1:0 2024-09-11T06:41:47.649 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:41:47.649 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:41:47.649 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:41:47.649 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:41:47.649 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=cluster 2024-09-11T06:41:47.649 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:41:47.649 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:41:47.649 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:41:47.649 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:41:47.649 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval cluster=cluster1 2024-09-11T06:41:47.650 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:41:47.650 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:41:47.650 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:41:47.650 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:41:47.650 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local local_cluster=cluster1:0 2024-09-11T06:41:47.650 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster=cluster1 2024-09-11T06:41:47.650 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance 2024-09-11T06:41:47.650 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:41:47.650 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local val=cluster1:0 2024-09-11T06:41:47.650 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T06:41:47.650 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance_var_name=instance 2024-09-11T06:41:47.650 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:41:47.650 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:41:47.650 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T06:41:47.650 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval local_cluster=cluster1 2024-09-11T06:41:47.651 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ local_cluster=cluster1 2024-09-11T06:41:47.651 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval instance=0 2024-09-11T06:41:47.651 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ instance=0 2024-09-11T06:41:47.651 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:41:47.671 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:41:47.671 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:41:47.671 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:41:47.671 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_26 2024-09-11T06:41:47.803 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:Invalid command: unused arguments: ['mirror/image_26'] 2024-09-11T06:41:47.803 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T06:41:47.803 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:admin_socket: invalid command 2024-09-11T06:41:47.808 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:41:47.808 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_26 2024-09-11T06:41:47.808 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1:1 2024-09-11T06:41:47.808 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:41:47.808 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:41:47.808 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-11T06:41:47.808 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:1 2024-09-11T06:41:47.808 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=cluster 2024-09-11T06:41:47.808 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:41:47.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:41:47.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=1 2024-09-11T06:41:47.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T06:41:47.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval cluster=cluster1 2024-09-11T06:41:47.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:41:47.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=1 2024-09-11T06:41:47.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=1 2024-09-11T06:41:47.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-11T06:41:47.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local local_cluster=cluster1:1 2024-09-11T06:41:47.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster=cluster1 2024-09-11T06:41:47.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance 2024-09-11T06:41:47.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T06:41:47.810 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local val=cluster1:1 2024-09-11T06:41:47.810 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T06:41:47.810 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance_var_name=instance 2024-09-11T06:41:47.810 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:41:47.810 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=1 2024-09-11T06:41:47.810 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-11T06:41:47.810 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval local_cluster=cluster1 2024-09-11T06:41:47.810 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ local_cluster=cluster1 2024-09-11T06:41:47.810 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval instance=1 2024-09-11T06:41:47.810 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ instance=1 2024-09-11T06:41:47.810 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T06:41:47.830 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:41:47.831 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:41:47.831 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:41:47.831 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_26 2024-09-11T06:41:47.961 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ return 0 2024-09-11T06:41:47.962 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T06:41:47.963 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ echo ' "state": "Replaying"' 2024-09-11T06:41:47.963 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ grep -i Replaying 2024-09-11T06:41:47.964 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: "state": "Replaying" 2024-09-11T06:41:47.966 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_state=started 2024-09-11T06:41:47.966 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test started = started 2024-09-11T06:41:47.967 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:41:47.967 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_26 2024-09-11T06:41:47.967 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:41:47.967 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:41:47.967 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:47.967 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_26 2024-09-11T06:41:47.967 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' journal = journal ']' 2024-09-11T06:41:47.967 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_26 2024-09-11T06:41:47.967 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:41:47.968 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:41:47.968 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:47.968 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_26 2024-09-11T06:41:47.968 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-11T06:41:47.968 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-11T06:41:47.968 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:41:47.968 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:41:47.968 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:41:48.167 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror image_26 2024-09-11T06:41:48.168 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:41:48.168 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:48.168 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_26 2024-09-11T06:41:48.168 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:41:48.168 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n image_26 ']' 2024-09-11T06:41:48.168 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/image_26' 2024-09-11T06:41:48.168 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_26 2024-09-11T06:41:48.168 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:41:48.168 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:41:48.168 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:41:48.168 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:41:48.169 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:41:48.169 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:41:48.169 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:41:48.169 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:41:48.169 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:41:48.172 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:41:48.172 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_26 2024-09-11T06:41:48.172 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:41:48.172 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:41:48.172 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:41:48.172 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:41:48.172 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:41:48.172 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:41:48.172 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:41:48.172 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:41:48.172 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:41:48.172 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:41:48.173 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:41:48.173 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:41:48.173 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:41:48.173 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:41:48.173 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:41:48.173 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:41:48.173 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:41:48.173 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:41:48.173 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:41:48.173 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:41:48.173 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:41:48.173 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:41:48.173 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:41:48.173 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:41:48.174 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:41:48.174 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:41:48.174 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:41:48.174 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:41:48.174 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:41:48.174 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:41:48.199 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:41:48.199 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:41:48.199 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:41:48.199 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_26 2024-09-11T06:41:48.321 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:Invalid command: unused arguments: ['mirror/image_26'] 2024-09-11T06:41:48.321 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd mirror flush : flush rbd mirror 2024-09-11T06:41:48.321 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:admin_socket: invalid command 2024-09-11T06:41:48.323 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:41:48.324 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_26 2024-09-11T06:41:48.324 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:1 2024-09-11T06:41:48.324 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:41:48.324 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:41:48.324 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-11T06:41:48.324 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:1 2024-09-11T06:41:48.324 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:41:48.324 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:41:48.324 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:41:48.324 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=1 2024-09-11T06:41:48.324 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-11T06:41:48.324 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:41:48.325 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:41:48.325 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=1 2024-09-11T06:41:48.325 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=1 2024-09-11T06:41:48.326 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-11T06:41:48.327 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:1 2024-09-11T06:41:48.327 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:41:48.327 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:41:48.327 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T06:41:48.327 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:1 2024-09-11T06:41:48.327 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:41:48.327 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:41:48.327 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:41:48.327 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=1 2024-09-11T06:41:48.327 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T06:41:48.327 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:41:48.327 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:41:48.327 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=1 2024-09-11T06:41:48.328 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=1 2024-09-11T06:41:48.328 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T06:41:48.350 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:41:48.350 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:41:48.350 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:41:48.350 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_26 2024-09-11T06:41:48.493 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:41:48.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror image_26 2024-09-11T06:41:48.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:41:48.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:41:48.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_26 2024-09-11T06:41:48.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror image_26 '' 2024-09-11T06:41:48.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:41:48.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:41:48.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_26 2024-09-11T06:41:48.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:41:48.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-image_26.status 2024-09-11T06:41:48.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-image_26.status 2024-09-11T06:41:48.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:41:48.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_26.status 2024-09-11T06:41:48.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_26 2024-09-11T06:41:48.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_26.status 2024-09-11T06:41:48.549 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 1 2024-09-11T06:41:48.549 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 1 2024-09-11T06:41:48.549 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:41:48.549 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:41:48.549 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:41:48.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_26.status 2024-09-11T06:41:48.553 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-09-11T06:41:48.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror image_26 2024-09-11T06:41:48.555 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:41:48.555 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:41:48.555 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_26 2024-09-11T06:41:48.555 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror image_26 '..*' 2024-09-11T06:41:48.555 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:41:48.555 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:41:48.555 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_26 2024-09-11T06:41:48.555 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:41:48.555 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-image_26.status 2024-09-11T06:41:48.555 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-image_26.status 2024-09-11T06:41:48.555 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:41:48.556 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_26.status 2024-09-11T06:41:48.556 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_26 2024-09-11T06:41:48.556 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_26.status 2024-09-11T06:41:48.602 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 1 2024-09-11T06:41:48.602 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 1 2024-09-11T06:41:48.602 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:41:48.602 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:41:48.602 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:41:48.605 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_26.status 2024-09-11T06:41:48.607 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-09-11T06:41:48.607 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=3, entry_tid=1' 2024-09-11T06:41:48.607 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:41:48.607 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_snap_present cluster1 mirror image_26 snap 2024-09-11T06:41:48.607 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:41:48.607 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:48.608 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_26 2024-09-11T06:41:48.608 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:41:48.608 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:41:48.608 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T06:41:48.608 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:41:49.608 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 info mirror/image_26@snap 2024-09-11T06:41:49.672 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:rbd image 'image_26': 2024-09-11T06:41:49.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: size 128 MiB in 32 objects 2024-09-11T06:41:49.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: order 22 (4 MiB objects) 2024-09-11T06:41:49.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: snapshot_count: 1 2024-09-11T06:41:49.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: id: 10a973dfade 2024-09-11T06:41:49.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: block_name_prefix: rbd_data.10a973dfade 2024-09-11T06:41:49.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: format: 2 2024-09-11T06:41:49.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: features: layering, exclusive-lock, journaling 2024-09-11T06:41:49.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: op_features: 2024-09-11T06:41:49.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: flags: 2024-09-11T06:41:49.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: create_timestamp: Wed Sep 11 06:34:41 2024 2024-09-11T06:41:49.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: access_timestamp: Wed Sep 11 06:34:41 2024 2024-09-11T06:41:49.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: modify_timestamp: Wed Sep 11 06:36:44 2024 2024-09-11T06:41:49.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: protected: False 2024-09-11T06:41:49.674 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: journal: 10a973dfade 2024-09-11T06:41:49.674 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring state: enabled 2024-09-11T06:41:49.674 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring mode: journal 2024-09-11T06:41:49.674 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring global id: 697b0370-2869-482d-b06b-fb73f8c993ed 2024-09-11T06:41:49.674 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring primary: false 2024-09-11T06:41:49.677 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:41:49.677 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ compare_image_snaps mirror image_26 snap 2024-09-11T06:41:49.677 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:49.677 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_26 2024-09-11T06:41:49.677 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:41:49.677 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local ret=0 2024-09-11T06:41:49.678 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local rmt_export=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_26.export 2024-09-11T06:41:49.678 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loc_export=/tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_26.export 2024-09-11T06:41:49.678 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_26.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_26.export 2024-09-11T06:41:49.678 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror export image_26@snap /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_26.export 2024-09-11T06:41:49.854 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T06:41:49.864 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 -p mirror export image_26@snap /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_26.export 2024-09-11T06:41:50.157 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T06:41:50.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmp /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_26.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_26.export 2024-09-11T06:41:50.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_26.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_26.export 2024-09-11T06:41:50.278 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:41:50.278 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:41:50.278 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_27 2024-09-11T06:41:50.278 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_snap cluster2 mirror image_27 snap 2024-09-11T06:41:50.278 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:41:50.278 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:50.279 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_27 2024-09-11T06:41:50.279 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:41:50.279 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror snap create image_27@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T06:41:50.654 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_started cluster1 mirror image_27 2024-09-11T06:41:50.654 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:41:50.654 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:50.654 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_27 2024-09-11T06:41:50.654 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_state cluster1 mirror image_27 started 2024-09-11T06:41:50.654 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:41:50.654 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:50.654 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_27 2024-09-11T06:41:50.654 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=started 2024-09-11T06:41:50.654 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:41:50.654 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:41:50.655 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:41:51.655 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_replay_state cluster1 mirror image_27 started 2024-09-11T06:41:51.655 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:41:51.656 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:51.656 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_27 2024-09-11T06:41:51.656 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=started 2024-09-11T06:41:51.656 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local status_result 2024-09-11T06:41:51.656 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=stopped 2024-09-11T06:41:51.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_27 2024-09-11T06:41:51.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -i state 2024-09-11T06:41:51.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_instance=cluster1 2024-09-11T06:41:51.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:41:51.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:41:51.658 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance=cluster1 2024-09-11T06:41:51.658 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local loop_instance 2024-09-11T06:41:51.658 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:41:51.658 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sleep 0 2024-09-11T06:41:51.658 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:41:51.658 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ seq 0 1 2024-09-11T06:41:51.661 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:41:51.662 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_27 2024-09-11T06:41:51.662 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1:0 2024-09-11T06:41:51.662 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:41:51.662 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:41:51.662 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:41:51.662 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:41:51.662 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=cluster 2024-09-11T06:41:51.662 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:41:51.662 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:41:51.662 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:41:51.662 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:41:51.662 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval cluster=cluster1 2024-09-11T06:41:51.662 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:41:51.663 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:41:51.663 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:41:51.663 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:41:51.663 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local local_cluster=cluster1:0 2024-09-11T06:41:51.663 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster=cluster1 2024-09-11T06:41:51.663 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance 2024-09-11T06:41:51.663 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:41:51.663 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local val=cluster1:0 2024-09-11T06:41:51.663 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T06:41:51.663 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance_var_name=instance 2024-09-11T06:41:51.663 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:41:51.663 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:41:51.663 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T06:41:51.663 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval local_cluster=cluster1 2024-09-11T06:41:51.663 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ local_cluster=cluster1 2024-09-11T06:41:51.664 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval instance=0 2024-09-11T06:41:51.664 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ instance=0 2024-09-11T06:41:51.664 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:41:51.683 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:41:51.683 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:41:51.684 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:41:51.684 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_27 2024-09-11T06:41:51.828 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ return 0 2024-09-11T06:41:51.828 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T06:41:51.828 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ echo ' "state": "Replaying"' 2024-09-11T06:41:51.829 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ grep -i Replaying 2024-09-11T06:41:51.830 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: "state": "Replaying" 2024-09-11T06:41:51.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_state=started 2024-09-11T06:41:51.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test started = started 2024-09-11T06:41:51.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:41:51.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_27 2024-09-11T06:41:51.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:41:51.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:41:51.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:51.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_27 2024-09-11T06:41:51.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' journal = journal ']' 2024-09-11T06:41:51.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_27 2024-09-11T06:41:51.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:41:51.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:41:51.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:51.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_27 2024-09-11T06:41:51.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-11T06:41:51.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-11T06:41:51.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:41:51.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:41:51.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:41:52.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror image_27 2024-09-11T06:41:52.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:41:52.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:52.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_27 2024-09-11T06:41:52.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:41:52.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n image_27 ']' 2024-09-11T06:41:52.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/image_27' 2024-09-11T06:41:52.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_27 2024-09-11T06:41:52.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:41:52.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:41:52.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:41:52.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:41:52.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:41:52.035 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:41:52.035 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:41:52.035 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:41:52.036 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:41:52.038 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:41:52.038 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_27 2024-09-11T06:41:52.038 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:41:52.038 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:41:52.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:41:52.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:41:52.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:41:52.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:41:52.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:41:52.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:41:52.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:41:52.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:41:52.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:41:52.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:41:52.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:41:52.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:41:52.041 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:41:52.041 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:41:52.041 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:41:52.041 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:41:52.041 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:41:52.041 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:41:52.041 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:41:52.041 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:41:52.041 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:41:52.041 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:41:52.041 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:41:52.041 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:41:52.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:41:52.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:41:52.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:41:52.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:41:52.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:41:52.067 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:41:52.067 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:41:52.067 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_27 2024-09-11T06:41:52.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:41:52.205 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror image_27 2024-09-11T06:41:52.205 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:41:52.205 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:41:52.205 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_27 2024-09-11T06:41:52.205 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror image_27 '' 2024-09-11T06:41:52.205 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:41:52.205 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:41:52.205 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_27 2024-09-11T06:41:52.205 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:41:52.206 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-image_27.status 2024-09-11T06:41:52.206 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-image_27.status 2024-09-11T06:41:52.206 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:41:52.207 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_27.status 2024-09-11T06:41:52.208 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_27 2024-09-11T06:41:52.208 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_27.status 2024-09-11T06:41:52.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 2 2024-09-11T06:41:52.254 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 2 2024-09-11T06:41:52.254 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:41:52.254 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=3, entry_tid=1], [object_number=8, tag_tid=3, entry_tid=0], [object_number=11, tag_tid=2, entry_tid=3], [object_number=10, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:41:52.254 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=9, tag_tid=3, entry_tid=1], [object_number=8, tag_tid=3, entry_tid=0], [object_number=11, tag_tid=2, entry_tid=3], [object_number=10, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:41:52.257 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_27.status 2024-09-11T06:41:52.258 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=9, tag_tid=3, entry_tid=1' 2024-09-11T06:41:52.259 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror image_27 2024-09-11T06:41:52.259 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:41:52.259 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:41:52.259 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_27 2024-09-11T06:41:52.259 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror image_27 '..*' 2024-09-11T06:41:52.259 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:41:52.260 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:41:52.260 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_27 2024-09-11T06:41:52.260 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:41:52.260 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-image_27.status 2024-09-11T06:41:52.260 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-image_27.status 2024-09-11T06:41:52.260 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:41:52.261 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_27.status 2024-09-11T06:41:52.261 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_27 2024-09-11T06:41:52.261 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_27.status 2024-09-11T06:41:52.307 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 2 2024-09-11T06:41:52.307 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 2 2024-09-11T06:41:52.307 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:41:52.307 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=3, entry_tid=1], [object_number=8, tag_tid=3, entry_tid=0], [object_number=11, tag_tid=2, entry_tid=3], [object_number=10, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:41:52.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=9, tag_tid=3, entry_tid=1], [object_number=8, tag_tid=3, entry_tid=0], [object_number=11, tag_tid=2, entry_tid=3], [object_number=10, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:41:52.310 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_27.status 2024-09-11T06:41:52.313 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=9, tag_tid=3, entry_tid=1' 2024-09-11T06:41:52.313 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=9, tag_tid=3, entry_tid=1' -a 'object_number=9, tag_tid=3, entry_tid=1' = 'object_number=9, tag_tid=3, entry_tid=1' 2024-09-11T06:41:52.313 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:41:52.313 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_snap_present cluster1 mirror image_27 snap 2024-09-11T06:41:52.313 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:41:52.313 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:52.313 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_27 2024-09-11T06:41:52.313 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:41:52.313 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:41:52.313 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T06:41:52.313 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:41:53.313 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 info mirror/image_27@snap 2024-09-11T06:41:53.380 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:rbd image 'image_27': 2024-09-11T06:41:53.380 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: size 128 MiB in 32 objects 2024-09-11T06:41:53.380 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: order 22 (4 MiB objects) 2024-09-11T06:41:53.380 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: snapshot_count: 1 2024-09-11T06:41:53.380 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: id: 10a6c3c2a9c1 2024-09-11T06:41:53.380 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: block_name_prefix: rbd_data.10a6c3c2a9c1 2024-09-11T06:41:53.380 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: format: 2 2024-09-11T06:41:53.380 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: features: layering, exclusive-lock, journaling 2024-09-11T06:41:53.380 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: op_features: 2024-09-11T06:41:53.380 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: flags: 2024-09-11T06:41:53.380 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: create_timestamp: Wed Sep 11 06:34:46 2024 2024-09-11T06:41:53.381 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: access_timestamp: Wed Sep 11 06:34:46 2024 2024-09-11T06:41:53.381 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: modify_timestamp: Wed Sep 11 06:37:07 2024 2024-09-11T06:41:53.381 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: protected: False 2024-09-11T06:41:53.381 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: journal: 10a6c3c2a9c1 2024-09-11T06:41:53.381 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring state: enabled 2024-09-11T06:41:53.381 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring mode: journal 2024-09-11T06:41:53.381 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring global id: 3d93829c-3486-4bc8-9d85-db688a8e1427 2024-09-11T06:41:53.381 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring primary: false 2024-09-11T06:41:53.385 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:41:53.385 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ compare_image_snaps mirror image_27 snap 2024-09-11T06:41:53.385 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:53.385 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_27 2024-09-11T06:41:53.385 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:41:53.385 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local ret=0 2024-09-11T06:41:53.385 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local rmt_export=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_27.export 2024-09-11T06:41:53.385 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loc_export=/tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_27.export 2024-09-11T06:41:53.385 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_27.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_27.export 2024-09-11T06:41:53.386 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror export image_27@snap /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_27.export 2024-09-11T06:41:53.582 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T06:41:53.590 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 -p mirror export image_27@snap /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_27.export 2024-09-11T06:41:53.989 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T06:41:54.005 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmp /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_27.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_27.export 2024-09-11T06:41:54.084 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_27.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_27.export 2024-09-11T06:41:54.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:41:54.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:41:54.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_28 2024-09-11T06:41:54.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_snap cluster2 mirror image_28 snap 2024-09-11T06:41:54.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:41:54.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:54.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_28 2024-09-11T06:41:54.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:41:54.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror snap create image_28@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T06:41:54.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_started cluster1 mirror image_28 2024-09-11T06:41:54.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:41:54.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:54.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_28 2024-09-11T06:41:54.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_state cluster1 mirror image_28 started 2024-09-11T06:41:54.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:41:54.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:54.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_28 2024-09-11T06:41:54.668 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=started 2024-09-11T06:41:54.668 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:41:54.668 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:41:54.668 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:41:55.668 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_replay_state cluster1 mirror image_28 started 2024-09-11T06:41:55.669 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:41:55.669 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:55.669 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_28 2024-09-11T06:41:55.669 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=started 2024-09-11T06:41:55.669 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local status_result 2024-09-11T06:41:55.669 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=stopped 2024-09-11T06:41:55.670 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_28 2024-09-11T06:41:55.670 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -i state 2024-09-11T06:41:55.670 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_instance=cluster1 2024-09-11T06:41:55.670 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:41:55.670 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:41:55.670 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance=cluster1 2024-09-11T06:41:55.670 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local loop_instance 2024-09-11T06:41:55.670 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:41:55.670 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sleep 0 2024-09-11T06:41:55.671 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:41:55.672 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ seq 0 1 2024-09-11T06:41:55.674 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:41:55.674 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_28 2024-09-11T06:41:55.674 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1:0 2024-09-11T06:41:55.674 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:41:55.674 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:41:55.674 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:41:55.674 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:41:55.674 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=cluster 2024-09-11T06:41:55.674 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:41:55.674 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:41:55.674 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:41:55.674 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:41:55.675 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval cluster=cluster1 2024-09-11T06:41:55.675 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:41:55.675 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:41:55.676 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:41:55.676 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:41:55.676 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local local_cluster=cluster1:0 2024-09-11T06:41:55.676 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster=cluster1 2024-09-11T06:41:55.677 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance 2024-09-11T06:41:55.677 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:41:55.677 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local val=cluster1:0 2024-09-11T06:41:55.677 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T06:41:55.677 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance_var_name=instance 2024-09-11T06:41:55.677 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:41:55.677 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:41:55.677 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T06:41:55.677 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval local_cluster=cluster1 2024-09-11T06:41:55.677 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ local_cluster=cluster1 2024-09-11T06:41:55.677 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval instance=0 2024-09-11T06:41:55.677 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ instance=0 2024-09-11T06:41:55.677 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:41:55.700 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:41:55.700 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:41:55.700 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:41:55.700 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_28 2024-09-11T06:41:55.831 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:Invalid command: unused arguments: ['mirror/image_28'] 2024-09-11T06:41:55.831 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T06:41:55.831 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:admin_socket: invalid command 2024-09-11T06:41:55.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:41:55.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_28 2024-09-11T06:41:55.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1:1 2024-09-11T06:41:55.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:41:55.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:41:55.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-11T06:41:55.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:1 2024-09-11T06:41:55.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=cluster 2024-09-11T06:41:55.835 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:41:55.835 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:41:55.835 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=1 2024-09-11T06:41:55.835 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T06:41:55.835 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval cluster=cluster1 2024-09-11T06:41:55.835 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:41:55.835 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=1 2024-09-11T06:41:55.835 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=1 2024-09-11T06:41:55.835 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-11T06:41:55.835 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local local_cluster=cluster1:1 2024-09-11T06:41:55.835 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster=cluster1 2024-09-11T06:41:55.835 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance 2024-09-11T06:41:55.835 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T06:41:55.835 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local val=cluster1:1 2024-09-11T06:41:55.836 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T06:41:55.836 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance_var_name=instance 2024-09-11T06:41:55.836 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:41:55.836 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=1 2024-09-11T06:41:55.836 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-11T06:41:55.836 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval local_cluster=cluster1 2024-09-11T06:41:55.836 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ local_cluster=cluster1 2024-09-11T06:41:55.836 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval instance=1 2024-09-11T06:41:55.836 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ instance=1 2024-09-11T06:41:55.836 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T06:41:55.859 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:41:55.859 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:41:55.860 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:41:55.860 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_28 2024-09-11T06:41:55.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ return 0 2024-09-11T06:41:55.994 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T06:41:55.995 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ echo ' "state": "Replaying"' 2024-09-11T06:41:55.995 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ grep -i Replaying 2024-09-11T06:41:55.996 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: "state": "Replaying" 2024-09-11T06:41:55.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_state=started 2024-09-11T06:41:55.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test started = started 2024-09-11T06:41:55.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:41:55.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_28 2024-09-11T06:41:55.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:41:55.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:41:55.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:55.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_28 2024-09-11T06:41:55.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' journal = journal ']' 2024-09-11T06:41:55.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_28 2024-09-11T06:41:55.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:41:55.999 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:41:55.999 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:55.999 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_28 2024-09-11T06:41:55.999 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-11T06:41:55.999 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-11T06:41:55.999 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:41:55.999 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:41:55.999 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:41:56.200 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror image_28 2024-09-11T06:41:56.200 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:41:56.200 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:56.200 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_28 2024-09-11T06:41:56.200 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:41:56.200 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n image_28 ']' 2024-09-11T06:41:56.200 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/image_28' 2024-09-11T06:41:56.200 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_28 2024-09-11T06:41:56.200 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:41:56.200 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:41:56.200 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:41:56.200 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:41:56.201 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:41:56.201 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:41:56.201 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:41:56.201 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:41:56.201 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:41:56.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:41:56.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_28 2024-09-11T06:41:56.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:41:56.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:41:56.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:41:56.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:41:56.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:41:56.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:41:56.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:41:56.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:41:56.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:41:56.205 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:41:56.205 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:41:56.205 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:41:56.205 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:41:56.205 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:41:56.206 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:41:56.207 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:41:56.207 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:41:56.207 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:41:56.207 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:41:56.207 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:41:56.207 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:41:56.207 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:41:56.207 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:41:56.207 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:41:56.207 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:41:56.207 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:41:56.207 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:41:56.207 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:41:56.207 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:41:56.208 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:41:56.232 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:41:56.232 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:41:56.232 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:41:56.232 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_28 2024-09-11T06:41:56.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:Invalid command: unused arguments: ['mirror/image_28'] 2024-09-11T06:41:56.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd mirror flush : flush rbd mirror 2024-09-11T06:41:56.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:admin_socket: invalid command 2024-09-11T06:41:56.358 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:41:56.359 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_28 2024-09-11T06:41:56.359 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:1 2024-09-11T06:41:56.359 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:41:56.359 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:41:56.359 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-11T06:41:56.359 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:1 2024-09-11T06:41:56.359 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:41:56.359 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:41:56.359 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:41:56.359 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=1 2024-09-11T06:41:56.359 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-11T06:41:56.359 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:41:56.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:41:56.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=1 2024-09-11T06:41:56.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=1 2024-09-11T06:41:56.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-11T06:41:56.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:1 2024-09-11T06:41:56.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:41:56.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:41:56.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T06:41:56.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:1 2024-09-11T06:41:56.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:41:56.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:41:56.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:41:56.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=1 2024-09-11T06:41:56.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T06:41:56.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:41:56.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:41:56.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=1 2024-09-11T06:41:56.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=1 2024-09-11T06:41:56.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T06:41:56.382 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:41:56.383 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:41:56.383 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:41:56.383 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_28 2024-09-11T06:41:56.520 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:41:56.521 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror image_28 2024-09-11T06:41:56.521 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:41:56.521 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:41:56.521 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_28 2024-09-11T06:41:56.521 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror image_28 '' 2024-09-11T06:41:56.521 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:41:56.521 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:41:56.521 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_28 2024-09-11T06:41:56.521 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:41:56.522 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-image_28.status 2024-09-11T06:41:56.522 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-image_28.status 2024-09-11T06:41:56.522 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:41:56.523 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_28.status 2024-09-11T06:41:56.524 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_28 2024-09-11T06:41:56.524 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_28.status 2024-09-11T06:41:56.575 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 1 2024-09-11T06:41:56.575 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 1 2024-09-11T06:41:56.575 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:41:56.575 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:41:56.575 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:41:56.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_28.status 2024-09-11T06:41:56.580 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-09-11T06:41:56.581 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror image_28 2024-09-11T06:41:56.581 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:41:56.581 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:41:56.581 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_28 2024-09-11T06:41:56.582 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror image_28 '..*' 2024-09-11T06:41:56.582 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:41:56.582 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:41:56.582 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_28 2024-09-11T06:41:56.582 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:41:56.582 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-image_28.status 2024-09-11T06:41:56.582 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-image_28.status 2024-09-11T06:41:56.582 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:41:56.582 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_28.status 2024-09-11T06:41:56.583 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_28 2024-09-11T06:41:56.583 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_28.status 2024-09-11T06:41:56.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 1 2024-09-11T06:41:56.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 1 2024-09-11T06:41:56.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:41:56.630 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:41:56.631 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:41:56.634 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_28.status 2024-09-11T06:41:56.636 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-09-11T06:41:56.636 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=3, entry_tid=1' 2024-09-11T06:41:56.636 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:41:56.636 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_snap_present cluster1 mirror image_28 snap 2024-09-11T06:41:56.636 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:41:56.636 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:56.636 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_28 2024-09-11T06:41:56.636 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:41:56.637 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:41:56.637 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T06:41:56.637 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:41:57.637 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 info mirror/image_28@snap 2024-09-11T06:41:57.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:rbd image 'image_28': 2024-09-11T06:41:57.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: size 128 MiB in 32 objects 2024-09-11T06:41:57.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: order 22 (4 MiB objects) 2024-09-11T06:41:57.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: snapshot_count: 1 2024-09-11T06:41:57.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: id: 10a9a7a94fdb 2024-09-11T06:41:57.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: block_name_prefix: rbd_data.10a9a7a94fdb 2024-09-11T06:41:57.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: format: 2 2024-09-11T06:41:57.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: features: layering, exclusive-lock, journaling 2024-09-11T06:41:57.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: op_features: 2024-09-11T06:41:57.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: flags: 2024-09-11T06:41:57.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: create_timestamp: Wed Sep 11 06:35:00 2024 2024-09-11T06:41:57.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: access_timestamp: Wed Sep 11 06:35:00 2024 2024-09-11T06:41:57.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: modify_timestamp: Wed Sep 11 06:36:16 2024 2024-09-11T06:41:57.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: protected: False 2024-09-11T06:41:57.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: journal: 10a9a7a94fdb 2024-09-11T06:41:57.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring state: enabled 2024-09-11T06:41:57.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring mode: journal 2024-09-11T06:41:57.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring global id: 02c77597-997a-45bd-88bc-393a1c4bbf01 2024-09-11T06:41:57.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring primary: false 2024-09-11T06:41:57.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:41:57.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ compare_image_snaps mirror image_28 snap 2024-09-11T06:41:57.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:57.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_28 2024-09-11T06:41:57.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:41:57.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local ret=0 2024-09-11T06:41:57.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local rmt_export=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_28.export 2024-09-11T06:41:57.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loc_export=/tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_28.export 2024-09-11T06:41:57.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_28.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_28.export 2024-09-11T06:41:57.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror export image_28@snap /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_28.export 2024-09-11T06:41:58.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T06:41:58.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 -p mirror export image_28@snap /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_28.export 2024-09-11T06:41:58.498 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T06:41:58.510 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmp /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_28.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_28.export 2024-09-11T06:41:58.589 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_28.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_28.export 2024-09-11T06:41:58.631 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:41:58.631 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:41:58.631 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_29 2024-09-11T06:41:58.631 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_snap cluster2 mirror image_29 snap 2024-09-11T06:41:58.631 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:41:58.631 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:58.631 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_29 2024-09-11T06:41:58.631 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:41:58.631 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror snap create image_29@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T06:41:59.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_started cluster1 mirror image_29 2024-09-11T06:41:59.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:41:59.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:59.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_29 2024-09-11T06:41:59.689 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_state cluster1 mirror image_29 started 2024-09-11T06:41:59.689 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:41:59.689 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:41:59.689 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_29 2024-09-11T06:41:59.689 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=started 2024-09-11T06:41:59.689 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:41:59.689 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:41:59.689 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:42:00.690 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_replay_state cluster1 mirror image_29 started 2024-09-11T06:42:00.690 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:42:00.690 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:00.690 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_29 2024-09-11T06:42:00.690 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=started 2024-09-11T06:42:00.690 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local status_result 2024-09-11T06:42:00.690 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=stopped 2024-09-11T06:42:00.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_29 2024-09-11T06:42:00.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -i state 2024-09-11T06:42:00.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_instance=cluster1 2024-09-11T06:42:00.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:42:00.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:42:00.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance=cluster1 2024-09-11T06:42:00.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local loop_instance 2024-09-11T06:42:00.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:42:00.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sleep 0 2024-09-11T06:42:00.693 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:42:00.693 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ seq 0 1 2024-09-11T06:42:00.695 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:42:00.696 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_29 2024-09-11T06:42:00.696 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1:0 2024-09-11T06:42:00.696 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:42:00.696 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:42:00.696 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:42:00.696 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:42:00.696 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=cluster 2024-09-11T06:42:00.696 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:42:00.696 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:42:00.696 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:42:00.696 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:42:00.696 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval cluster=cluster1 2024-09-11T06:42:00.697 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:42:00.697 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:42:00.697 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:42:00.697 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:42:00.697 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local local_cluster=cluster1:0 2024-09-11T06:42:00.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster=cluster1 2024-09-11T06:42:00.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance 2024-09-11T06:42:00.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:42:00.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local val=cluster1:0 2024-09-11T06:42:00.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T06:42:00.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance_var_name=instance 2024-09-11T06:42:00.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:42:00.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:42:00.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T06:42:00.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval local_cluster=cluster1 2024-09-11T06:42:00.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ local_cluster=cluster1 2024-09-11T06:42:00.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval instance=0 2024-09-11T06:42:00.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ instance=0 2024-09-11T06:42:00.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:42:00.721 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:42:00.722 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:42:00.722 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:42:00.722 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_29 2024-09-11T06:42:00.868 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ return 0 2024-09-11T06:42:00.868 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T06:42:00.869 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ echo ' "state": "Replaying"' 2024-09-11T06:42:00.869 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ grep -i Replaying 2024-09-11T06:42:00.871 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: "state": "Replaying" 2024-09-11T06:42:00.873 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_state=started 2024-09-11T06:42:00.873 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test started = started 2024-09-11T06:42:00.873 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:42:00.873 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_29 2024-09-11T06:42:00.873 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:42:00.873 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:42:00.873 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:00.873 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_29 2024-09-11T06:42:00.874 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' journal = journal ']' 2024-09-11T06:42:00.874 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_29 2024-09-11T06:42:00.874 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:42:00.874 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:42:00.874 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:00.874 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_29 2024-09-11T06:42:00.874 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-11T06:42:00.874 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-11T06:42:00.874 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:42:00.874 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:42:00.874 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:42:01.074 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror image_29 2024-09-11T06:42:01.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:42:01.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:01.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_29 2024-09-11T06:42:01.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:42:01.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n image_29 ']' 2024-09-11T06:42:01.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/image_29' 2024-09-11T06:42:01.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_29 2024-09-11T06:42:01.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:42:01.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:42:01.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:42:01.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:42:01.076 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:42:01.076 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:42:01.076 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:42:01.076 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:42:01.077 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:42:01.079 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:42:01.079 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_29 2024-09-11T06:42:01.079 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:42:01.079 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:42:01.079 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:42:01.079 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:42:01.079 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:42:01.079 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:42:01.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:42:01.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:42:01.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:42:01.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:42:01.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:42:01.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:42:01.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:42:01.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:42:01.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:42:01.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:42:01.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:42:01.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:42:01.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:42:01.081 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:42:01.081 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:42:01.081 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:42:01.081 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:42:01.081 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:42:01.081 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:42:01.081 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:42:01.081 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:42:01.081 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:42:01.081 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:42:01.081 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:42:01.103 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:42:01.104 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:42:01.104 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:42:01.104 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_29 2024-09-11T06:42:01.244 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:42:01.245 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror image_29 2024-09-11T06:42:01.246 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:42:01.246 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:42:01.246 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_29 2024-09-11T06:42:01.246 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror image_29 '' 2024-09-11T06:42:01.246 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:42:01.246 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:42:01.246 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_29 2024-09-11T06:42:01.246 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:42:01.246 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-image_29.status 2024-09-11T06:42:01.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-image_29.status 2024-09-11T06:42:01.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:42:01.248 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_29.status 2024-09-11T06:42:01.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_29 2024-09-11T06:42:01.249 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_29.status 2024-09-11T06:42:01.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 2 2024-09-11T06:42:01.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 2 2024-09-11T06:42:01.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:42:01.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=3, entry_tid=1], [object_number=8, tag_tid=3, entry_tid=0], [object_number=11, tag_tid=2, entry_tid=3], [object_number=10, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:42:01.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=9, tag_tid=3, entry_tid=1], [object_number=8, tag_tid=3, entry_tid=0], [object_number=11, tag_tid=2, entry_tid=3], [object_number=10, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:42:01.306 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_29.status 2024-09-11T06:42:01.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=9, tag_tid=3, entry_tid=1' 2024-09-11T06:42:01.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror image_29 2024-09-11T06:42:01.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:42:01.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:42:01.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_29 2024-09-11T06:42:01.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror image_29 '..*' 2024-09-11T06:42:01.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:42:01.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:42:01.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_29 2024-09-11T06:42:01.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:42:01.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-image_29.status 2024-09-11T06:42:01.310 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-image_29.status 2024-09-11T06:42:01.310 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:42:01.310 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_29.status 2024-09-11T06:42:01.310 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_29 2024-09-11T06:42:01.310 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_29.status 2024-09-11T06:42:01.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 2 2024-09-11T06:42:01.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 2 2024-09-11T06:42:01.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:42:01.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=3, entry_tid=1], [object_number=8, tag_tid=3, entry_tid=0], [object_number=11, tag_tid=2, entry_tid=3], [object_number=10, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:42:01.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=9, tag_tid=3, entry_tid=1], [object_number=8, tag_tid=3, entry_tid=0], [object_number=11, tag_tid=2, entry_tid=3], [object_number=10, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:42:01.358 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_29.status 2024-09-11T06:42:01.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=9, tag_tid=3, entry_tid=1' 2024-09-11T06:42:01.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=9, tag_tid=3, entry_tid=1' -a 'object_number=9, tag_tid=3, entry_tid=1' = 'object_number=9, tag_tid=3, entry_tid=1' 2024-09-11T06:42:01.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:42:01.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_snap_present cluster1 mirror image_29 snap 2024-09-11T06:42:01.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:42:01.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:01.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_29 2024-09-11T06:42:01.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:42:01.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:42:01.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T06:42:01.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:42:02.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 info mirror/image_29@snap 2024-09-11T06:42:02.428 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:rbd image 'image_29': 2024-09-11T06:42:02.429 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: size 128 MiB in 32 objects 2024-09-11T06:42:02.429 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: order 22 (4 MiB objects) 2024-09-11T06:42:02.429 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: snapshot_count: 1 2024-09-11T06:42:02.429 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: id: 10a65bba848a 2024-09-11T06:42:02.429 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: block_name_prefix: rbd_data.10a65bba848a 2024-09-11T06:42:02.429 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: format: 2 2024-09-11T06:42:02.429 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: features: layering, exclusive-lock, journaling 2024-09-11T06:42:02.429 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: op_features: 2024-09-11T06:42:02.429 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: flags: 2024-09-11T06:42:02.429 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: create_timestamp: Wed Sep 11 06:35:07 2024 2024-09-11T06:42:02.429 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: access_timestamp: Wed Sep 11 06:35:07 2024 2024-09-11T06:42:02.430 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: modify_timestamp: Wed Sep 11 06:37:08 2024 2024-09-11T06:42:02.430 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: protected: False 2024-09-11T06:42:02.430 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: journal: 10a65bba848a 2024-09-11T06:42:02.430 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring state: enabled 2024-09-11T06:42:02.430 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring mode: journal 2024-09-11T06:42:02.430 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring global id: a9a3090e-1ac3-448d-a06d-da23dafe6875 2024-09-11T06:42:02.430 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring primary: false 2024-09-11T06:42:02.433 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:42:02.433 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ compare_image_snaps mirror image_29 snap 2024-09-11T06:42:02.434 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:02.434 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_29 2024-09-11T06:42:02.434 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:42:02.434 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local ret=0 2024-09-11T06:42:02.434 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local rmt_export=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_29.export 2024-09-11T06:42:02.434 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loc_export=/tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_29.export 2024-09-11T06:42:02.434 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_29.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_29.export 2024-09-11T06:42:02.434 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror export image_29@snap /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_29.export 2024-09-11T06:42:02.633 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T06:42:02.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 -p mirror export image_29@snap /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_29.export 2024-09-11T06:42:03.087 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T06:42:03.104 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmp /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_29.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_29.export 2024-09-11T06:42:03.183 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_29.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_29.export 2024-09-11T06:42:03.226 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:42:03.226 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:42:03.226 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_30 2024-09-11T06:42:03.226 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_snap cluster2 mirror image_30 snap 2024-09-11T06:42:03.227 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:42:03.227 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:03.227 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_30 2024-09-11T06:42:03.227 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:42:03.227 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror snap create image_30@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T06:42:03.701 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_started cluster1 mirror image_30 2024-09-11T06:42:03.701 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:42:03.701 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:03.701 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_30 2024-09-11T06:42:03.702 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_state cluster1 mirror image_30 started 2024-09-11T06:42:03.702 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:42:03.702 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:03.702 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_30 2024-09-11T06:42:03.702 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=started 2024-09-11T06:42:03.702 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:42:03.702 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:42:03.702 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:42:04.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_replay_state cluster1 mirror image_30 started 2024-09-11T06:42:04.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:42:04.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:04.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_30 2024-09-11T06:42:04.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=started 2024-09-11T06:42:04.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local status_result 2024-09-11T06:42:04.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=stopped 2024-09-11T06:42:04.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_30 2024-09-11T06:42:04.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -i state 2024-09-11T06:42:04.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_instance=cluster1 2024-09-11T06:42:04.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:42:04.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:42:04.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance=cluster1 2024-09-11T06:42:04.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local loop_instance 2024-09-11T06:42:04.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:42:04.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sleep 0 2024-09-11T06:42:04.706 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:42:04.706 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ seq 0 1 2024-09-11T06:42:04.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:42:04.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_30 2024-09-11T06:42:04.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1:0 2024-09-11T06:42:04.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:42:04.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:42:04.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:42:04.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:42:04.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=cluster 2024-09-11T06:42:04.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:42:04.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:42:04.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:42:04.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:42:04.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval cluster=cluster1 2024-09-11T06:42:04.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:42:04.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:42:04.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:42:04.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:42:04.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local local_cluster=cluster1:0 2024-09-11T06:42:04.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster=cluster1 2024-09-11T06:42:04.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance 2024-09-11T06:42:04.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:42:04.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local val=cluster1:0 2024-09-11T06:42:04.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T06:42:04.711 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance_var_name=instance 2024-09-11T06:42:04.711 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:42:04.711 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:42:04.711 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T06:42:04.711 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval local_cluster=cluster1 2024-09-11T06:42:04.711 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ local_cluster=cluster1 2024-09-11T06:42:04.711 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval instance=0 2024-09-11T06:42:04.711 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ instance=0 2024-09-11T06:42:04.711 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:42:04.731 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:42:04.731 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:42:04.731 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:42:04.731 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_30 2024-09-11T06:42:04.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2024-09-11T06:42:04.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T06:42:04.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:admin_socket: invalid command 2024-09-11T06:42:04.892 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:42:04.892 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_30 2024-09-11T06:42:04.892 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1:1 2024-09-11T06:42:04.892 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:42:04.892 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:42:04.893 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-11T06:42:04.893 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:1 2024-09-11T06:42:04.893 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=cluster 2024-09-11T06:42:04.893 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:42:04.893 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:42:04.893 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=1 2024-09-11T06:42:04.893 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T06:42:04.893 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval cluster=cluster1 2024-09-11T06:42:04.893 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:42:04.893 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=1 2024-09-11T06:42:04.893 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=1 2024-09-11T06:42:04.893 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-11T06:42:04.893 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local local_cluster=cluster1:1 2024-09-11T06:42:04.894 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster=cluster1 2024-09-11T06:42:04.894 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance 2024-09-11T06:42:04.894 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T06:42:04.894 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local val=cluster1:1 2024-09-11T06:42:04.894 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T06:42:04.894 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance_var_name=instance 2024-09-11T06:42:04.894 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:42:04.894 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=1 2024-09-11T06:42:04.894 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-11T06:42:04.894 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval local_cluster=cluster1 2024-09-11T06:42:04.894 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ local_cluster=cluster1 2024-09-11T06:42:04.894 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval instance=1 2024-09-11T06:42:04.894 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ instance=1 2024-09-11T06:42:04.895 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T06:42:04.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:42:04.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:42:04.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:42:04.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_30 2024-09-11T06:42:05.051 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ return 0 2024-09-11T06:42:05.052 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T06:42:05.053 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ echo ' "state": "Replaying"' 2024-09-11T06:42:05.053 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ grep -i Replaying 2024-09-11T06:42:05.055 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: "state": "Replaying" 2024-09-11T06:42:05.057 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_state=started 2024-09-11T06:42:05.057 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test started = started 2024-09-11T06:42:05.057 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:42:05.057 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_30 2024-09-11T06:42:05.057 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:42:05.057 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:42:05.057 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:05.057 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_30 2024-09-11T06:42:05.058 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' journal = journal ']' 2024-09-11T06:42:05.058 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_30 2024-09-11T06:42:05.058 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:42:05.058 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:42:05.058 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:05.058 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_30 2024-09-11T06:42:05.058 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-11T06:42:05.058 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-11T06:42:05.058 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:42:05.058 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:42:05.058 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:42:05.258 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror image_30 2024-09-11T06:42:05.259 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:42:05.259 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:05.259 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_30 2024-09-11T06:42:05.259 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:42:05.259 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n image_30 ']' 2024-09-11T06:42:05.259 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/image_30' 2024-09-11T06:42:05.259 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_30 2024-09-11T06:42:05.259 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:42:05.259 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:42:05.259 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:42:05.259 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:42:05.259 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:42:05.259 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:42:05.260 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:42:05.260 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:42:05.260 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:42:05.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:42:05.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_30 2024-09-11T06:42:05.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:42:05.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:42:05.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:42:05.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:42:05.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:42:05.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:42:05.264 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:42:05.264 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:42:05.264 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:42:05.264 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:42:05.264 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:42:05.264 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:42:05.264 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:42:05.264 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:42:05.264 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:42:05.264 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:42:05.264 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:42:05.264 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:42:05.264 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:42:05.265 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:42:05.265 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:42:05.265 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:42:05.265 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:42:05.265 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:42:05.265 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:42:05.265 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:42:05.265 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:42:05.265 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:42:05.265 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:42:05.265 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:42:05.288 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:42:05.289 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:42:05.289 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:42:05.289 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_30 2024-09-11T06:42:05.409 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2024-09-11T06:42:05.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd mirror flush : flush rbd mirror 2024-09-11T06:42:05.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:admin_socket: invalid command 2024-09-11T06:42:05.413 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:42:05.413 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_30 2024-09-11T06:42:05.413 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:1 2024-09-11T06:42:05.413 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:42:05.413 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:42:05.413 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-11T06:42:05.413 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:1 2024-09-11T06:42:05.413 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:42:05.413 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:42:05.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:42:05.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=1 2024-09-11T06:42:05.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-11T06:42:05.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:42:05.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:42:05.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=1 2024-09-11T06:42:05.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=1 2024-09-11T06:42:05.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-11T06:42:05.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:1 2024-09-11T06:42:05.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:42:05.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:42:05.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T06:42:05.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:1 2024-09-11T06:42:05.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:42:05.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:42:05.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:42:05.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=1 2024-09-11T06:42:05.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T06:42:05.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:42:05.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:42:05.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=1 2024-09-11T06:42:05.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=1 2024-09-11T06:42:05.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T06:42:05.437 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:42:05.437 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:42:05.437 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:42:05.437 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_30 2024-09-11T06:42:05.580 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:42:05.582 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror image_30 2024-09-11T06:42:05.582 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:42:05.582 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:42:05.582 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_30 2024-09-11T06:42:05.582 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror image_30 '' 2024-09-11T06:42:05.582 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:42:05.582 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:42:05.582 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_30 2024-09-11T06:42:05.582 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:42:05.583 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-image_30.status 2024-09-11T06:42:05.583 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-image_30.status 2024-09-11T06:42:05.583 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:42:05.584 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_30.status 2024-09-11T06:42:05.584 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_30 2024-09-11T06:42:05.584 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_30.status 2024-09-11T06:42:05.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 2 2024-09-11T06:42:05.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 2 2024-09-11T06:42:05.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:42:05.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=3, entry_tid=1], [object_number=8, tag_tid=3, entry_tid=0], [object_number=11, tag_tid=2, entry_tid=3], [object_number=10, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:42:05.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=9, tag_tid=3, entry_tid=1], [object_number=8, tag_tid=3, entry_tid=0], [object_number=11, tag_tid=2, entry_tid=3], [object_number=10, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:42:05.650 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_30.status 2024-09-11T06:42:05.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=9, tag_tid=3, entry_tid=1' 2024-09-11T06:42:05.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror image_30 2024-09-11T06:42:05.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:42:05.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:42:05.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_30 2024-09-11T06:42:05.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror image_30 '..*' 2024-09-11T06:42:05.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:42:05.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:42:05.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_30 2024-09-11T06:42:05.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:42:05.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-image_30.status 2024-09-11T06:42:05.654 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-image_30.status 2024-09-11T06:42:05.654 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:42:05.654 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_30.status 2024-09-11T06:42:05.654 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_30 2024-09-11T06:42:05.654 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_30.status 2024-09-11T06:42:05.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 2 2024-09-11T06:42:05.700 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 2 2024-09-11T06:42:05.700 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:42:05.700 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=3, entry_tid=1], [object_number=8, tag_tid=3, entry_tid=0], [object_number=11, tag_tid=2, entry_tid=3], [object_number=10, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:42:05.700 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=9, tag_tid=3, entry_tid=1], [object_number=8, tag_tid=3, entry_tid=0], [object_number=11, tag_tid=2, entry_tid=3], [object_number=10, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:42:05.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_30.status 2024-09-11T06:42:05.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=9, tag_tid=3, entry_tid=1' 2024-09-11T06:42:05.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=9, tag_tid=3, entry_tid=1' -a 'object_number=9, tag_tid=3, entry_tid=1' = 'object_number=9, tag_tid=3, entry_tid=1' 2024-09-11T06:42:05.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:42:05.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_snap_present cluster1 mirror image_30 snap 2024-09-11T06:42:05.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:42:05.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:05.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_30 2024-09-11T06:42:05.706 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:42:05.706 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:42:05.706 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T06:42:05.706 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:42:06.706 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 info mirror/image_30@snap 2024-09-11T06:42:06.777 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:rbd image 'image_30': 2024-09-11T06:42:06.777 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: size 128 MiB in 32 objects 2024-09-11T06:42:06.777 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: order 22 (4 MiB objects) 2024-09-11T06:42:06.777 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: snapshot_count: 1 2024-09-11T06:42:06.777 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: id: 10a9e0aa484e 2024-09-11T06:42:06.777 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: block_name_prefix: rbd_data.10a9e0aa484e 2024-09-11T06:42:06.777 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: format: 2 2024-09-11T06:42:06.777 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: features: layering, exclusive-lock, journaling 2024-09-11T06:42:06.777 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: op_features: 2024-09-11T06:42:06.777 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: flags: 2024-09-11T06:42:06.777 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: create_timestamp: Wed Sep 11 06:35:17 2024 2024-09-11T06:42:06.777 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: access_timestamp: Wed Sep 11 06:35:17 2024 2024-09-11T06:42:06.778 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: modify_timestamp: Wed Sep 11 06:37:11 2024 2024-09-11T06:42:06.778 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: protected: False 2024-09-11T06:42:06.778 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: journal: 10a9e0aa484e 2024-09-11T06:42:06.778 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring state: enabled 2024-09-11T06:42:06.778 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring mode: journal 2024-09-11T06:42:06.778 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring global id: f136bf02-144a-4082-a765-61f4a1e569c0 2024-09-11T06:42:06.778 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring primary: false 2024-09-11T06:42:06.781 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:42:06.781 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ compare_image_snaps mirror image_30 snap 2024-09-11T06:42:06.781 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:06.782 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_30 2024-09-11T06:42:06.782 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:42:06.782 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local ret=0 2024-09-11T06:42:06.782 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local rmt_export=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_30.export 2024-09-11T06:42:06.782 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loc_export=/tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_30.export 2024-09-11T06:42:06.782 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_30.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_30.export 2024-09-11T06:42:06.782 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror export image_30@snap /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_30.export 2024-09-11T06:42:06.983 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T06:42:06.991 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 -p mirror export image_30@snap /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_30.export 2024-09-11T06:42:07.350 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T06:42:07.363 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmp /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_30.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_30.export 2024-09-11T06:42:07.440 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_30.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_30.export 2024-09-11T06:42:07.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:42:07.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:42:07.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_31 2024-09-11T06:42:07.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_snap cluster2 mirror image_31 snap 2024-09-11T06:42:07.483 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:42:07.483 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:07.483 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_31 2024-09-11T06:42:07.483 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:42:07.483 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror snap create image_31@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T06:42:08.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_started cluster1 mirror image_31 2024-09-11T06:42:08.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:42:08.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:08.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_31 2024-09-11T06:42:08.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_state cluster1 mirror image_31 started 2024-09-11T06:42:08.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:42:08.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:08.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_31 2024-09-11T06:42:08.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=started 2024-09-11T06:42:08.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:42:08.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:42:08.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:42:09.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_replay_state cluster1 mirror image_31 started 2024-09-11T06:42:09.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:42:09.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:09.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_31 2024-09-11T06:42:09.716 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=started 2024-09-11T06:42:09.716 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local status_result 2024-09-11T06:42:09.716 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=stopped 2024-09-11T06:42:09.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_31 2024-09-11T06:42:09.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -i state 2024-09-11T06:42:09.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_instance=cluster1 2024-09-11T06:42:09.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:42:09.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:42:09.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance=cluster1 2024-09-11T06:42:09.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local loop_instance 2024-09-11T06:42:09.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:42:09.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sleep 0 2024-09-11T06:42:09.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:42:09.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ seq 0 1 2024-09-11T06:42:09.721 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:42:09.721 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_31 2024-09-11T06:42:09.721 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1:0 2024-09-11T06:42:09.721 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:42:09.721 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:42:09.722 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:42:09.722 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:42:09.722 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=cluster 2024-09-11T06:42:09.722 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:42:09.722 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:42:09.722 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:42:09.722 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:42:09.722 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval cluster=cluster1 2024-09-11T06:42:09.722 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:42:09.722 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:42:09.722 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:42:09.724 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:42:09.724 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local local_cluster=cluster1:0 2024-09-11T06:42:09.724 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster=cluster1 2024-09-11T06:42:09.724 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance 2024-09-11T06:42:09.724 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:42:09.724 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local val=cluster1:0 2024-09-11T06:42:09.724 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T06:42:09.724 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance_var_name=instance 2024-09-11T06:42:09.724 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:42:09.725 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:42:09.725 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T06:42:09.725 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval local_cluster=cluster1 2024-09-11T06:42:09.725 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ local_cluster=cluster1 2024-09-11T06:42:09.725 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval instance=0 2024-09-11T06:42:09.725 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ instance=0 2024-09-11T06:42:09.725 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:42:09.748 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:42:09.748 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:42:09.748 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:42:09.748 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_31 2024-09-11T06:42:09.921 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ return 0 2024-09-11T06:42:09.922 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T06:42:09.922 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ echo ' "state": "Replaying"' 2024-09-11T06:42:09.923 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ grep -i Replaying 2024-09-11T06:42:09.923 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: "state": "Replaying" 2024-09-11T06:42:09.924 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_state=started 2024-09-11T06:42:09.924 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test started = started 2024-09-11T06:42:09.924 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:42:09.924 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_31 2024-09-11T06:42:09.925 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:42:09.925 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:42:09.925 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:09.925 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_31 2024-09-11T06:42:09.925 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' journal = journal ']' 2024-09-11T06:42:09.925 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_31 2024-09-11T06:42:09.925 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:42:09.925 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:42:09.925 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:09.925 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_31 2024-09-11T06:42:09.925 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-11T06:42:09.926 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-11T06:42:09.926 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:42:09.926 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:42:09.926 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:42:10.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror image_31 2024-09-11T06:42:10.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:42:10.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:10.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_31 2024-09-11T06:42:10.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:42:10.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n image_31 ']' 2024-09-11T06:42:10.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/image_31' 2024-09-11T06:42:10.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_31 2024-09-11T06:42:10.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:42:10.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:42:10.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:42:10.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:42:10.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:42:10.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:42:10.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:42:10.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:42:10.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:42:10.130 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:42:10.130 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_31 2024-09-11T06:42:10.130 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:42:10.130 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:42:10.130 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:42:10.130 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:42:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:42:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:42:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:42:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:42:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:42:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:42:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:42:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:42:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:42:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:42:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:42:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:42:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:42:10.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:42:10.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:42:10.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:42:10.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:42:10.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:42:10.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:42:10.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:42:10.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:42:10.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:42:10.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:42:10.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:42:10.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:42:10.132 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:42:10.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:42:10.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:42:10.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:42:10.158 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_31 2024-09-11T06:42:10.300 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:42:10.301 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror image_31 2024-09-11T06:42:10.301 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:42:10.301 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:42:10.301 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_31 2024-09-11T06:42:10.301 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror image_31 '' 2024-09-11T06:42:10.301 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:42:10.301 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:42:10.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_31 2024-09-11T06:42:10.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:42:10.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-image_31.status 2024-09-11T06:42:10.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-image_31.status 2024-09-11T06:42:10.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:42:10.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_31.status 2024-09-11T06:42:10.305 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_31 2024-09-11T06:42:10.305 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_31.status 2024-09-11T06:42:10.356 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 2 2024-09-11T06:42:10.356 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 2 2024-09-11T06:42:10.356 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:42:10.356 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=3, entry_tid=1], [object_number=8, tag_tid=3, entry_tid=0], [object_number=11, tag_tid=2, entry_tid=3], [object_number=10, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:42:10.356 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=9, tag_tid=3, entry_tid=1], [object_number=8, tag_tid=3, entry_tid=0], [object_number=11, tag_tid=2, entry_tid=3], [object_number=10, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:42:10.359 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_31.status 2024-09-11T06:42:10.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=9, tag_tid=3, entry_tid=1' 2024-09-11T06:42:10.362 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror image_31 2024-09-11T06:42:10.362 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:42:10.362 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:42:10.362 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_31 2024-09-11T06:42:10.362 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror image_31 '..*' 2024-09-11T06:42:10.362 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:42:10.362 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:42:10.362 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_31 2024-09-11T06:42:10.362 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:42:10.362 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-image_31.status 2024-09-11T06:42:10.362 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-image_31.status 2024-09-11T06:42:10.362 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:42:10.363 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_31.status 2024-09-11T06:42:10.363 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_31 2024-09-11T06:42:10.363 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_31.status 2024-09-11T06:42:10.409 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 2 2024-09-11T06:42:10.409 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 2 2024-09-11T06:42:10.409 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:42:10.409 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=3, entry_tid=1], [object_number=8, tag_tid=3, entry_tid=0], [object_number=11, tag_tid=2, entry_tid=3], [object_number=10, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:42:10.409 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=9, tag_tid=3, entry_tid=1], [object_number=8, tag_tid=3, entry_tid=0], [object_number=11, tag_tid=2, entry_tid=3], [object_number=10, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:42:10.413 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_31.status 2024-09-11T06:42:10.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=9, tag_tid=3, entry_tid=1' 2024-09-11T06:42:10.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=9, tag_tid=3, entry_tid=1' -a 'object_number=9, tag_tid=3, entry_tid=1' = 'object_number=9, tag_tid=3, entry_tid=1' 2024-09-11T06:42:10.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:42:10.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_snap_present cluster1 mirror image_31 snap 2024-09-11T06:42:10.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:42:10.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:10.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_31 2024-09-11T06:42:10.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:42:10.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:42:10.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T06:42:10.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:42:11.416 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 info mirror/image_31@snap 2024-09-11T06:42:11.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:rbd image 'image_31': 2024-09-11T06:42:11.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: size 128 MiB in 32 objects 2024-09-11T06:42:11.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: order 22 (4 MiB objects) 2024-09-11T06:42:11.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: snapshot_count: 1 2024-09-11T06:42:11.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: id: 10a632df6ae8 2024-09-11T06:42:11.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: block_name_prefix: rbd_data.10a632df6ae8 2024-09-11T06:42:11.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: format: 2 2024-09-11T06:42:11.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: features: layering, exclusive-lock, journaling 2024-09-11T06:42:11.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: op_features: 2024-09-11T06:42:11.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: flags: 2024-09-11T06:42:11.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: create_timestamp: Wed Sep 11 06:35:28 2024 2024-09-11T06:42:11.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: access_timestamp: Wed Sep 11 06:35:28 2024 2024-09-11T06:42:11.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: modify_timestamp: Wed Sep 11 06:38:27 2024 2024-09-11T06:42:11.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: protected: False 2024-09-11T06:42:11.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: journal: 10a632df6ae8 2024-09-11T06:42:11.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring state: enabled 2024-09-11T06:42:11.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring mode: journal 2024-09-11T06:42:11.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring global id: 90400c5f-fe88-4e51-bc9d-8872fda72709 2024-09-11T06:42:11.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring primary: false 2024-09-11T06:42:11.485 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:42:11.486 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ compare_image_snaps mirror image_31 snap 2024-09-11T06:42:11.486 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:11.486 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_31 2024-09-11T06:42:11.486 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:42:11.486 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local ret=0 2024-09-11T06:42:11.486 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local rmt_export=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_31.export 2024-09-11T06:42:11.486 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loc_export=/tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_31.export 2024-09-11T06:42:11.486 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_31.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_31.export 2024-09-11T06:42:11.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror export image_31@snap /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_31.export 2024-09-11T06:42:11.677 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T06:42:11.684 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 -p mirror export image_31@snap /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_31.export 2024-09-11T06:42:12.068 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T06:42:12.082 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmp /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_31.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_31.export 2024-09-11T06:42:12.161 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_31.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_31.export 2024-09-11T06:42:12.203 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:42:12.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:42:12.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_32 2024-09-11T06:42:12.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_snap cluster2 mirror image_32 snap 2024-09-11T06:42:12.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:42:12.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:12.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_32 2024-09-11T06:42:12.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:42:12.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror snap create image_32@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T06:42:12.741 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_started cluster1 mirror image_32 2024-09-11T06:42:12.741 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:42:12.741 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:12.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_32 2024-09-11T06:42:12.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_state cluster1 mirror image_32 started 2024-09-11T06:42:12.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:42:12.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:12.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_32 2024-09-11T06:42:12.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=started 2024-09-11T06:42:12.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:42:12.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:42:12.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:42:13.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_replay_state cluster1 mirror image_32 started 2024-09-11T06:42:13.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:42:13.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:13.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_32 2024-09-11T06:42:13.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=started 2024-09-11T06:42:13.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local status_result 2024-09-11T06:42:13.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=stopped 2024-09-11T06:42:13.745 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_32 2024-09-11T06:42:13.745 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_instance=cluster1 2024-09-11T06:42:13.745 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:42:13.745 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:42:13.745 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance=cluster1 2024-09-11T06:42:13.745 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local loop_instance 2024-09-11T06:42:13.746 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:42:13.746 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sleep 0 2024-09-11T06:42:13.746 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -i state 2024-09-11T06:42:13.746 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:42:13.747 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ seq 0 1 2024-09-11T06:42:13.749 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:42:13.749 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_32 2024-09-11T06:42:13.749 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1:0 2024-09-11T06:42:13.749 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:42:13.749 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:42:13.750 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:42:13.750 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:42:13.750 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=cluster 2024-09-11T06:42:13.750 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:42:13.750 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:42:13.750 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:42:13.750 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:42:13.750 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval cluster=cluster1 2024-09-11T06:42:13.750 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:42:13.750 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:42:13.750 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:42:13.750 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:42:13.750 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local local_cluster=cluster1:0 2024-09-11T06:42:13.750 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster=cluster1 2024-09-11T06:42:13.750 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance 2024-09-11T06:42:13.751 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:42:13.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local val=cluster1:0 2024-09-11T06:42:13.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T06:42:13.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance_var_name=instance 2024-09-11T06:42:13.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:42:13.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:42:13.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T06:42:13.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval local_cluster=cluster1 2024-09-11T06:42:13.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ local_cluster=cluster1 2024-09-11T06:42:13.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval instance=0 2024-09-11T06:42:13.753 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ instance=0 2024-09-11T06:42:13.753 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:42:13.772 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:42:13.772 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:42:13.772 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:42:13.772 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_32 2024-09-11T06:42:13.900 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:Invalid command: unused arguments: ['mirror/image_32'] 2024-09-11T06:42:13.900 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T06:42:13.900 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:admin_socket: invalid command 2024-09-11T06:42:13.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:42:13.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_32 2024-09-11T06:42:13.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1:1 2024-09-11T06:42:13.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:42:13.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:42:13.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-11T06:42:13.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:1 2024-09-11T06:42:13.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=cluster 2024-09-11T06:42:13.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:42:13.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:42:13.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=1 2024-09-11T06:42:13.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T06:42:13.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval cluster=cluster1 2024-09-11T06:42:13.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:42:13.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=1 2024-09-11T06:42:13.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=1 2024-09-11T06:42:13.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-11T06:42:13.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local local_cluster=cluster1:1 2024-09-11T06:42:13.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster=cluster1 2024-09-11T06:42:13.905 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance 2024-09-11T06:42:13.905 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T06:42:13.905 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local val=cluster1:1 2024-09-11T06:42:13.905 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T06:42:13.905 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance_var_name=instance 2024-09-11T06:42:13.905 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:42:13.905 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=1 2024-09-11T06:42:13.905 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-11T06:42:13.905 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval local_cluster=cluster1 2024-09-11T06:42:13.905 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ local_cluster=cluster1 2024-09-11T06:42:13.905 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval instance=1 2024-09-11T06:42:13.906 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ instance=1 2024-09-11T06:42:13.906 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T06:42:13.929 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:42:13.930 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:42:13.930 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:42:13.930 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_32 2024-09-11T06:42:14.057 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ return 0 2024-09-11T06:42:14.058 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T06:42:14.058 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ echo ' "state": "Replaying"' 2024-09-11T06:42:14.058 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ grep -i Replaying 2024-09-11T06:42:14.060 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: "state": "Replaying" 2024-09-11T06:42:14.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_state=started 2024-09-11T06:42:14.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test started = started 2024-09-11T06:42:14.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:42:14.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_32 2024-09-11T06:42:14.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:42:14.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:42:14.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:14.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_32 2024-09-11T06:42:14.063 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' journal = journal ']' 2024-09-11T06:42:14.063 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_32 2024-09-11T06:42:14.063 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:42:14.063 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:42:14.063 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:14.063 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_32 2024-09-11T06:42:14.063 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-11T06:42:14.063 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-11T06:42:14.063 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:42:14.063 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:42:14.063 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:42:14.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror image_32 2024-09-11T06:42:14.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:42:14.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:14.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_32 2024-09-11T06:42:14.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:42:14.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n image_32 ']' 2024-09-11T06:42:14.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/image_32' 2024-09-11T06:42:14.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_32 2024-09-11T06:42:14.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:42:14.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:42:14.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:42:14.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:42:14.264 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:42:14.264 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:42:14.264 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:42:14.264 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:42:14.264 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:42:14.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:42:14.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_32 2024-09-11T06:42:14.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:42:14.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:42:14.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:42:14.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:42:14.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:42:14.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:42:14.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:42:14.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:42:14.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:42:14.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:42:14.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:42:14.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:42:14.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:42:14.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:42:14.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:42:14.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:42:14.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:42:14.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:42:14.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:42:14.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:42:14.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:42:14.269 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:42:14.269 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:42:14.269 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:42:14.269 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:42:14.269 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:42:14.269 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:42:14.269 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:42:14.269 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:42:14.269 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:42:14.294 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:42:14.294 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:42:14.294 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:42:14.294 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_32 2024-09-11T06:42:14.420 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:Invalid command: unused arguments: ['mirror/image_32'] 2024-09-11T06:42:14.420 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd mirror flush : flush rbd mirror 2024-09-11T06:42:14.420 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:admin_socket: invalid command 2024-09-11T06:42:14.421 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:42:14.422 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_32 2024-09-11T06:42:14.422 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:1 2024-09-11T06:42:14.422 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:42:14.422 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:42:14.422 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-11T06:42:14.422 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:1 2024-09-11T06:42:14.422 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:42:14.422 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:42:14.422 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:42:14.422 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=1 2024-09-11T06:42:14.422 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-11T06:42:14.422 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:42:14.423 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:42:14.423 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=1 2024-09-11T06:42:14.423 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=1 2024-09-11T06:42:14.424 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-11T06:42:14.424 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:1 2024-09-11T06:42:14.424 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:42:14.424 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:42:14.425 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T06:42:14.425 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:1 2024-09-11T06:42:14.425 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:42:14.425 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:42:14.425 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:42:14.425 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=1 2024-09-11T06:42:14.425 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T06:42:14.425 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:42:14.425 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:42:14.425 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=1 2024-09-11T06:42:14.425 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=1 2024-09-11T06:42:14.425 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T06:42:14.449 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:42:14.449 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:42:14.449 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:42:14.449 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_32 2024-09-11T06:42:14.598 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:42:14.600 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror image_32 2024-09-11T06:42:14.600 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:42:14.600 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:42:14.600 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_32 2024-09-11T06:42:14.600 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror image_32 '' 2024-09-11T06:42:14.600 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:42:14.600 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:42:14.600 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_32 2024-09-11T06:42:14.600 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:42:14.600 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-image_32.status 2024-09-11T06:42:14.601 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-image_32.status 2024-09-11T06:42:14.601 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:42:14.603 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_32.status 2024-09-11T06:42:14.603 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_32 2024-09-11T06:42:14.603 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_32.status 2024-09-11T06:42:14.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 1 2024-09-11T06:42:14.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 1 2024-09-11T06:42:14.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:42:14.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:42:14.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:42:14.670 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_32.status 2024-09-11T06:42:14.672 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-09-11T06:42:14.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror image_32 2024-09-11T06:42:14.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:42:14.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:42:14.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_32 2024-09-11T06:42:14.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror image_32 '..*' 2024-09-11T06:42:14.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:42:14.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:42:14.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_32 2024-09-11T06:42:14.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:42:14.674 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-image_32.status 2024-09-11T06:42:14.674 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-image_32.status 2024-09-11T06:42:14.674 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:42:14.674 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_32.status 2024-09-11T06:42:14.674 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_32 2024-09-11T06:42:14.674 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_32.status 2024-09-11T06:42:14.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 1 2024-09-11T06:42:14.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 1 2024-09-11T06:42:14.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:42:14.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:42:14.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:42:14.721 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_32.status 2024-09-11T06:42:14.724 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-09-11T06:42:14.724 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=3, entry_tid=1' 2024-09-11T06:42:14.724 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:42:14.724 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_snap_present cluster1 mirror image_32 snap 2024-09-11T06:42:14.724 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:42:14.724 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:14.724 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_32 2024-09-11T06:42:14.724 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:42:14.724 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:42:14.725 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T06:42:14.725 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:42:15.724 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 info mirror/image_32@snap 2024-09-11T06:42:15.792 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:rbd image 'image_32': 2024-09-11T06:42:15.792 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: size 128 MiB in 32 objects 2024-09-11T06:42:15.792 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: order 22 (4 MiB objects) 2024-09-11T06:42:15.793 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: snapshot_count: 1 2024-09-11T06:42:15.793 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: id: 10a9b727d581 2024-09-11T06:42:15.793 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: block_name_prefix: rbd_data.10a9b727d581 2024-09-11T06:42:15.793 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: format: 2 2024-09-11T06:42:15.793 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: features: layering, exclusive-lock, journaling 2024-09-11T06:42:15.793 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: op_features: 2024-09-11T06:42:15.793 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: flags: 2024-09-11T06:42:15.793 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: create_timestamp: Wed Sep 11 06:35:40 2024 2024-09-11T06:42:15.793 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: access_timestamp: Wed Sep 11 06:35:40 2024 2024-09-11T06:42:15.793 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: modify_timestamp: Wed Sep 11 06:36:49 2024 2024-09-11T06:42:15.793 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: protected: False 2024-09-11T06:42:15.793 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: journal: 10a9b727d581 2024-09-11T06:42:15.793 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring state: enabled 2024-09-11T06:42:15.793 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring mode: journal 2024-09-11T06:42:15.794 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring global id: b6b347bb-5127-4a1e-9b38-75af802a016f 2024-09-11T06:42:15.794 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring primary: false 2024-09-11T06:42:15.797 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:42:15.798 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ compare_image_snaps mirror image_32 snap 2024-09-11T06:42:15.798 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:15.798 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_32 2024-09-11T06:42:15.798 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:42:15.798 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local ret=0 2024-09-11T06:42:15.798 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local rmt_export=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_32.export 2024-09-11T06:42:15.798 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loc_export=/tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_32.export 2024-09-11T06:42:15.798 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_32.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_32.export 2024-09-11T06:42:15.798 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror export image_32@snap /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_32.export 2024-09-11T06:42:15.975 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T06:42:15.983 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 -p mirror export image_32@snap /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_32.export 2024-09-11T06:42:16.294 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T06:42:16.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmp /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_32.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_32.export 2024-09-11T06:42:16.386 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_32.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_32.export 2024-09-11T06:42:16.428 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:42:16.428 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:42:16.428 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_33 2024-09-11T06:42:16.428 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_snap cluster2 mirror image_33 snap 2024-09-11T06:42:16.428 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:42:16.428 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:16.428 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_33 2024-09-11T06:42:16.428 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:42:16.428 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror snap create image_33@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T06:42:17.746 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_started cluster1 mirror image_33 2024-09-11T06:42:17.746 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:42:17.746 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:17.746 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_33 2024-09-11T06:42:17.747 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_state cluster1 mirror image_33 started 2024-09-11T06:42:17.747 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:42:17.747 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:17.747 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_33 2024-09-11T06:42:17.747 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=started 2024-09-11T06:42:17.747 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:42:17.747 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:42:17.747 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:42:18.748 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_replay_state cluster1 mirror image_33 started 2024-09-11T06:42:18.748 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:42:18.748 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:18.748 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_33 2024-09-11T06:42:18.748 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=started 2024-09-11T06:42:18.748 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local status_result 2024-09-11T06:42:18.748 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=stopped 2024-09-11T06:42:18.750 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_33 2024-09-11T06:42:18.750 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -i state 2024-09-11T06:42:18.750 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_instance=cluster1 2024-09-11T06:42:18.750 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:42:18.750 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:42:18.750 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance=cluster1 2024-09-11T06:42:18.750 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local loop_instance 2024-09-11T06:42:18.750 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:42:18.750 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sleep 0 2024-09-11T06:42:18.751 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:42:18.751 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ seq 0 1 2024-09-11T06:42:18.754 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:42:18.754 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_33 2024-09-11T06:42:18.754 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1:0 2024-09-11T06:42:18.754 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:42:18.754 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:42:18.754 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:42:18.754 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:42:18.754 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=cluster 2024-09-11T06:42:18.754 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:42:18.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:42:18.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:42:18.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:42:18.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval cluster=cluster1 2024-09-11T06:42:18.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:42:18.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:42:18.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:42:18.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:42:18.756 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local local_cluster=cluster1:0 2024-09-11T06:42:18.756 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster=cluster1 2024-09-11T06:42:18.756 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance 2024-09-11T06:42:18.756 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:42:18.757 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local val=cluster1:0 2024-09-11T06:42:18.757 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T06:42:18.757 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance_var_name=instance 2024-09-11T06:42:18.757 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:42:18.757 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:42:18.758 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T06:42:18.758 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval local_cluster=cluster1 2024-09-11T06:42:18.758 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ local_cluster=cluster1 2024-09-11T06:42:18.758 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval instance=0 2024-09-11T06:42:18.758 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ instance=0 2024-09-11T06:42:18.758 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:42:18.779 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:42:18.779 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:42:18.779 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:42:18.779 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_33 2024-09-11T06:42:18.923 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ return 0 2024-09-11T06:42:18.923 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T06:42:18.924 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ echo ' "state": "Replaying"' 2024-09-11T06:42:18.924 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ grep -i Replaying 2024-09-11T06:42:18.925 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: "state": "Replaying" 2024-09-11T06:42:18.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_state=started 2024-09-11T06:42:18.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test started = started 2024-09-11T06:42:18.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:42:18.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_33 2024-09-11T06:42:18.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:42:18.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:42:18.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:18.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_33 2024-09-11T06:42:18.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' journal = journal ']' 2024-09-11T06:42:18.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_33 2024-09-11T06:42:18.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:42:18.928 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:42:18.928 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:18.928 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_33 2024-09-11T06:42:18.928 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-11T06:42:18.928 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-11T06:42:18.928 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:42:18.928 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:42:18.928 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:42:19.129 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror image_33 2024-09-11T06:42:19.129 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:42:19.129 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:19.129 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_33 2024-09-11T06:42:19.129 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:42:19.130 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n image_33 ']' 2024-09-11T06:42:19.130 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/image_33' 2024-09-11T06:42:19.130 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_33 2024-09-11T06:42:19.130 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:42:19.130 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:42:19.130 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:42:19.130 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:42:19.130 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:42:19.130 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:42:19.130 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:42:19.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:42:19.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:42:19.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:42:19.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_33 2024-09-11T06:42:19.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:42:19.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:42:19.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:42:19.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:42:19.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:42:19.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:42:19.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:42:19.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:42:19.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:42:19.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:42:19.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:42:19.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:42:19.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:42:19.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:42:19.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:42:19.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:42:19.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:42:19.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:42:19.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:42:19.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:42:19.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:42:19.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:42:19.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:42:19.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:42:19.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:42:19.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:42:19.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:42:19.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:42:19.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:42:19.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:42:19.160 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:42:19.161 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:42:19.161 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:42:19.161 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_33 2024-09-11T06:42:19.300 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:42:19.301 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror image_33 2024-09-11T06:42:19.301 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:42:19.301 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:42:19.301 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_33 2024-09-11T06:42:19.301 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror image_33 '' 2024-09-11T06:42:19.301 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:42:19.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:42:19.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_33 2024-09-11T06:42:19.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:42:19.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-image_33.status 2024-09-11T06:42:19.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-image_33.status 2024-09-11T06:42:19.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:42:19.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_33.status 2024-09-11T06:42:19.305 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_33 2024-09-11T06:42:19.305 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_33.status 2024-09-11T06:42:19.356 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 0 2024-09-11T06:42:19.356 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 0 2024-09-11T06:42:19.356 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:42:19.356 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:42:19.357 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:42:19.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_33.status 2024-09-11T06:42:19.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=1, tag_tid=3, entry_tid=1' 2024-09-11T06:42:19.363 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror image_33 2024-09-11T06:42:19.363 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:42:19.363 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:42:19.363 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_33 2024-09-11T06:42:19.363 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror image_33 '..*' 2024-09-11T06:42:19.363 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:42:19.363 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:42:19.363 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_33 2024-09-11T06:42:19.364 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:42:19.364 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-image_33.status 2024-09-11T06:42:19.364 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-image_33.status 2024-09-11T06:42:19.364 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:42:19.364 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_33.status 2024-09-11T06:42:19.364 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_33 2024-09-11T06:42:19.365 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_33.status 2024-09-11T06:42:19.409 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 0 2024-09-11T06:42:19.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 0 2024-09-11T06:42:19.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:42:19.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:42:19.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0], [object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:42:19.413 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_33.status 2024-09-11T06:42:19.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=1, tag_tid=3, entry_tid=1' 2024-09-11T06:42:19.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=1, tag_tid=3, entry_tid=1' -a 'object_number=1, tag_tid=3, entry_tid=1' = 'object_number=1, tag_tid=3, entry_tid=1' 2024-09-11T06:42:19.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:42:19.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_snap_present cluster1 mirror image_33 snap 2024-09-11T06:42:19.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:42:19.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:19.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_33 2024-09-11T06:42:19.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:42:19.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:42:19.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T06:42:19.415 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:42:20.416 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 info mirror/image_33@snap 2024-09-11T06:42:20.484 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:rbd image 'image_33': 2024-09-11T06:42:20.484 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: size 128 MiB in 32 objects 2024-09-11T06:42:20.484 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: order 22 (4 MiB objects) 2024-09-11T06:42:20.484 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: snapshot_count: 1 2024-09-11T06:42:20.484 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: id: 10a691127670 2024-09-11T06:42:20.484 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: block_name_prefix: rbd_data.10a691127670 2024-09-11T06:42:20.485 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: format: 2 2024-09-11T06:42:20.485 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: features: layering, exclusive-lock, journaling 2024-09-11T06:42:20.485 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: op_features: 2024-09-11T06:42:20.485 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: flags: 2024-09-11T06:42:20.485 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: create_timestamp: Wed Sep 11 06:35:47 2024 2024-09-11T06:42:20.485 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: access_timestamp: Wed Sep 11 06:35:47 2024 2024-09-11T06:42:20.485 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: modify_timestamp: Wed Sep 11 06:37:03 2024 2024-09-11T06:42:20.485 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: protected: False 2024-09-11T06:42:20.485 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: journal: 10a691127670 2024-09-11T06:42:20.485 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring state: enabled 2024-09-11T06:42:20.485 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring mode: journal 2024-09-11T06:42:20.485 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring global id: 9f836c94-b174-4c53-b32d-754f78a70ada 2024-09-11T06:42:20.485 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring primary: false 2024-09-11T06:42:20.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:42:20.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ compare_image_snaps mirror image_33 snap 2024-09-11T06:42:20.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:20.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_33 2024-09-11T06:42:20.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:42:20.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local ret=0 2024-09-11T06:42:20.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local rmt_export=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_33.export 2024-09-11T06:42:20.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loc_export=/tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_33.export 2024-09-11T06:42:20.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_33.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_33.export 2024-09-11T06:42:20.490 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror export image_33@snap /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_33.export 2024-09-11T06:42:20.655 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T06:42:20.664 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 -p mirror export image_33@snap /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_33.export 2024-09-11T06:42:20.953 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T06:42:20.964 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmp /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_33.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_33.export 2024-09-11T06:42:21.044 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_33.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_33.export 2024-09-11T06:42:21.086 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:42:21.086 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:42:21.086 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_34 2024-09-11T06:42:21.086 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_snap cluster2 mirror image_34 snap 2024-09-11T06:42:21.086 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:42:21.086 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:21.086 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_34 2024-09-11T06:42:21.086 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:42:21.087 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror snap create image_34@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T06:42:21.757 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_started cluster1 mirror image_34 2024-09-11T06:42:21.757 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:42:21.757 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:21.757 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_34 2024-09-11T06:42:21.757 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_state cluster1 mirror image_34 started 2024-09-11T06:42:21.757 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:42:21.758 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:21.758 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_34 2024-09-11T06:42:21.758 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=started 2024-09-11T06:42:21.758 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:42:21.758 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:42:21.758 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:42:22.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_replay_state cluster1 mirror image_34 started 2024-09-11T06:42:22.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:42:22.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:22.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_34 2024-09-11T06:42:22.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=started 2024-09-11T06:42:22.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local status_result 2024-09-11T06:42:22.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=stopped 2024-09-11T06:42:22.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_34 2024-09-11T06:42:22.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -i state 2024-09-11T06:42:22.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_instance=cluster1 2024-09-11T06:42:22.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:42:22.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:42:22.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance=cluster1 2024-09-11T06:42:22.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local loop_instance 2024-09-11T06:42:22.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:42:22.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sleep 0 2024-09-11T06:42:22.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:42:22.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ seq 0 1 2024-09-11T06:42:22.764 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:42:22.764 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_34 2024-09-11T06:42:22.764 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1:0 2024-09-11T06:42:22.764 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:42:22.764 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:42:22.764 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:42:22.764 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:42:22.765 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=cluster 2024-09-11T06:42:22.765 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:42:22.765 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:42:22.765 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:42:22.765 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:42:22.765 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval cluster=cluster1 2024-09-11T06:42:22.765 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:42:22.765 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:42:22.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:42:22.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:42:22.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local local_cluster=cluster1:0 2024-09-11T06:42:22.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster=cluster1 2024-09-11T06:42:22.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance 2024-09-11T06:42:22.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:42:22.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local val=cluster1:0 2024-09-11T06:42:22.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T06:42:22.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance_var_name=instance 2024-09-11T06:42:22.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:42:22.768 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:42:22.768 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T06:42:22.768 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval local_cluster=cluster1 2024-09-11T06:42:22.768 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ local_cluster=cluster1 2024-09-11T06:42:22.768 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval instance=0 2024-09-11T06:42:22.768 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ instance=0 2024-09-11T06:42:22.768 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:42:22.793 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:42:22.794 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:42:22.794 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:42:22.794 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_34 2024-09-11T06:42:22.934 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:Invalid command: unused arguments: ['mirror/image_34'] 2024-09-11T06:42:22.934 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T06:42:22.934 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:admin_socket: invalid command 2024-09-11T06:42:22.937 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:42:22.937 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_34 2024-09-11T06:42:22.937 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1:1 2024-09-11T06:42:22.937 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:42:22.937 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:42:22.937 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-11T06:42:22.937 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:1 2024-09-11T06:42:22.937 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=cluster 2024-09-11T06:42:22.937 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:42:22.937 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:42:22.937 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=1 2024-09-11T06:42:22.938 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T06:42:22.938 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval cluster=cluster1 2024-09-11T06:42:22.938 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:42:22.938 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=1 2024-09-11T06:42:22.938 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=1 2024-09-11T06:42:22.938 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-11T06:42:22.938 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local local_cluster=cluster1:1 2024-09-11T06:42:22.938 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster=cluster1 2024-09-11T06:42:22.938 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance 2024-09-11T06:42:22.938 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T06:42:22.938 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local val=cluster1:1 2024-09-11T06:42:22.938 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T06:42:22.938 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance_var_name=instance 2024-09-11T06:42:22.938 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:42:22.938 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=1 2024-09-11T06:42:22.938 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-11T06:42:22.939 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval local_cluster=cluster1 2024-09-11T06:42:22.939 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ local_cluster=cluster1 2024-09-11T06:42:22.939 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval instance=1 2024-09-11T06:42:22.939 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ instance=1 2024-09-11T06:42:22.939 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T06:42:22.961 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:42:22.961 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:42:22.961 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:42:22.961 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_34 2024-09-11T06:42:23.091 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ return 0 2024-09-11T06:42:23.091 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T06:42:23.092 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ echo ' "state": "Replaying"' 2024-09-11T06:42:23.092 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ grep -i Replaying 2024-09-11T06:42:23.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: "state": "Replaying" 2024-09-11T06:42:23.096 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_state=started 2024-09-11T06:42:23.096 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test started = started 2024-09-11T06:42:23.096 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:42:23.096 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_34 2024-09-11T06:42:23.096 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:42:23.096 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:42:23.096 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:23.096 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_34 2024-09-11T06:42:23.096 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' journal = journal ']' 2024-09-11T06:42:23.096 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_34 2024-09-11T06:42:23.096 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:42:23.096 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:42:23.097 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:23.097 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_34 2024-09-11T06:42:23.097 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-11T06:42:23.097 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-11T06:42:23.097 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:42:23.097 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:42:23.097 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:42:23.297 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror image_34 2024-09-11T06:42:23.298 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:42:23.298 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:23.298 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_34 2024-09-11T06:42:23.298 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:42:23.298 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n image_34 ']' 2024-09-11T06:42:23.298 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/image_34' 2024-09-11T06:42:23.298 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_34 2024-09-11T06:42:23.298 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:42:23.298 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:42:23.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:42:23.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:42:23.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:42:23.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:42:23.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:42:23.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:42:23.300 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:42:23.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:42:23.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_34 2024-09-11T06:42:23.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:42:23.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:42:23.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:42:23.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:42:23.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:42:23.305 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:42:23.305 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:42:23.305 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:42:23.305 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:42:23.305 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:42:23.305 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:42:23.305 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:42:23.305 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:42:23.305 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:42:23.305 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:42:23.305 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:42:23.305 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:42:23.306 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:42:23.306 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:42:23.306 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:42:23.306 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:42:23.306 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:42:23.306 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:42:23.306 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:42:23.306 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:42:23.306 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:42:23.306 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:42:23.306 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:42:23.306 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:42:23.306 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:42:23.334 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:42:23.334 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:42:23.334 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:42:23.335 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_34 2024-09-11T06:42:23.454 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:Invalid command: unused arguments: ['mirror/image_34'] 2024-09-11T06:42:23.454 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd mirror flush : flush rbd mirror 2024-09-11T06:42:23.454 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:admin_socket: invalid command 2024-09-11T06:42:23.458 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:42:23.458 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_34 2024-09-11T06:42:23.458 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:1 2024-09-11T06:42:23.458 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:42:23.458 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:42:23.459 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-11T06:42:23.459 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:1 2024-09-11T06:42:23.459 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:42:23.459 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:42:23.459 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:42:23.459 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=1 2024-09-11T06:42:23.459 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-11T06:42:23.459 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:42:23.459 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:42:23.459 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=1 2024-09-11T06:42:23.459 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=1 2024-09-11T06:42:23.459 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-11T06:42:23.459 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:1 2024-09-11T06:42:23.459 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:42:23.459 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:42:23.460 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T06:42:23.460 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:1 2024-09-11T06:42:23.460 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:42:23.460 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:42:23.460 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:42:23.460 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=1 2024-09-11T06:42:23.460 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T06:42:23.460 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:42:23.460 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:42:23.460 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=1 2024-09-11T06:42:23.460 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=1 2024-09-11T06:42:23.460 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T06:42:23.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:42:23.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:42:23.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:42:23.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_34 2024-09-11T06:42:23.623 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:42:23.624 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror image_34 2024-09-11T06:42:23.624 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:42:23.624 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:42:23.624 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_34 2024-09-11T06:42:23.624 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror image_34 '' 2024-09-11T06:42:23.624 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:42:23.625 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:42:23.625 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_34 2024-09-11T06:42:23.625 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:42:23.625 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-image_34.status 2024-09-11T06:42:23.625 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-image_34.status 2024-09-11T06:42:23.625 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:42:23.626 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_34.status 2024-09-11T06:42:23.627 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_34 2024-09-11T06:42:23.627 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_34.status 2024-09-11T06:42:23.684 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 1 2024-09-11T06:42:23.684 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 1 2024-09-11T06:42:23.684 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:42:23.684 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:42:23.684 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:42:23.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_34.status 2024-09-11T06:42:23.689 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-09-11T06:42:23.690 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror image_34 2024-09-11T06:42:23.690 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:42:23.690 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:42:23.690 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_34 2024-09-11T06:42:23.690 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror image_34 '..*' 2024-09-11T06:42:23.690 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:42:23.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:42:23.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_34 2024-09-11T06:42:23.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:42:23.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-image_34.status 2024-09-11T06:42:23.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-image_34.status 2024-09-11T06:42:23.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:42:23.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_34.status 2024-09-11T06:42:23.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_34 2024-09-11T06:42:23.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_34.status 2024-09-11T06:42:23.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 1 2024-09-11T06:42:23.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 1 2024-09-11T06:42:23.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:42:23.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:42:23.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:42:23.741 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_34.status 2024-09-11T06:42:23.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-09-11T06:42:23.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=3, entry_tid=1' 2024-09-11T06:42:23.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:42:23.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_snap_present cluster1 mirror image_34 snap 2024-09-11T06:42:23.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:42:23.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:23.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_34 2024-09-11T06:42:23.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:42:23.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:42:23.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T06:42:23.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:42:24.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 info mirror/image_34@snap 2024-09-11T06:42:24.810 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:rbd image 'image_34': 2024-09-11T06:42:24.810 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: size 128 MiB in 32 objects 2024-09-11T06:42:24.810 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: order 22 (4 MiB objects) 2024-09-11T06:42:24.810 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: snapshot_count: 1 2024-09-11T06:42:24.810 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: id: 10a978be3e8c 2024-09-11T06:42:24.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: block_name_prefix: rbd_data.10a978be3e8c 2024-09-11T06:42:24.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: format: 2 2024-09-11T06:42:24.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: features: layering, exclusive-lock, journaling 2024-09-11T06:42:24.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: op_features: 2024-09-11T06:42:24.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: flags: 2024-09-11T06:42:24.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: create_timestamp: Wed Sep 11 06:35:55 2024 2024-09-11T06:42:24.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: access_timestamp: Wed Sep 11 06:35:55 2024 2024-09-11T06:42:24.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: modify_timestamp: Wed Sep 11 06:38:00 2024 2024-09-11T06:42:24.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: protected: False 2024-09-11T06:42:24.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: journal: 10a978be3e8c 2024-09-11T06:42:24.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring state: enabled 2024-09-11T06:42:24.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring mode: journal 2024-09-11T06:42:24.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring global id: 8a994a1b-1259-4b15-8bcb-eb5add230a8d 2024-09-11T06:42:24.812 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring primary: false 2024-09-11T06:42:24.815 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:42:24.815 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ compare_image_snaps mirror image_34 snap 2024-09-11T06:42:24.815 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:24.815 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_34 2024-09-11T06:42:24.815 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:42:24.815 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local ret=0 2024-09-11T06:42:24.815 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local rmt_export=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_34.export 2024-09-11T06:42:24.816 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loc_export=/tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_34.export 2024-09-11T06:42:24.816 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_34.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_34.export 2024-09-11T06:42:24.816 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror export image_34@snap /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_34.export 2024-09-11T06:42:25.060 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T06:42:25.069 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 -p mirror export image_34@snap /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_34.export 2024-09-11T06:42:25.409 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T06:42:25.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmp /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_34.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_34.export 2024-09-11T06:42:25.498 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_34.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_34.export 2024-09-11T06:42:25.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:42:25.543 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:42:25.543 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_35 2024-09-11T06:42:25.543 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_snap cluster2 mirror image_35 snap 2024-09-11T06:42:25.543 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:42:25.543 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:25.543 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_35 2024-09-11T06:42:25.543 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:42:25.543 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror snap create image_35@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T06:42:25.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_started cluster1 mirror image_35 2024-09-11T06:42:25.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:42:25.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:25.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_35 2024-09-11T06:42:25.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_state cluster1 mirror image_35 started 2024-09-11T06:42:25.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:42:25.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:25.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_35 2024-09-11T06:42:25.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=started 2024-09-11T06:42:25.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:42:25.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:42:25.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:42:26.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_replay_state cluster1 mirror image_35 started 2024-09-11T06:42:26.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:42:26.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:26.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_35 2024-09-11T06:42:26.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=started 2024-09-11T06:42:26.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local status_result 2024-09-11T06:42:26.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=stopped 2024-09-11T06:42:26.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_35 2024-09-11T06:42:26.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -i state 2024-09-11T06:42:26.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_instance=cluster1 2024-09-11T06:42:26.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:42:26.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:42:26.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance=cluster1 2024-09-11T06:42:26.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local loop_instance 2024-09-11T06:42:26.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:42:26.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sleep 0 2024-09-11T06:42:26.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:42:26.764 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ seq 0 1 2024-09-11T06:42:26.765 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:42:26.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_35 2024-09-11T06:42:26.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1:0 2024-09-11T06:42:26.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:42:26.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:42:26.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:42:26.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:42:26.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=cluster 2024-09-11T06:42:26.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:42:26.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:42:26.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:42:26.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:42:26.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval cluster=cluster1 2024-09-11T06:42:26.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:42:26.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:42:26.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:42:26.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:42:26.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local local_cluster=cluster1:0 2024-09-11T06:42:26.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster=cluster1 2024-09-11T06:42:26.768 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance 2024-09-11T06:42:26.768 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:42:26.768 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local val=cluster1:0 2024-09-11T06:42:26.769 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T06:42:26.769 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance_var_name=instance 2024-09-11T06:42:26.769 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:42:26.769 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:42:26.769 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T06:42:26.769 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval local_cluster=cluster1 2024-09-11T06:42:26.769 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ local_cluster=cluster1 2024-09-11T06:42:26.769 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval instance=0 2024-09-11T06:42:26.769 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ instance=0 2024-09-11T06:42:26.769 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:42:26.788 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:42:26.788 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:42:26.788 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:42:26.789 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_35 2024-09-11T06:42:26.929 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ return 0 2024-09-11T06:42:26.929 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T06:42:26.930 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ echo ' "state": "Replaying"' 2024-09-11T06:42:26.930 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ grep -i Replaying 2024-09-11T06:42:26.932 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: "state": "Replaying" 2024-09-11T06:42:26.933 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_state=started 2024-09-11T06:42:26.933 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test started = started 2024-09-11T06:42:26.934 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:42:26.934 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_35 2024-09-11T06:42:26.934 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:42:26.934 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:42:26.934 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:26.934 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_35 2024-09-11T06:42:26.934 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' journal = journal ']' 2024-09-11T06:42:26.934 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_35 2024-09-11T06:42:26.934 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:42:26.934 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:42:26.934 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:26.934 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_35 2024-09-11T06:42:26.935 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-11T06:42:26.935 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-11T06:42:26.935 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:42:26.935 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:42:26.935 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:42:27.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror image_35 2024-09-11T06:42:27.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:42:27.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:27.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_35 2024-09-11T06:42:27.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:42:27.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n image_35 ']' 2024-09-11T06:42:27.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/image_35' 2024-09-11T06:42:27.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_35 2024-09-11T06:42:27.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:42:27.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:42:27.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:42:27.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:42:27.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:42:27.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:42:27.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:42:27.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:42:27.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:42:27.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:42:27.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_35 2024-09-11T06:42:27.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:42:27.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:42:27.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:42:27.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:42:27.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:42:27.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:42:27.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:42:27.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:42:27.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:42:27.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:42:27.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:42:27.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:42:27.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:42:27.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:42:27.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:42:27.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:42:27.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:42:27.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:42:27.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:42:27.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:42:27.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:42:27.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:42:27.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:42:27.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:42:27.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:42:27.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:42:27.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:42:27.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:42:27.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:42:27.143 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:42:27.166 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:42:27.166 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:42:27.166 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:42:27.166 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_35 2024-09-11T06:42:27.307 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:42:27.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror image_35 2024-09-11T06:42:27.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:42:27.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:42:27.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_35 2024-09-11T06:42:27.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror image_35 '' 2024-09-11T06:42:27.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:42:27.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:42:27.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_35 2024-09-11T06:42:27.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:42:27.310 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-image_35.status 2024-09-11T06:42:27.310 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-image_35.status 2024-09-11T06:42:27.310 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:42:27.312 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_35.status 2024-09-11T06:42:27.313 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_35 2024-09-11T06:42:27.313 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_35.status 2024-09-11T06:42:27.358 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 2 2024-09-11T06:42:27.358 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 2 2024-09-11T06:42:27.358 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:42:27.358 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=3, entry_tid=1], [object_number=8, tag_tid=3, entry_tid=0], [object_number=11, tag_tid=2, entry_tid=3], [object_number=10, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:42:27.358 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=9, tag_tid=3, entry_tid=1], [object_number=8, tag_tid=3, entry_tid=0], [object_number=11, tag_tid=2, entry_tid=3], [object_number=10, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:42:27.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_35.status 2024-09-11T06:42:27.363 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=9, tag_tid=3, entry_tid=1' 2024-09-11T06:42:27.364 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror image_35 2024-09-11T06:42:27.364 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:42:27.365 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:42:27.365 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_35 2024-09-11T06:42:27.365 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror image_35 '..*' 2024-09-11T06:42:27.365 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:42:27.365 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:42:27.365 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_35 2024-09-11T06:42:27.365 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:42:27.365 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-image_35.status 2024-09-11T06:42:27.365 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-image_35.status 2024-09-11T06:42:27.365 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:42:27.366 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_35.status 2024-09-11T06:42:27.366 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_35 2024-09-11T06:42:27.366 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_35.status 2024-09-11T06:42:27.411 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 2 2024-09-11T06:42:27.411 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 2 2024-09-11T06:42:27.411 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:42:27.411 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=3, entry_tid=1], [object_number=8, tag_tid=3, entry_tid=0], [object_number=11, tag_tid=2, entry_tid=3], [object_number=10, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:42:27.411 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=9, tag_tid=3, entry_tid=1], [object_number=8, tag_tid=3, entry_tid=0], [object_number=11, tag_tid=2, entry_tid=3], [object_number=10, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:42:27.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_35.status 2024-09-11T06:42:27.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=9, tag_tid=3, entry_tid=1' 2024-09-11T06:42:27.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=9, tag_tid=3, entry_tid=1' -a 'object_number=9, tag_tid=3, entry_tid=1' = 'object_number=9, tag_tid=3, entry_tid=1' 2024-09-11T06:42:27.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:42:27.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_snap_present cluster1 mirror image_35 snap 2024-09-11T06:42:27.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:42:27.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:27.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_35 2024-09-11T06:42:27.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:42:27.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:42:27.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T06:42:27.418 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:42:28.418 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 info mirror/image_35@snap 2024-09-11T06:42:28.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:rbd image 'image_35': 2024-09-11T06:42:28.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: size 128 MiB in 32 objects 2024-09-11T06:42:28.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: order 22 (4 MiB objects) 2024-09-11T06:42:28.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: snapshot_count: 1 2024-09-11T06:42:28.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: id: 10a68ee30262 2024-09-11T06:42:28.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: block_name_prefix: rbd_data.10a68ee30262 2024-09-11T06:42:28.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: format: 2 2024-09-11T06:42:28.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: features: layering, exclusive-lock, journaling 2024-09-11T06:42:28.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: op_features: 2024-09-11T06:42:28.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: flags: 2024-09-11T06:42:28.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: create_timestamp: Wed Sep 11 06:36:03 2024 2024-09-11T06:42:28.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: access_timestamp: Wed Sep 11 06:36:03 2024 2024-09-11T06:42:28.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: modify_timestamp: Wed Sep 11 06:37:58 2024 2024-09-11T06:42:28.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: protected: False 2024-09-11T06:42:28.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: journal: 10a68ee30262 2024-09-11T06:42:28.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring state: enabled 2024-09-11T06:42:28.490 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring mode: journal 2024-09-11T06:42:28.490 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring global id: 42fc1706-41a3-43aa-9e1f-283776468e54 2024-09-11T06:42:28.490 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring primary: false 2024-09-11T06:42:28.493 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:42:28.493 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ compare_image_snaps mirror image_35 snap 2024-09-11T06:42:28.493 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:28.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_35 2024-09-11T06:42:28.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:42:28.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local ret=0 2024-09-11T06:42:28.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local rmt_export=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_35.export 2024-09-11T06:42:28.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loc_export=/tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_35.export 2024-09-11T06:42:28.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_35.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_35.export 2024-09-11T06:42:28.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror export image_35@snap /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_35.export 2024-09-11T06:42:28.676 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T06:42:28.685 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 -p mirror export image_35@snap /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_35.export 2024-09-11T06:42:29.177 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T06:42:29.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmp /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_35.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_35.export 2024-09-11T06:42:29.269 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_35.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_35.export 2024-09-11T06:42:29.312 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:42:29.312 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:42:29.312 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_36 2024-09-11T06:42:29.313 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_snap cluster2 mirror image_36 snap 2024-09-11T06:42:29.313 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:42:29.313 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:29.313 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_36 2024-09-11T06:42:29.313 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:42:29.313 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror snap create image_36@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T06:42:29.839 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_started cluster1 mirror image_36 2024-09-11T06:42:29.840 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:42:29.840 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:29.840 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_36 2024-09-11T06:42:29.840 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_state cluster1 mirror image_36 started 2024-09-11T06:42:29.840 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:42:29.840 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:29.840 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_36 2024-09-11T06:42:29.840 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=started 2024-09-11T06:42:29.840 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:42:29.840 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:42:29.840 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:42:30.841 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_replay_state cluster1 mirror image_36 started 2024-09-11T06:42:30.842 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:42:30.842 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:30.842 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_36 2024-09-11T06:42:30.842 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=started 2024-09-11T06:42:30.842 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local status_result 2024-09-11T06:42:30.843 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=stopped 2024-09-11T06:42:30.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_36 2024-09-11T06:42:30.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -i state 2024-09-11T06:42:30.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_instance=cluster1 2024-09-11T06:42:30.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:42:30.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:42:30.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance=cluster1 2024-09-11T06:42:30.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local loop_instance 2024-09-11T06:42:30.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:42:30.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sleep 0 2024-09-11T06:42:30.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:42:30.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ seq 0 1 2024-09-11T06:42:30.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:42:30.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_36 2024-09-11T06:42:30.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1:0 2024-09-11T06:42:30.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:42:30.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:42:30.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:42:30.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:42:30.854 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=cluster 2024-09-11T06:42:30.854 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:42:30.854 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:42:30.854 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:42:30.854 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:42:30.854 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval cluster=cluster1 2024-09-11T06:42:30.855 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:42:30.855 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:42:30.855 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:42:30.855 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:42:30.855 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local local_cluster=cluster1:0 2024-09-11T06:42:30.855 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster=cluster1 2024-09-11T06:42:30.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance 2024-09-11T06:42:30.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:42:30.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local val=cluster1:0 2024-09-11T06:42:30.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T06:42:30.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance_var_name=instance 2024-09-11T06:42:30.856 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:42:30.857 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:42:30.857 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T06:42:30.857 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval local_cluster=cluster1 2024-09-11T06:42:30.857 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ local_cluster=cluster1 2024-09-11T06:42:30.857 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval instance=0 2024-09-11T06:42:30.857 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ instance=0 2024-09-11T06:42:30.858 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:42:30.876 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:42:30.876 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:42:30.876 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:42:30.877 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_36 2024-09-11T06:42:31.009 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2024-09-11T06:42:31.010 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T06:42:31.010 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:admin_socket: invalid command 2024-09-11T06:42:31.012 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:42:31.012 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_36 2024-09-11T06:42:31.012 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1:1 2024-09-11T06:42:31.012 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:42:31.012 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:42:31.012 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-11T06:42:31.012 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:1 2024-09-11T06:42:31.012 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=cluster 2024-09-11T06:42:31.012 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:42:31.012 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:42:31.012 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=1 2024-09-11T06:42:31.013 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T06:42:31.013 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval cluster=cluster1 2024-09-11T06:42:31.013 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:42:31.013 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=1 2024-09-11T06:42:31.013 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=1 2024-09-11T06:42:31.014 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-11T06:42:31.014 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local local_cluster=cluster1:1 2024-09-11T06:42:31.014 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster=cluster1 2024-09-11T06:42:31.015 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance 2024-09-11T06:42:31.015 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T06:42:31.015 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local val=cluster1:1 2024-09-11T06:42:31.015 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T06:42:31.015 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance_var_name=instance 2024-09-11T06:42:31.015 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:42:31.015 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=1 2024-09-11T06:42:31.015 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-11T06:42:31.015 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval local_cluster=cluster1 2024-09-11T06:42:31.015 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ local_cluster=cluster1 2024-09-11T06:42:31.016 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval instance=1 2024-09-11T06:42:31.016 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ instance=1 2024-09-11T06:42:31.016 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T06:42:31.037 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:42:31.037 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:42:31.037 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:42:31.038 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_36 2024-09-11T06:42:31.187 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ return 0 2024-09-11T06:42:31.188 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T06:42:31.188 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ echo ' "state": "Replaying"' 2024-09-11T06:42:31.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ grep -i Replaying 2024-09-11T06:42:31.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: "state": "Replaying" 2024-09-11T06:42:31.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_state=started 2024-09-11T06:42:31.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test started = started 2024-09-11T06:42:31.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:42:31.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_36 2024-09-11T06:42:31.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:42:31.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:42:31.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:31.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_36 2024-09-11T06:42:31.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' journal = journal ']' 2024-09-11T06:42:31.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_36 2024-09-11T06:42:31.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:42:31.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:42:31.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:31.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_36 2024-09-11T06:42:31.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-11T06:42:31.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-11T06:42:31.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:42:31.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:42:31.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:42:31.394 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror image_36 2024-09-11T06:42:31.394 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:42:31.395 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:31.395 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_36 2024-09-11T06:42:31.395 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:42:31.395 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n image_36 ']' 2024-09-11T06:42:31.395 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/image_36' 2024-09-11T06:42:31.395 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_36 2024-09-11T06:42:31.395 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:42:31.395 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:42:31.395 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:42:31.395 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:42:31.395 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:42:31.395 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:42:31.396 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:42:31.396 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:42:31.396 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:42:31.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:42:31.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_36 2024-09-11T06:42:31.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:42:31.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:42:31.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:42:31.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:42:31.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:42:31.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:42:31.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:42:31.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:42:31.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:42:31.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:42:31.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:42:31.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:42:31.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:42:31.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:42:31.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:42:31.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:42:31.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:42:31.401 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:42:31.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:42:31.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:42:31.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:42:31.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:42:31.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:42:31.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:42:31.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:42:31.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:42:31.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:42:31.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:42:31.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:42:31.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:42:31.425 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:42:31.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:42:31.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:42:31.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_36 2024-09-11T06:42:31.547 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2024-09-11T06:42:31.547 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd mirror flush : flush rbd mirror 2024-09-11T06:42:31.548 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:admin_socket: invalid command 2024-09-11T06:42:31.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:42:31.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_36 2024-09-11T06:42:31.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:1 2024-09-11T06:42:31.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:42:31.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:42:31.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-11T06:42:31.555 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:1 2024-09-11T06:42:31.555 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:42:31.555 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:42:31.555 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:42:31.555 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=1 2024-09-11T06:42:31.556 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-11T06:42:31.556 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:42:31.556 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:42:31.556 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=1 2024-09-11T06:42:31.556 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=1 2024-09-11T06:42:31.556 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-11T06:42:31.557 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:1 2024-09-11T06:42:31.557 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:42:31.557 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:42:31.557 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T06:42:31.557 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:1 2024-09-11T06:42:31.558 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:42:31.558 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:42:31.558 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:42:31.558 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=1 2024-09-11T06:42:31.558 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T06:42:31.558 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:42:31.559 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:42:31.559 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=1 2024-09-11T06:42:31.559 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=1 2024-09-11T06:42:31.559 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T06:42:31.574 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:42:31.575 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:42:31.575 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:42:31.575 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_36 2024-09-11T06:42:31.723 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:42:31.725 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror image_36 2024-09-11T06:42:31.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:42:31.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:42:31.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_36 2024-09-11T06:42:31.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror image_36 '' 2024-09-11T06:42:31.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:42:31.729 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:42:31.729 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_36 2024-09-11T06:42:31.729 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:42:31.729 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-image_36.status 2024-09-11T06:42:31.729 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-image_36.status 2024-09-11T06:42:31.729 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:42:31.729 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_36.status 2024-09-11T06:42:31.730 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_36 2024-09-11T06:42:31.730 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_36.status 2024-09-11T06:42:31.779 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 2 2024-09-11T06:42:31.779 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 2 2024-09-11T06:42:31.779 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:42:31.779 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=3, entry_tid=1], [object_number=8, tag_tid=3, entry_tid=0], [object_number=11, tag_tid=2, entry_tid=3], [object_number=10, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:42:31.779 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=9, tag_tid=3, entry_tid=1], [object_number=8, tag_tid=3, entry_tid=0], [object_number=11, tag_tid=2, entry_tid=3], [object_number=10, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:42:31.783 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_36.status 2024-09-11T06:42:31.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=9, tag_tid=3, entry_tid=1' 2024-09-11T06:42:31.785 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror image_36 2024-09-11T06:42:31.785 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:42:31.785 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:42:31.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_36 2024-09-11T06:42:31.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror image_36 '..*' 2024-09-11T06:42:31.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:42:31.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:42:31.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_36 2024-09-11T06:42:31.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:42:31.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-image_36.status 2024-09-11T06:42:31.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-image_36.status 2024-09-11T06:42:31.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:42:31.787 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_36.status 2024-09-11T06:42:31.787 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_36 2024-09-11T06:42:31.787 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_36.status 2024-09-11T06:42:31.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 2 2024-09-11T06:42:31.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 2 2024-09-11T06:42:31.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:42:31.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=3, entry_tid=1], [object_number=8, tag_tid=3, entry_tid=0], [object_number=11, tag_tid=2, entry_tid=3], [object_number=10, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:42:31.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=9, tag_tid=3, entry_tid=1], [object_number=8, tag_tid=3, entry_tid=0], [object_number=11, tag_tid=2, entry_tid=3], [object_number=10, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:42:31.836 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_36.status 2024-09-11T06:42:31.838 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=9, tag_tid=3, entry_tid=1' 2024-09-11T06:42:31.838 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=9, tag_tid=3, entry_tid=1' -a 'object_number=9, tag_tid=3, entry_tid=1' = 'object_number=9, tag_tid=3, entry_tid=1' 2024-09-11T06:42:31.838 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:42:31.838 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_snap_present cluster1 mirror image_36 snap 2024-09-11T06:42:31.838 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:42:31.838 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:31.839 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_36 2024-09-11T06:42:31.839 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:42:31.839 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:42:31.839 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T06:42:31.839 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:42:32.839 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 info mirror/image_36@snap 2024-09-11T06:42:32.902 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:rbd image 'image_36': 2024-09-11T06:42:32.902 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: size 128 MiB in 32 objects 2024-09-11T06:42:32.902 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: order 22 (4 MiB objects) 2024-09-11T06:42:32.902 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: snapshot_count: 1 2024-09-11T06:42:32.902 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: id: 10a961f12a70 2024-09-11T06:42:32.902 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: block_name_prefix: rbd_data.10a961f12a70 2024-09-11T06:42:32.902 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: format: 2 2024-09-11T06:42:32.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: features: layering, exclusive-lock, journaling 2024-09-11T06:42:32.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: op_features: 2024-09-11T06:42:32.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: flags: 2024-09-11T06:42:32.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: create_timestamp: Wed Sep 11 06:36:16 2024 2024-09-11T06:42:32.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: access_timestamp: Wed Sep 11 06:36:16 2024 2024-09-11T06:42:32.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: modify_timestamp: Wed Sep 11 06:38:50 2024 2024-09-11T06:42:32.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: protected: False 2024-09-11T06:42:32.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: journal: 10a961f12a70 2024-09-11T06:42:32.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring state: enabled 2024-09-11T06:42:32.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring mode: journal 2024-09-11T06:42:32.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring global id: 0b6d42a0-450a-430d-adf7-058a2508a4f6 2024-09-11T06:42:32.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring primary: false 2024-09-11T06:42:32.906 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:42:32.906 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ compare_image_snaps mirror image_36 snap 2024-09-11T06:42:32.907 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:32.907 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_36 2024-09-11T06:42:32.907 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:42:32.907 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local ret=0 2024-09-11T06:42:32.907 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local rmt_export=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_36.export 2024-09-11T06:42:32.907 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loc_export=/tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_36.export 2024-09-11T06:42:32.907 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_36.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_36.export 2024-09-11T06:42:32.907 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror export image_36@snap /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_36.export 2024-09-11T06:42:33.103 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T06:42:33.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 -p mirror export image_36@snap /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_36.export 2024-09-11T06:42:33.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T06:42:33.594 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmp /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_36.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_36.export 2024-09-11T06:42:33.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_36.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_36.export 2024-09-11T06:42:33.716 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:42:33.716 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:42:33.716 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_37 2024-09-11T06:42:33.716 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_snap cluster2 mirror image_37 snap 2024-09-11T06:42:33.716 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:42:33.716 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:33.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_37 2024-09-11T06:42:33.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:42:33.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror snap create image_37@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T06:42:34.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_started cluster1 mirror image_37 2024-09-11T06:42:34.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:42:34.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:34.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_37 2024-09-11T06:42:34.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_state cluster1 mirror image_37 started 2024-09-11T06:42:34.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:42:34.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:34.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_37 2024-09-11T06:42:34.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=started 2024-09-11T06:42:34.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:42:34.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:42:34.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:42:35.645 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_replay_state cluster1 mirror image_37 started 2024-09-11T06:42:35.645 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:42:35.645 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:35.645 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_37 2024-09-11T06:42:35.645 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=started 2024-09-11T06:42:35.645 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local status_result 2024-09-11T06:42:35.645 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=stopped 2024-09-11T06:42:35.646 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_37 2024-09-11T06:42:35.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_instance=cluster1 2024-09-11T06:42:35.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:42:35.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:42:35.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance=cluster1 2024-09-11T06:42:35.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local loop_instance 2024-09-11T06:42:35.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:42:35.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sleep 0 2024-09-11T06:42:35.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -i state 2024-09-11T06:42:35.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:42:35.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ seq 0 1 2024-09-11T06:42:35.650 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:42:35.650 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_37 2024-09-11T06:42:35.650 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1:0 2024-09-11T06:42:35.650 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:42:35.650 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:42:35.650 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:42:35.651 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:42:35.651 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=cluster 2024-09-11T06:42:35.651 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:42:35.651 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:42:35.651 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:42:35.651 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:42:35.651 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval cluster=cluster1 2024-09-11T06:42:35.651 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:42:35.651 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:42:35.651 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:42:35.651 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:42:35.651 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local local_cluster=cluster1:0 2024-09-11T06:42:35.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster=cluster1 2024-09-11T06:42:35.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance 2024-09-11T06:42:35.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:42:35.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local val=cluster1:0 2024-09-11T06:42:35.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T06:42:35.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance_var_name=instance 2024-09-11T06:42:35.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:42:35.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:42:35.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T06:42:35.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval local_cluster=cluster1 2024-09-11T06:42:35.654 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ local_cluster=cluster1 2024-09-11T06:42:35.654 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval instance=0 2024-09-11T06:42:35.654 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ instance=0 2024-09-11T06:42:35.654 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:42:35.677 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:42:35.677 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:42:35.677 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:42:35.677 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_37 2024-09-11T06:42:35.820 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ return 0 2024-09-11T06:42:35.820 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T06:42:35.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ echo ' "state": "Replaying"' 2024-09-11T06:42:35.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ grep -i Replaying 2024-09-11T06:42:35.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: "state": "Replaying" 2024-09-11T06:42:35.823 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_state=started 2024-09-11T06:42:35.823 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test started = started 2024-09-11T06:42:35.823 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:42:35.824 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_37 2024-09-11T06:42:35.824 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:42:35.824 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:42:35.824 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:35.824 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_37 2024-09-11T06:42:35.824 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' journal = journal ']' 2024-09-11T06:42:35.824 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_37 2024-09-11T06:42:35.824 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:42:35.824 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:42:35.824 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:35.824 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_37 2024-09-11T06:42:35.824 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-11T06:42:35.824 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-11T06:42:35.825 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:42:35.825 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:42:35.825 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:42:36.026 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror image_37 2024-09-11T06:42:36.026 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:42:36.026 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:36.026 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_37 2024-09-11T06:42:36.026 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:42:36.026 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n image_37 ']' 2024-09-11T06:42:36.026 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/image_37' 2024-09-11T06:42:36.026 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_37 2024-09-11T06:42:36.026 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:42:36.026 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:42:36.026 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:42:36.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:42:36.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:42:36.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:42:36.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:42:36.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:42:36.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:42:36.029 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:42:36.029 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_37 2024-09-11T06:42:36.029 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:42:36.029 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:42:36.029 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:42:36.029 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:42:36.029 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:42:36.030 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:42:36.030 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:42:36.030 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:42:36.030 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:42:36.030 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:42:36.030 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:42:36.030 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:42:36.030 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:42:36.030 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:42:36.030 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:42:36.030 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:42:36.030 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:42:36.030 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:42:36.031 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:42:36.031 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:42:36.031 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:42:36.031 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:42:36.031 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:42:36.031 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:42:36.031 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:42:36.031 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:42:36.031 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:42:36.031 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:42:36.031 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:42:36.031 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:42:36.053 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:42:36.054 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:42:36.054 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:42:36.054 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_37 2024-09-11T06:42:36.202 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:42:36.202 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror image_37 2024-09-11T06:42:36.202 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:42:36.202 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:42:36.203 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_37 2024-09-11T06:42:36.203 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror image_37 '' 2024-09-11T06:42:36.203 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:42:36.203 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:42:36.203 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_37 2024-09-11T06:42:36.203 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:42:36.203 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-image_37.status 2024-09-11T06:42:36.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-image_37.status 2024-09-11T06:42:36.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:42:36.205 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_37.status 2024-09-11T06:42:36.205 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_37 2024-09-11T06:42:36.205 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_37.status 2024-09-11T06:42:36.250 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 1 2024-09-11T06:42:36.250 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 1 2024-09-11T06:42:36.250 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:42:36.250 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:42:36.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:42:36.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_37.status 2024-09-11T06:42:36.255 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-09-11T06:42:36.256 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror image_37 2024-09-11T06:42:36.256 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:42:36.256 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:42:36.256 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_37 2024-09-11T06:42:36.256 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror image_37 '..*' 2024-09-11T06:42:36.256 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:42:36.256 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:42:36.256 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_37 2024-09-11T06:42:36.256 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:42:36.257 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-image_37.status 2024-09-11T06:42:36.257 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-image_37.status 2024-09-11T06:42:36.257 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:42:36.257 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_37.status 2024-09-11T06:42:36.257 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_37 2024-09-11T06:42:36.257 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_37.status 2024-09-11T06:42:36.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 1 2024-09-11T06:42:36.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 1 2024-09-11T06:42:36.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:42:36.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:42:36.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:42:36.306 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_37.status 2024-09-11T06:42:36.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-09-11T06:42:36.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=3, entry_tid=1' 2024-09-11T06:42:36.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:42:36.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_snap_present cluster1 mirror image_37 snap 2024-09-11T06:42:36.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:42:36.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:36.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_37 2024-09-11T06:42:36.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:42:36.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:42:36.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T06:42:36.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:42:37.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 info mirror/image_37@snap 2024-09-11T06:42:37.373 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:rbd image 'image_37': 2024-09-11T06:42:37.373 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: size 128 MiB in 32 objects 2024-09-11T06:42:37.373 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: order 22 (4 MiB objects) 2024-09-11T06:42:37.373 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: snapshot_count: 1 2024-09-11T06:42:37.373 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: id: 10a6d7bf0115 2024-09-11T06:42:37.373 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: block_name_prefix: rbd_data.10a6d7bf0115 2024-09-11T06:42:37.373 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: format: 2 2024-09-11T06:42:37.373 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: features: layering, exclusive-lock, journaling 2024-09-11T06:42:37.373 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: op_features: 2024-09-11T06:42:37.373 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: flags: 2024-09-11T06:42:37.374 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: create_timestamp: Wed Sep 11 06:36:29 2024 2024-09-11T06:42:37.374 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: access_timestamp: Wed Sep 11 06:36:29 2024 2024-09-11T06:42:37.374 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: modify_timestamp: Wed Sep 11 06:37:51 2024 2024-09-11T06:42:37.374 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: protected: False 2024-09-11T06:42:37.374 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: journal: 10a6d7bf0115 2024-09-11T06:42:37.374 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring state: enabled 2024-09-11T06:42:37.374 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring mode: journal 2024-09-11T06:42:37.374 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring global id: f19b6aff-dff6-4c89-997c-6743eb27808d 2024-09-11T06:42:37.374 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring primary: false 2024-09-11T06:42:37.378 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:42:37.378 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ compare_image_snaps mirror image_37 snap 2024-09-11T06:42:37.378 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:37.378 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_37 2024-09-11T06:42:37.378 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:42:37.378 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local ret=0 2024-09-11T06:42:37.378 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local rmt_export=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_37.export 2024-09-11T06:42:37.378 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loc_export=/tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_37.export 2024-09-11T06:42:37.378 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_37.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_37.export 2024-09-11T06:42:37.379 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror export image_37@snap /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_37.export 2024-09-11T06:42:37.540 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T06:42:37.547 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 -p mirror export image_37@snap /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_37.export 2024-09-11T06:42:37.813 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T06:42:37.822 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmp /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_37.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_37.export 2024-09-11T06:42:37.899 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_37.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_37.export 2024-09-11T06:42:37.942 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:42:37.942 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:42:37.942 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_38 2024-09-11T06:42:37.942 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_snap cluster2 mirror image_38 snap 2024-09-11T06:42:37.942 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:42:37.942 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:37.942 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_38 2024-09-11T06:42:37.942 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:42:37.942 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror snap create image_38@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T06:42:38.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_started cluster1 mirror image_38 2024-09-11T06:42:38.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:42:38.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:38.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_38 2024-09-11T06:42:38.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_state cluster1 mirror image_38 started 2024-09-11T06:42:38.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:42:38.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:38.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_38 2024-09-11T06:42:38.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=started 2024-09-11T06:42:38.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:42:38.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:42:38.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:42:39.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_replay_state cluster1 mirror image_38 started 2024-09-11T06:42:39.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:42:39.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:39.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_38 2024-09-11T06:42:39.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=started 2024-09-11T06:42:39.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local status_result 2024-09-11T06:42:39.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=stopped 2024-09-11T06:42:39.645 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_38 2024-09-11T06:42:39.646 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -i state 2024-09-11T06:42:39.646 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_instance=cluster1 2024-09-11T06:42:39.646 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:42:39.646 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:42:39.646 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance=cluster1 2024-09-11T06:42:39.646 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local loop_instance 2024-09-11T06:42:39.646 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:42:39.646 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sleep 0 2024-09-11T06:42:39.646 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:42:39.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ seq 0 1 2024-09-11T06:42:39.650 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:42:39.650 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_38 2024-09-11T06:42:39.650 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1:0 2024-09-11T06:42:39.650 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:42:39.650 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:42:39.650 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:42:39.650 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:42:39.650 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=cluster 2024-09-11T06:42:39.650 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:42:39.651 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:42:39.651 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:42:39.651 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:42:39.651 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval cluster=cluster1 2024-09-11T06:42:39.651 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:42:39.651 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:42:39.651 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:42:39.651 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:42:39.651 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local local_cluster=cluster1:0 2024-09-11T06:42:39.651 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster=cluster1 2024-09-11T06:42:39.651 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance 2024-09-11T06:42:39.651 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:42:39.651 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local val=cluster1:0 2024-09-11T06:42:39.651 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T06:42:39.652 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance_var_name=instance 2024-09-11T06:42:39.652 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:42:39.652 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:42:39.652 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T06:42:39.652 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval local_cluster=cluster1 2024-09-11T06:42:39.652 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ local_cluster=cluster1 2024-09-11T06:42:39.652 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval instance=0 2024-09-11T06:42:39.652 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ instance=0 2024-09-11T06:42:39.652 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:42:39.670 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:42:39.671 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:42:39.671 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:42:39.671 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_38 2024-09-11T06:42:39.800 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2024-09-11T06:42:39.801 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T06:42:39.801 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:admin_socket: invalid command 2024-09-11T06:42:39.804 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:42:39.804 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_38 2024-09-11T06:42:39.804 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1:1 2024-09-11T06:42:39.804 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:42:39.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:42:39.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-11T06:42:39.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:1 2024-09-11T06:42:39.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=cluster 2024-09-11T06:42:39.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:42:39.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:42:39.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=1 2024-09-11T06:42:39.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T06:42:39.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval cluster=cluster1 2024-09-11T06:42:39.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:42:39.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=1 2024-09-11T06:42:39.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=1 2024-09-11T06:42:39.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-11T06:42:39.806 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local local_cluster=cluster1:1 2024-09-11T06:42:39.806 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster=cluster1 2024-09-11T06:42:39.806 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance 2024-09-11T06:42:39.806 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T06:42:39.806 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local val=cluster1:1 2024-09-11T06:42:39.806 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T06:42:39.806 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance_var_name=instance 2024-09-11T06:42:39.806 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:42:39.806 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=1 2024-09-11T06:42:39.806 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-11T06:42:39.806 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval local_cluster=cluster1 2024-09-11T06:42:39.806 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ local_cluster=cluster1 2024-09-11T06:42:39.806 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval instance=1 2024-09-11T06:42:39.806 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ instance=1 2024-09-11T06:42:39.806 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T06:42:39.829 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:42:39.829 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:42:39.829 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:42:39.829 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_38 2024-09-11T06:42:39.986 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ return 0 2024-09-11T06:42:39.987 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T06:42:39.987 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ grep -i Replaying 2024-09-11T06:42:39.987 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ echo ' "state": "Replaying"' 2024-09-11T06:42:39.988 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: "state": "Replaying" 2024-09-11T06:42:39.989 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_state=started 2024-09-11T06:42:39.990 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test started = started 2024-09-11T06:42:39.990 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:42:39.990 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_38 2024-09-11T06:42:39.990 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:42:39.990 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:42:39.990 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:39.990 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_38 2024-09-11T06:42:39.990 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' journal = journal ']' 2024-09-11T06:42:39.990 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_38 2024-09-11T06:42:39.990 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:42:39.990 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:42:39.990 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:39.991 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_38 2024-09-11T06:42:39.991 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-11T06:42:39.991 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-11T06:42:39.991 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:42:39.991 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:42:39.991 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:42:40.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror image_38 2024-09-11T06:42:40.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:42:40.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:40.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_38 2024-09-11T06:42:40.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:42:40.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n image_38 ']' 2024-09-11T06:42:40.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/image_38' 2024-09-11T06:42:40.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_38 2024-09-11T06:42:40.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:42:40.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:42:40.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:42:40.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:42:40.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:42:40.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:42:40.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:42:40.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:42:40.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:42:40.196 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:42:40.196 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_38 2024-09-11T06:42:40.197 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:42:40.197 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:42:40.197 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:42:40.197 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:42:40.197 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:42:40.197 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:42:40.197 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:42:40.197 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:42:40.197 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:42:40.197 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:42:40.197 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:42:40.197 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:42:40.197 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:42:40.197 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:42:40.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:42:40.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:42:40.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:42:40.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:42:40.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:42:40.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:42:40.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:42:40.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:42:40.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:42:40.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:42:40.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:42:40.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:42:40.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:42:40.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:42:40.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:42:40.199 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:42:40.221 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:42:40.222 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:42:40.222 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:42:40.222 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_38 2024-09-11T06:42:40.351 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2024-09-11T06:42:40.351 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd mirror flush : flush rbd mirror 2024-09-11T06:42:40.351 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:admin_socket: invalid command 2024-09-11T06:42:40.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:42:40.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_38 2024-09-11T06:42:40.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:1 2024-09-11T06:42:40.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:42:40.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:42:40.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-11T06:42:40.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:1 2024-09-11T06:42:40.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:42:40.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:42:40.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:42:40.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=1 2024-09-11T06:42:40.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-11T06:42:40.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:42:40.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:42:40.356 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=1 2024-09-11T06:42:40.356 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=1 2024-09-11T06:42:40.356 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-11T06:42:40.356 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:1 2024-09-11T06:42:40.356 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:42:40.356 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:42:40.356 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T06:42:40.356 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:1 2024-09-11T06:42:40.356 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:42:40.356 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:42:40.356 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:42:40.356 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=1 2024-09-11T06:42:40.356 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T06:42:40.356 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:42:40.357 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:42:40.357 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=1 2024-09-11T06:42:40.357 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=1 2024-09-11T06:42:40.357 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T06:42:40.380 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:42:40.381 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:42:40.381 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:42:40.381 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_38 2024-09-11T06:42:40.516 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:42:40.516 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror image_38 2024-09-11T06:42:40.517 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:42:40.517 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:42:40.517 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_38 2024-09-11T06:42:40.517 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror image_38 '' 2024-09-11T06:42:40.517 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:42:40.517 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:42:40.517 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_38 2024-09-11T06:42:40.517 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:42:40.517 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-image_38.status 2024-09-11T06:42:40.518 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-image_38.status 2024-09-11T06:42:40.518 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:42:40.518 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_38.status 2024-09-11T06:42:40.519 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_38 2024-09-11T06:42:40.519 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_38.status 2024-09-11T06:42:40.564 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 1 2024-09-11T06:42:40.564 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 1 2024-09-11T06:42:40.564 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:42:40.564 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:42:40.564 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:42:40.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_38.status 2024-09-11T06:42:40.568 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-09-11T06:42:40.570 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror image_38 2024-09-11T06:42:40.570 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:42:40.570 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:42:40.570 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_38 2024-09-11T06:42:40.570 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror image_38 '..*' 2024-09-11T06:42:40.570 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:42:40.570 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:42:40.570 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_38 2024-09-11T06:42:40.570 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:42:40.570 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-image_38.status 2024-09-11T06:42:40.570 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-image_38.status 2024-09-11T06:42:40.570 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:42:40.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_38.status 2024-09-11T06:42:40.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_38 2024-09-11T06:42:40.571 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_38.status 2024-09-11T06:42:40.617 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 1 2024-09-11T06:42:40.617 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 1 2024-09-11T06:42:40.617 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:42:40.617 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:42:40.617 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:42:40.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_38.status 2024-09-11T06:42:40.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-09-11T06:42:40.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=3, entry_tid=1' 2024-09-11T06:42:40.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:42:40.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_snap_present cluster1 mirror image_38 snap 2024-09-11T06:42:40.623 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:42:40.623 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:40.623 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_38 2024-09-11T06:42:40.623 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:42:40.623 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:42:40.623 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T06:42:40.623 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:42:41.623 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 info mirror/image_38@snap 2024-09-11T06:42:41.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:rbd image 'image_38': 2024-09-11T06:42:41.693 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: size 128 MiB in 32 objects 2024-09-11T06:42:41.693 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: order 22 (4 MiB objects) 2024-09-11T06:42:41.693 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: snapshot_count: 1 2024-09-11T06:42:41.693 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: id: 10a9246eb98e 2024-09-11T06:42:41.693 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: block_name_prefix: rbd_data.10a9246eb98e 2024-09-11T06:42:41.693 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: format: 2 2024-09-11T06:42:41.693 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: features: layering, exclusive-lock, journaling 2024-09-11T06:42:41.693 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: op_features: 2024-09-11T06:42:41.693 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: flags: 2024-09-11T06:42:41.693 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: create_timestamp: Wed Sep 11 06:36:37 2024 2024-09-11T06:42:41.693 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: access_timestamp: Wed Sep 11 06:36:37 2024 2024-09-11T06:42:41.693 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: modify_timestamp: Wed Sep 11 06:37:53 2024 2024-09-11T06:42:41.694 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: protected: False 2024-09-11T06:42:41.694 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: journal: 10a9246eb98e 2024-09-11T06:42:41.694 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring state: enabled 2024-09-11T06:42:41.694 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring mode: journal 2024-09-11T06:42:41.694 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring global id: 98bdefea-e705-406b-a3e3-4f7574a9b426 2024-09-11T06:42:41.694 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring primary: false 2024-09-11T06:42:41.697 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:42:41.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ compare_image_snaps mirror image_38 snap 2024-09-11T06:42:41.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:41.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_38 2024-09-11T06:42:41.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:42:41.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local ret=0 2024-09-11T06:42:41.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local rmt_export=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_38.export 2024-09-11T06:42:41.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loc_export=/tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_38.export 2024-09-11T06:42:41.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_38.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_38.export 2024-09-11T06:42:41.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror export image_38@snap /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_38.export 2024-09-11T06:42:41.864 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T06:42:41.872 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 -p mirror export image_38@snap /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_38.export 2024-09-11T06:42:42.156 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T06:42:42.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmp /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_38.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_38.export 2024-09-11T06:42:42.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_38.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_38.export 2024-09-11T06:42:42.283 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:42:42.283 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:42:42.284 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_39 2024-09-11T06:42:42.284 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_snap cluster2 mirror image_39 snap 2024-09-11T06:42:42.284 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:42:42.284 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:42.284 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_39 2024-09-11T06:42:42.284 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:42:42.284 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror snap create image_39@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T06:42:42.656 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_started cluster1 mirror image_39 2024-09-11T06:42:42.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:42:42.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:42.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_39 2024-09-11T06:42:42.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_state cluster1 mirror image_39 started 2024-09-11T06:42:42.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:42:42.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:42.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_39 2024-09-11T06:42:42.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=started 2024-09-11T06:42:42.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:42:42.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:42:42.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:42:43.658 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_replay_state cluster1 mirror image_39 started 2024-09-11T06:42:43.658 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:42:43.658 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:43.658 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_39 2024-09-11T06:42:43.658 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=started 2024-09-11T06:42:43.658 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local status_result 2024-09-11T06:42:43.659 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=stopped 2024-09-11T06:42:43.660 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_39 2024-09-11T06:42:43.660 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_instance=cluster1 2024-09-11T06:42:43.660 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:42:43.660 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:42:43.660 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance=cluster1 2024-09-11T06:42:43.660 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local loop_instance 2024-09-11T06:42:43.660 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:42:43.660 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sleep 0 2024-09-11T06:42:43.660 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -i state 2024-09-11T06:42:43.661 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:42:43.661 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ seq 0 1 2024-09-11T06:42:43.664 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:42:43.665 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_39 2024-09-11T06:42:43.665 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1:0 2024-09-11T06:42:43.665 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:42:43.665 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:42:43.665 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:42:43.665 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:42:43.665 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=cluster 2024-09-11T06:42:43.665 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:42:43.665 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:42:43.665 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:42:43.665 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:42:43.665 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval cluster=cluster1 2024-09-11T06:42:43.665 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:42:43.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:42:43.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:42:43.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:42:43.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local local_cluster=cluster1:0 2024-09-11T06:42:43.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster=cluster1 2024-09-11T06:42:43.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance 2024-09-11T06:42:43.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:42:43.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local val=cluster1:0 2024-09-11T06:42:43.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T06:42:43.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance_var_name=instance 2024-09-11T06:42:43.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:42:43.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:42:43.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T06:42:43.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval local_cluster=cluster1 2024-09-11T06:42:43.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ local_cluster=cluster1 2024-09-11T06:42:43.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval instance=0 2024-09-11T06:42:43.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ instance=0 2024-09-11T06:42:43.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:42:43.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:42:43.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:42:43.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:42:43.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_39 2024-09-11T06:42:43.828 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ return 0 2024-09-11T06:42:43.829 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T06:42:43.829 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ echo ' "state": "Replaying"' 2024-09-11T06:42:43.829 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ grep -i Replaying 2024-09-11T06:42:43.831 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: "state": "Replaying" 2024-09-11T06:42:43.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_state=started 2024-09-11T06:42:43.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test started = started 2024-09-11T06:42:43.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:42:43.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_39 2024-09-11T06:42:43.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:42:43.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:42:43.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:43.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_39 2024-09-11T06:42:43.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' journal = journal ']' 2024-09-11T06:42:43.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_39 2024-09-11T06:42:43.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:42:43.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:42:43.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:43.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_39 2024-09-11T06:42:43.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-11T06:42:43.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-11T06:42:43.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:42:43.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:42:43.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:42:44.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror image_39 2024-09-11T06:42:44.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:42:44.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:44.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_39 2024-09-11T06:42:44.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:42:44.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n image_39 ']' 2024-09-11T06:42:44.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/image_39' 2024-09-11T06:42:44.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_39 2024-09-11T06:42:44.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:42:44.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:42:44.035 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:42:44.035 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:42:44.035 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:42:44.035 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:42:44.035 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:42:44.035 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:42:44.035 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:42:44.038 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:42:44.038 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_39 2024-09-11T06:42:44.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:42:44.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:42:44.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:42:44.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:42:44.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:42:44.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:42:44.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:42:44.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:42:44.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:42:44.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:42:44.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:42:44.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:42:44.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:42:44.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:42:44.040 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:42:44.040 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:42:44.040 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:42:44.040 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:42:44.040 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:42:44.040 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:42:44.040 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:42:44.040 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:42:44.040 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:42:44.040 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:42:44.040 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:42:44.040 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:42:44.040 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:42:44.041 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:42:44.041 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:42:44.041 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:42:44.064 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:42:44.064 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:42:44.064 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:42:44.064 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_39 2024-09-11T06:42:44.197 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:42:44.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror image_39 2024-09-11T06:42:44.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:42:44.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:42:44.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_39 2024-09-11T06:42:44.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror image_39 '' 2024-09-11T06:42:44.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:42:44.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:42:44.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_39 2024-09-11T06:42:44.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:42:44.199 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-image_39.status 2024-09-11T06:42:44.199 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-image_39.status 2024-09-11T06:42:44.199 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:42:44.201 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_39.status 2024-09-11T06:42:44.201 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_39 2024-09-11T06:42:44.201 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_39.status 2024-09-11T06:42:44.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 2 2024-09-11T06:42:44.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 2 2024-09-11T06:42:44.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:42:44.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=3, entry_tid=1], [object_number=8, tag_tid=3, entry_tid=0], [object_number=11, tag_tid=2, entry_tid=3], [object_number=10, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:42:44.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=11, tag_tid=2, entry_tid=3], [object_number=10, tag_tid=2, entry_tid=2], [object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0]]], state=connected] 2024-09-11T06:42:44.250 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_39.status 2024-09-11T06:42:44.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=9, tag_tid=3, entry_tid=1' 2024-09-11T06:42:44.252 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror image_39 2024-09-11T06:42:44.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:42:44.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:42:44.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_39 2024-09-11T06:42:44.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror image_39 '..*' 2024-09-11T06:42:44.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:42:44.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:42:44.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_39 2024-09-11T06:42:44.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:42:44.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-image_39.status 2024-09-11T06:42:44.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-image_39.status 2024-09-11T06:42:44.254 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:42:44.254 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_39.status 2024-09-11T06:42:44.255 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_39 2024-09-11T06:42:44.255 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_39.status 2024-09-11T06:42:44.298 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 2 2024-09-11T06:42:44.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 2 2024-09-11T06:42:44.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:42:44.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=3, entry_tid=1], [object_number=8, tag_tid=3, entry_tid=0], [object_number=11, tag_tid=2, entry_tid=3], [object_number=10, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:42:44.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=11, tag_tid=2, entry_tid=3], [object_number=10, tag_tid=2, entry_tid=2], [object_number=9, tag_tid=2, entry_tid=1], [object_number=8, tag_tid=2, entry_tid=0]]], state=connected] 2024-09-11T06:42:44.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_39.status 2024-09-11T06:42:44.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=11, tag_tid=2, entry_tid=3' 2024-09-11T06:42:44.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=9, tag_tid=3, entry_tid=1' -a 'object_number=9, tag_tid=3, entry_tid=1' = 'object_number=11, tag_tid=2, entry_tid=3' 2024-09-11T06:42:44.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=11, tag_tid=2, entry_tid=3' '!=' '' 2024-09-11T06:42:44.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:42:44.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 'object_number=11, tag_tid=2, entry_tid=3' = '' 2024-09-11T06:42:44.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ last_mirror_pos='object_number=11, tag_tid=2, entry_tid=3' 2024-09-11T06:42:44.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=9, tag_tid=3, entry_tid=1' 2024-09-11T06:42:44.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:42:44.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:42:44.305 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_tag=3 2024-09-11T06:42:44.306 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=11, tag_tid=2, entry_tid=3' 2024-09-11T06:42:44.306 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'tag_tid=[0-9]*' 2024-09-11T06:42:44.306 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:42:44.307 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_tag=2 2024-09-11T06:42:44.307 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=9, tag_tid=3, entry_tid=1' 2024-09-11T06:42:44.307 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:42:44.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:42:44.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_entry=1 2024-09-11T06:42:44.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo 'object_number=11, tag_tid=2, entry_tid=3' 2024-09-11T06:42:44.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -Eo 'entry_tid=[0-9]*' 2024-09-11T06:42:44.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut -d= -f 2 2024-09-11T06:42:44.311 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_entry=3 2024-09-11T06:42:44.311 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 3 = 2 -a 1 -le 3 2024-09-11T06:42:44.311 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:42:44.311 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:42:44.311 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:42:44.513 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror image_39 2024-09-11T06:42:44.513 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:42:44.513 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:44.513 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_39 2024-09-11T06:42:44.513 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:42:44.513 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n image_39 ']' 2024-09-11T06:42:44.513 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/image_39' 2024-09-11T06:42:44.513 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_39 2024-09-11T06:42:44.513 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:42:44.513 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:42:44.514 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:42:44.514 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:42:44.514 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:42:44.514 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:42:44.514 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:42:44.514 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:42:44.515 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:42:44.516 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:42:44.517 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_39 2024-09-11T06:42:44.517 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:42:44.517 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:42:44.517 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:42:44.517 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:42:44.517 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:42:44.517 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:42:44.517 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:42:44.517 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:42:44.517 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:42:44.517 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:42:44.517 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:42:44.517 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:42:44.518 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:42:44.518 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:42:44.519 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:42:44.519 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:42:44.519 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:42:44.519 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:42:44.519 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:42:44.519 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:42:44.519 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:42:44.519 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:42:44.519 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:42:44.519 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:42:44.520 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:42:44.520 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:42:44.520 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:42:44.520 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:42:44.520 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:42:44.520 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:42:44.544 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:42:44.544 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:42:44.544 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:42:44.544 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_39 2024-09-11T06:42:44.689 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:42:44.690 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror image_39 2024-09-11T06:42:44.690 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:42:44.690 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:42:44.690 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_39 2024-09-11T06:42:44.690 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror image_39 '' 2024-09-11T06:42:44.690 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:42:44.690 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:42:44.690 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_39 2024-09-11T06:42:44.690 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:42:44.690 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-image_39.status 2024-09-11T06:42:44.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-image_39.status 2024-09-11T06:42:44.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:42:44.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_39.status 2024-09-11T06:42:44.693 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_39.status 2024-09-11T06:42:44.693 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_39 2024-09-11T06:42:44.746 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 2 2024-09-11T06:42:44.746 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 2 2024-09-11T06:42:44.746 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:42:44.747 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=3, entry_tid=1], [object_number=8, tag_tid=3, entry_tid=0], [object_number=11, tag_tid=2, entry_tid=3], [object_number=10, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:42:44.747 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=9, tag_tid=3, entry_tid=1], [object_number=8, tag_tid=3, entry_tid=0], [object_number=11, tag_tid=2, entry_tid=3], [object_number=10, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:42:44.750 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_39.status 2024-09-11T06:42:44.753 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=9, tag_tid=3, entry_tid=1' 2024-09-11T06:42:44.753 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror image_39 2024-09-11T06:42:44.753 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:42:44.753 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:42:44.753 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_39 2024-09-11T06:42:44.753 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror image_39 '..*' 2024-09-11T06:42:44.753 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:42:44.754 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:42:44.754 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_39 2024-09-11T06:42:44.754 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:42:44.754 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-image_39.status 2024-09-11T06:42:44.754 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-image_39.status 2024-09-11T06:42:44.754 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:42:44.754 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_39.status 2024-09-11T06:42:44.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_39 2024-09-11T06:42:44.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_39.status 2024-09-11T06:42:44.800 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 2 2024-09-11T06:42:44.801 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 2 2024-09-11T06:42:44.801 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:42:44.801 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=3, entry_tid=1], [object_number=8, tag_tid=3, entry_tid=0], [object_number=11, tag_tid=2, entry_tid=3], [object_number=10, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:42:44.801 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=9, tag_tid=3, entry_tid=1], [object_number=8, tag_tid=3, entry_tid=0], [object_number=11, tag_tid=2, entry_tid=3], [object_number=10, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:42:44.804 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_39.status 2024-09-11T06:42:44.806 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=9, tag_tid=3, entry_tid=1' 2024-09-11T06:42:44.806 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=9, tag_tid=3, entry_tid=1' -a 'object_number=9, tag_tid=3, entry_tid=1' = 'object_number=9, tag_tid=3, entry_tid=1' 2024-09-11T06:42:44.806 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:42:44.806 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_snap_present cluster1 mirror image_39 snap 2024-09-11T06:42:44.806 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:42:44.806 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:44.806 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_39 2024-09-11T06:42:44.806 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:42:44.806 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:42:44.806 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T06:42:44.807 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:42:45.807 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 info mirror/image_39@snap 2024-09-11T06:42:45.878 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:rbd image 'image_39': 2024-09-11T06:42:45.878 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: size 128 MiB in 32 objects 2024-09-11T06:42:45.878 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: order 22 (4 MiB objects) 2024-09-11T06:42:45.878 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: snapshot_count: 1 2024-09-11T06:42:45.878 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: id: 10a631329e56 2024-09-11T06:42:45.878 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: block_name_prefix: rbd_data.10a631329e56 2024-09-11T06:42:45.878 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: format: 2 2024-09-11T06:42:45.878 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: features: layering, exclusive-lock, journaling 2024-09-11T06:42:45.878 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: op_features: 2024-09-11T06:42:45.878 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: flags: 2024-09-11T06:42:45.878 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: create_timestamp: Wed Sep 11 06:36:45 2024 2024-09-11T06:42:45.878 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: access_timestamp: Wed Sep 11 06:36:45 2024 2024-09-11T06:42:45.879 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: modify_timestamp: Wed Sep 11 06:39:51 2024 2024-09-11T06:42:45.879 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: protected: False 2024-09-11T06:42:45.879 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: journal: 10a631329e56 2024-09-11T06:42:45.879 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring state: enabled 2024-09-11T06:42:45.879 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring mode: journal 2024-09-11T06:42:45.879 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring global id: 2b78c4e1-398b-432f-adc9-b0850774ce2a 2024-09-11T06:42:45.879 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring primary: false 2024-09-11T06:42:45.882 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:42:45.882 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ compare_image_snaps mirror image_39 snap 2024-09-11T06:42:45.883 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:45.883 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_39 2024-09-11T06:42:45.883 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:42:45.883 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local ret=0 2024-09-11T06:42:45.883 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local rmt_export=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_39.export 2024-09-11T06:42:45.883 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loc_export=/tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_39.export 2024-09-11T06:42:45.883 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_39.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_39.export 2024-09-11T06:42:45.885 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror export image_39@snap /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_39.export 2024-09-11T06:42:46.098 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T06:42:46.105 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 -p mirror export image_39@snap /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_39.export 2024-09-11T06:42:46.511 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T06:42:46.526 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmp /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_39.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_39.export 2024-09-11T06:42:46.604 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_39.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_39.export 2024-09-11T06:42:46.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:42:46.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:42:46.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_40 2024-09-11T06:42:46.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_snap cluster2 mirror image_40 snap 2024-09-11T06:42:46.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:42:46.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:46.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_40 2024-09-11T06:42:46.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:42:46.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror snap create image_40@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T06:42:47.683 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_started cluster1 mirror image_40 2024-09-11T06:42:47.683 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:42:47.683 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:47.683 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_40 2024-09-11T06:42:47.683 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_state cluster1 mirror image_40 started 2024-09-11T06:42:47.684 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:42:47.684 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:47.684 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_40 2024-09-11T06:42:47.684 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=started 2024-09-11T06:42:47.684 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:42:47.684 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:42:47.684 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:42:48.685 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_replay_state cluster1 mirror image_40 started 2024-09-11T06:42:48.685 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:42:48.685 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:48.685 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_40 2024-09-11T06:42:48.685 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=started 2024-09-11T06:42:48.685 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local status_result 2024-09-11T06:42:48.686 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=stopped 2024-09-11T06:42:48.686 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_40 2024-09-11T06:42:48.686 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_instance=cluster1 2024-09-11T06:42:48.686 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:42:48.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:42:48.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance=cluster1 2024-09-11T06:42:48.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local loop_instance 2024-09-11T06:42:48.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:42:48.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sleep 0 2024-09-11T06:42:48.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -i state 2024-09-11T06:42:48.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:42:48.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ seq 0 1 2024-09-11T06:42:48.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:42:48.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_40 2024-09-11T06:42:48.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1:0 2024-09-11T06:42:48.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:42:48.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:42:48.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:42:48.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:42:48.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=cluster 2024-09-11T06:42:48.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:42:48.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:42:48.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:42:48.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:42:48.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval cluster=cluster1 2024-09-11T06:42:48.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:42:48.693 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:42:48.693 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:42:48.693 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:42:48.693 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local local_cluster=cluster1:0 2024-09-11T06:42:48.693 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster=cluster1 2024-09-11T06:42:48.693 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance 2024-09-11T06:42:48.693 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:42:48.693 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local val=cluster1:0 2024-09-11T06:42:48.693 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T06:42:48.693 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance_var_name=instance 2024-09-11T06:42:48.693 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:42:48.693 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:42:48.694 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T06:42:48.694 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval local_cluster=cluster1 2024-09-11T06:42:48.694 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ local_cluster=cluster1 2024-09-11T06:42:48.694 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval instance=0 2024-09-11T06:42:48.694 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ instance=0 2024-09-11T06:42:48.694 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:42:48.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:42:48.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:42:48.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:42:48.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_40 2024-09-11T06:42:48.842 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:Invalid command: unused arguments: ['mirror/image_40'] 2024-09-11T06:42:48.842 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T06:42:48.842 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:admin_socket: invalid command 2024-09-11T06:42:48.844 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:42:48.844 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_40 2024-09-11T06:42:48.844 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1:1 2024-09-11T06:42:48.844 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:42:48.844 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:42:48.844 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-11T06:42:48.844 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:1 2024-09-11T06:42:48.844 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=cluster 2024-09-11T06:42:48.845 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:42:48.845 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:42:48.845 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=1 2024-09-11T06:42:48.845 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T06:42:48.845 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval cluster=cluster1 2024-09-11T06:42:48.845 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:42:48.845 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=1 2024-09-11T06:42:48.845 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=1 2024-09-11T06:42:48.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-11T06:42:48.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local local_cluster=cluster1:1 2024-09-11T06:42:48.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster=cluster1 2024-09-11T06:42:48.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance 2024-09-11T06:42:48.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T06:42:48.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local val=cluster1:1 2024-09-11T06:42:48.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T06:42:48.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance_var_name=instance 2024-09-11T06:42:48.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:42:48.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=1 2024-09-11T06:42:48.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-11T06:42:48.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval local_cluster=cluster1 2024-09-11T06:42:48.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ local_cluster=cluster1 2024-09-11T06:42:48.848 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval instance=1 2024-09-11T06:42:48.848 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ instance=1 2024-09-11T06:42:48.848 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T06:42:48.869 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:42:48.869 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:42:48.869 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:42:48.869 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_40 2024-09-11T06:42:49.009 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ return 0 2024-09-11T06:42:49.010 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T06:42:49.010 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ grep -i Replaying 2024-09-11T06:42:49.010 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ echo ' "state": "Replaying"' 2024-09-11T06:42:49.012 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: "state": "Replaying" 2024-09-11T06:42:49.013 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_state=started 2024-09-11T06:42:49.013 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test started = started 2024-09-11T06:42:49.013 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:42:49.013 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_40 2024-09-11T06:42:49.013 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:42:49.013 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:42:49.013 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:49.013 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_40 2024-09-11T06:42:49.013 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' journal = journal ']' 2024-09-11T06:42:49.013 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_40 2024-09-11T06:42:49.013 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:42:49.013 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:42:49.014 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:49.014 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_40 2024-09-11T06:42:49.014 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-11T06:42:49.014 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-11T06:42:49.014 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:42:49.014 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:42:49.014 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:42:49.214 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror image_40 2024-09-11T06:42:49.214 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:42:49.214 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:49.214 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_40 2024-09-11T06:42:49.214 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:42:49.215 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n image_40 ']' 2024-09-11T06:42:49.215 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/image_40' 2024-09-11T06:42:49.215 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_40 2024-09-11T06:42:49.215 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:42:49.215 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:42:49.215 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:42:49.215 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:42:49.215 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:42:49.215 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:42:49.215 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:42:49.216 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:42:49.216 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:42:49.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:42:49.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_40 2024-09-11T06:42:49.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:42:49.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:42:49.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:42:49.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:42:49.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:42:49.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:42:49.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:42:49.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:42:49.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:42:49.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:42:49.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:42:49.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:42:49.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:42:49.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:42:49.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:42:49.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:42:49.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:42:49.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:42:49.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:42:49.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:42:49.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:42:49.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:42:49.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:42:49.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:42:49.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:42:49.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:42:49.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:42:49.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:42:49.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:42:49.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:42:49.244 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:42:49.244 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:42:49.244 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:42:49.244 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_40 2024-09-11T06:42:49.368 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:Invalid command: unused arguments: ['mirror/image_40'] 2024-09-11T06:42:49.369 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd mirror flush : flush rbd mirror 2024-09-11T06:42:49.369 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:admin_socket: invalid command 2024-09-11T06:42:49.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:42:49.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_40 2024-09-11T06:42:49.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:1 2024-09-11T06:42:49.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:42:49.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:42:49.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-11T06:42:49.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:1 2024-09-11T06:42:49.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:42:49.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:42:49.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:42:49.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=1 2024-09-11T06:42:49.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-11T06:42:49.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:42:49.372 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:42:49.372 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=1 2024-09-11T06:42:49.372 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=1 2024-09-11T06:42:49.373 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-11T06:42:49.374 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:1 2024-09-11T06:42:49.374 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:42:49.374 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:42:49.374 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T06:42:49.374 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:1 2024-09-11T06:42:49.374 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:42:49.374 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:42:49.374 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:42:49.374 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=1 2024-09-11T06:42:49.375 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T06:42:49.375 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:42:49.375 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:42:49.375 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=1 2024-09-11T06:42:49.375 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=1 2024-09-11T06:42:49.375 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T06:42:49.397 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:42:49.398 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:42:49.398 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:42:49.398 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_40 2024-09-11T06:42:49.537 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:42:49.538 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror image_40 2024-09-11T06:42:49.538 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:42:49.538 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:42:49.538 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_40 2024-09-11T06:42:49.538 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror image_40 '' 2024-09-11T06:42:49.538 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:42:49.538 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:42:49.538 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_40 2024-09-11T06:42:49.539 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:42:49.539 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-image_40.status 2024-09-11T06:42:49.540 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-image_40.status 2024-09-11T06:42:49.540 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:42:49.541 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_40.status 2024-09-11T06:42:49.541 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_40 2024-09-11T06:42:49.541 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_40.status 2024-09-11T06:42:49.592 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 1 2024-09-11T06:42:49.592 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 1 2024-09-11T06:42:49.592 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:42:49.593 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:42:49.593 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:42:49.596 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_40.status 2024-09-11T06:42:49.597 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-09-11T06:42:49.598 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror image_40 2024-09-11T06:42:49.598 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:42:49.599 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:42:49.599 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_40 2024-09-11T06:42:49.599 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror image_40 '..*' 2024-09-11T06:42:49.599 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:42:49.599 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:42:49.599 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_40 2024-09-11T06:42:49.599 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:42:49.599 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-image_40.status 2024-09-11T06:42:49.599 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-image_40.status 2024-09-11T06:42:49.599 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:42:49.600 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_40.status 2024-09-11T06:42:49.600 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_40 2024-09-11T06:42:49.600 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_40.status 2024-09-11T06:42:49.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 1 2024-09-11T06:42:49.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 1 2024-09-11T06:42:49.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:42:49.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:42:49.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:42:49.651 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_40.status 2024-09-11T06:42:49.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-09-11T06:42:49.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=3, entry_tid=1' 2024-09-11T06:42:49.654 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:42:49.654 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_snap_present cluster1 mirror image_40 snap 2024-09-11T06:42:49.654 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:42:49.654 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:49.654 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_40 2024-09-11T06:42:49.654 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:42:49.654 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:42:49.654 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T06:42:49.654 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:42:50.654 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 info mirror/image_40@snap 2024-09-11T06:42:50.723 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:rbd image 'image_40': 2024-09-11T06:42:50.723 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: size 128 MiB in 32 objects 2024-09-11T06:42:50.723 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: order 22 (4 MiB objects) 2024-09-11T06:42:50.724 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: snapshot_count: 1 2024-09-11T06:42:50.724 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: id: 10a99df17fb7 2024-09-11T06:42:50.724 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: block_name_prefix: rbd_data.10a99df17fb7 2024-09-11T06:42:50.724 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: format: 2 2024-09-11T06:42:50.724 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: features: layering, exclusive-lock, journaling 2024-09-11T06:42:50.724 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: op_features: 2024-09-11T06:42:50.724 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: flags: 2024-09-11T06:42:50.724 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: create_timestamp: Wed Sep 11 06:37:00 2024 2024-09-11T06:42:50.725 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: access_timestamp: Wed Sep 11 06:37:00 2024 2024-09-11T06:42:50.725 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: modify_timestamp: Wed Sep 11 06:38:57 2024 2024-09-11T06:42:50.725 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: protected: False 2024-09-11T06:42:50.725 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: journal: 10a99df17fb7 2024-09-11T06:42:50.725 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring state: enabled 2024-09-11T06:42:50.725 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring mode: journal 2024-09-11T06:42:50.725 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring global id: 08a130b3-fcaa-4da0-8f9a-9640f705ea66 2024-09-11T06:42:50.726 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring primary: false 2024-09-11T06:42:50.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:42:50.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ compare_image_snaps mirror image_40 snap 2024-09-11T06:42:50.729 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:50.729 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_40 2024-09-11T06:42:50.729 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:42:50.729 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local ret=0 2024-09-11T06:42:50.729 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local rmt_export=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_40.export 2024-09-11T06:42:50.730 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loc_export=/tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_40.export 2024-09-11T06:42:50.730 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_40.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_40.export 2024-09-11T06:42:50.730 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror export image_40@snap /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_40.export 2024-09-11T06:42:50.872 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:42:50.866+0000 7fb45318f640 -1 reset not still connected to 0x561315ba4270 2024-09-11T06:42:50.872 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:42:50.866+0000 7fb45318f640 -1 reset not still connected to 0x561315ba5520 2024-09-11T06:42:50.885 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:42:50.870+0000 7fb45318f640 -1 reset not still connected to 0x56131a1ffe10 2024-09-11T06:42:50.885 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:42:50.870+0000 7fb45318f640 -1 reset not still connected to 0x561322931860 2024-09-11T06:42:50.891 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:42:50.886+0000 7fb45318f640 -1 reset not still connected to 0x561322ded930 2024-09-11T06:42:50.891 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:42:50.886+0000 7fb45318f640 -1 reset not still connected to 0x561330ed0410 2024-09-11T06:42:50.892 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:42:50.886+0000 7fb45318f640 -1 reset not still connected to 0x56133339c410 2024-09-11T06:42:50.892 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:42:50.886+0000 7fb45318f640 -1 reset not still connected to 0x56133c1909c0 2024-09-11T06:42:50.900 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:42:50.894+0000 7fb45318f640 -1 reset not still connected to 0x56133c2a2dd0 2024-09-11T06:42:50.901 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:42:50.894+0000 7fb45318f640 -1 reset not still connected to 0x561341f69450 2024-09-11T06:42:50.901 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:42:50.894+0000 7fb45318f640 -1 reset not still connected to 0x561342043930 2024-09-11T06:42:50.901 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:42:50.894+0000 7fb45318f640 -1 reset not still connected to 0x5613480872b0 2024-09-11T06:42:50.901 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:42:50.894+0000 7fb45318f640 -1 reset not still connected to 0x5613480e41a0 2024-09-11T06:42:50.901 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:42:50.894+0000 7fb45318f640 -1 reset not still connected to 0x5613488b0820 2024-09-11T06:42:50.901 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:42:50.894+0000 7fb45318f640 -1 reset not still connected to 0x5613488b0d00 2024-09-11T06:42:50.901 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:42:50.894+0000 7fb45318f640 -1 reset not still connected to 0x56134d76fe10 2024-09-11T06:42:50.901 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:42:50.894+0000 7fb45318f640 -1 reset not still connected to 0x56134d81fd40 2024-09-11T06:42:50.901 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:42:50.894+0000 7fb45318f640 -1 reset not still connected to 0x561354d57040 2024-09-11T06:42:50.901 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:42:50.894+0000 7fb45318f640 -1 reset not still connected to 0x561354d77110 2024-09-11T06:42:50.901 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:42:50.894+0000 7fb45318f640 -1 reset not still connected to 0x561355040c30 2024-09-11T06:42:50.901 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:42:50.894+0000 7fb45318f640 -1 reset not still connected to 0x5613551fd110 2024-09-11T06:42:50.901 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:42:50.894+0000 7fb45318f640 -1 reset not still connected to 0x5613557c4680 2024-09-11T06:42:50.924 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T06:42:50.932 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 -p mirror export image_40@snap /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_40.export 2024-09-11T06:42:51.289 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T06:42:51.301 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmp /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_40.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_40.export 2024-09-11T06:42:51.384 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_40.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_40.export 2024-09-11T06:42:51.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:42:51.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:42:51.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_41 2024-09-11T06:42:51.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_snap cluster2 mirror image_41 snap 2024-09-11T06:42:51.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:42:51.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:51.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_41 2024-09-11T06:42:51.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:42:51.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror snap create image_41@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T06:42:51.681 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_started cluster1 mirror image_41 2024-09-11T06:42:51.681 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:42:51.681 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:51.681 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_41 2024-09-11T06:42:51.681 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_state cluster1 mirror image_41 started 2024-09-11T06:42:51.681 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:42:51.681 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:51.681 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_41 2024-09-11T06:42:51.682 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=started 2024-09-11T06:42:51.682 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:42:51.682 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:42:51.682 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:42:52.682 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_replay_state cluster1 mirror image_41 started 2024-09-11T06:42:52.682 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:42:52.683 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:52.683 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_41 2024-09-11T06:42:52.683 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=started 2024-09-11T06:42:52.683 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local status_result 2024-09-11T06:42:52.683 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=stopped 2024-09-11T06:42:52.684 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_41 2024-09-11T06:42:52.684 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -i state 2024-09-11T06:42:52.684 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_instance=cluster1 2024-09-11T06:42:52.684 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:42:52.684 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:42:52.684 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance=cluster1 2024-09-11T06:42:52.684 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local loop_instance 2024-09-11T06:42:52.684 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:42:52.684 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sleep 0 2024-09-11T06:42:52.685 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:42:52.685 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ seq 0 1 2024-09-11T06:42:52.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:42:52.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_41 2024-09-11T06:42:52.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1:0 2024-09-11T06:42:52.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:42:52.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:42:52.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:42:52.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:42:52.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=cluster 2024-09-11T06:42:52.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:42:52.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:42:52.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:42:52.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:42:52.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval cluster=cluster1 2024-09-11T06:42:52.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:42:52.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:42:52.689 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:42:52.689 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:42:52.689 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local local_cluster=cluster1:0 2024-09-11T06:42:52.690 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster=cluster1 2024-09-11T06:42:52.690 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance 2024-09-11T06:42:52.690 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:42:52.690 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local val=cluster1:0 2024-09-11T06:42:52.690 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T06:42:52.690 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance_var_name=instance 2024-09-11T06:42:52.690 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:42:52.690 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:42:52.690 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T06:42:52.690 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval local_cluster=cluster1 2024-09-11T06:42:52.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ local_cluster=cluster1 2024-09-11T06:42:52.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval instance=0 2024-09-11T06:42:52.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ instance=0 2024-09-11T06:42:52.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:42:52.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:42:52.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:42:52.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:42:52.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_41 2024-09-11T06:42:52.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ return 0 2024-09-11T06:42:52.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T06:42:52.848 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ grep -i Replaying 2024-09-11T06:42:52.848 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ echo ' "state": "Replaying"' 2024-09-11T06:42:52.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: "state": "Replaying" 2024-09-11T06:42:52.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_state=started 2024-09-11T06:42:52.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test started = started 2024-09-11T06:42:52.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:42:52.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_41 2024-09-11T06:42:52.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:42:52.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:42:52.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:52.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_41 2024-09-11T06:42:52.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' journal = journal ']' 2024-09-11T06:42:52.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_41 2024-09-11T06:42:52.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:42:52.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:42:52.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:52.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_41 2024-09-11T06:42:52.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-11T06:42:52.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-11T06:42:52.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:42:52.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:42:52.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:42:53.052 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror image_41 2024-09-11T06:42:53.052 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:42:53.052 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:53.052 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_41 2024-09-11T06:42:53.053 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:42:53.053 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n image_41 ']' 2024-09-11T06:42:53.053 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/image_41' 2024-09-11T06:42:53.053 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_41 2024-09-11T06:42:53.053 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:42:53.053 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:42:53.053 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:42:53.053 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:42:53.053 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:42:53.053 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:42:53.053 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:42:53.054 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:42:53.054 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:42:53.057 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:42:53.057 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_41 2024-09-11T06:42:53.057 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:42:53.057 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:42:53.057 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:42:53.057 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:42:53.057 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:42:53.057 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:42:53.057 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:42:53.057 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:42:53.058 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:42:53.058 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:42:53.058 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:42:53.058 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:42:53.058 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:42:53.058 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:42:53.058 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:42:53.058 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:42:53.058 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:42:53.058 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:42:53.058 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:42:53.058 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:42:53.058 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:42:53.058 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:42:53.059 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:42:53.059 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:42:53.059 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:42:53.059 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:42:53.059 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:42:53.059 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:42:53.059 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:42:53.059 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:42:53.083 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:42:53.084 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:42:53.084 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:42:53.084 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_41 2024-09-11T06:42:53.224 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:42:53.225 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror image_41 2024-09-11T06:42:53.225 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:42:53.226 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:42:53.226 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_41 2024-09-11T06:42:53.226 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror image_41 '' 2024-09-11T06:42:53.226 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:42:53.226 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:42:53.226 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_41 2024-09-11T06:42:53.226 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:42:53.227 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-image_41.status 2024-09-11T06:42:53.227 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-image_41.status 2024-09-11T06:42:53.227 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:42:53.229 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_41.status 2024-09-11T06:42:53.230 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_41 2024-09-11T06:42:53.230 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_41.status 2024-09-11T06:42:53.276 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 1 2024-09-11T06:42:53.276 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 1 2024-09-11T06:42:53.276 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:42:53.276 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:42:53.276 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:42:53.279 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_41.status 2024-09-11T06:42:53.280 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-09-11T06:42:53.283 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror image_41 2024-09-11T06:42:53.283 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:42:53.283 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:42:53.283 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_41 2024-09-11T06:42:53.284 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror image_41 '..*' 2024-09-11T06:42:53.284 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:42:53.284 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:42:53.284 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_41 2024-09-11T06:42:53.284 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:42:53.284 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-image_41.status 2024-09-11T06:42:53.285 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-image_41.status 2024-09-11T06:42:53.285 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:42:53.285 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_41.status 2024-09-11T06:42:53.285 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_41 2024-09-11T06:42:53.285 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_41.status 2024-09-11T06:42:53.333 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 1 2024-09-11T06:42:53.334 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 1 2024-09-11T06:42:53.334 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:42:53.334 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:42:53.334 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:42:53.337 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_41.status 2024-09-11T06:42:53.339 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-09-11T06:42:53.339 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=3, entry_tid=1' 2024-09-11T06:42:53.339 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:42:53.339 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_snap_present cluster1 mirror image_41 snap 2024-09-11T06:42:53.340 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:42:53.340 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:53.340 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_41 2024-09-11T06:42:53.340 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:42:53.340 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:42:53.340 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T06:42:53.340 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:42:54.340 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 info mirror/image_41@snap 2024-09-11T06:42:54.407 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:rbd image 'image_41': 2024-09-11T06:42:54.407 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: size 128 MiB in 32 objects 2024-09-11T06:42:54.407 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: order 22 (4 MiB objects) 2024-09-11T06:42:54.408 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: snapshot_count: 1 2024-09-11T06:42:54.408 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: id: 10a67e762c02 2024-09-11T06:42:54.408 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: block_name_prefix: rbd_data.10a67e762c02 2024-09-11T06:42:54.408 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: format: 2 2024-09-11T06:42:54.408 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: features: layering, exclusive-lock, journaling 2024-09-11T06:42:54.408 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: op_features: 2024-09-11T06:42:54.408 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: flags: 2024-09-11T06:42:54.409 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: create_timestamp: Wed Sep 11 06:37:07 2024 2024-09-11T06:42:54.409 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: access_timestamp: Wed Sep 11 06:37:07 2024 2024-09-11T06:42:54.409 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: modify_timestamp: Wed Sep 11 06:38:33 2024 2024-09-11T06:42:54.409 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: protected: False 2024-09-11T06:42:54.409 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: journal: 10a67e762c02 2024-09-11T06:42:54.409 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring state: enabled 2024-09-11T06:42:54.409 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring mode: journal 2024-09-11T06:42:54.409 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring global id: 17b7d629-e0c7-4994-8e1f-2e22ae134a9c 2024-09-11T06:42:54.409 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring primary: false 2024-09-11T06:42:54.412 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:42:54.412 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ compare_image_snaps mirror image_41 snap 2024-09-11T06:42:54.412 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:54.412 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_41 2024-09-11T06:42:54.412 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:42:54.412 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local ret=0 2024-09-11T06:42:54.412 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local rmt_export=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_41.export 2024-09-11T06:42:54.413 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loc_export=/tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_41.export 2024-09-11T06:42:54.413 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_41.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_41.export 2024-09-11T06:42:54.413 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror export image_41@snap /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_41.export 2024-09-11T06:42:54.586 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T06:42:54.594 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 -p mirror export image_41@snap /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_41.export 2024-09-11T06:42:54.897 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:42:54.897+0000 7efd718c5640 -1 reset not still connected to 0x560b15963110 2024-09-11T06:42:54.897 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:42:54.897+0000 7efd718c5640 -1 reset not still connected to 0x560b15a70d00 2024-09-11T06:42:54.897 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:42:54.897+0000 7efd718c5640 -1 reset not still connected to 0x560b15a71ad0 2024-09-11T06:42:54.912 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:42:54.913+0000 7efd718c5640 -1 reset not still connected to 0x560b159635f0 2024-09-11T06:42:54.912 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:42:54.913+0000 7efd718c5640 -1 reset not still connected to 0x560b15a70dd0 2024-09-11T06:42:54.912 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:42:54.913+0000 7efd718c5640 -1 reset not still connected to 0x560b15a71520 2024-09-11T06:42:54.962 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T06:42:54.969 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmp /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_41.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_41.export 2024-09-11T06:42:55.041 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_41.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_41.export 2024-09-11T06:42:55.082 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:42:55.082 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:42:55.082 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_42 2024-09-11T06:42:55.083 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_snap cluster2 mirror image_42 snap 2024-09-11T06:42:55.083 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:42:55.083 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:55.083 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_42 2024-09-11T06:42:55.083 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:42:55.083 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror snap create image_42@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T06:42:55.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_started cluster1 mirror image_42 2024-09-11T06:42:55.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:42:55.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:55.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_42 2024-09-11T06:42:55.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_state cluster1 mirror image_42 started 2024-09-11T06:42:55.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:42:55.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:55.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_42 2024-09-11T06:42:55.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=started 2024-09-11T06:42:55.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:42:55.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:42:55.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:42:56.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_replay_state cluster1 mirror image_42 started 2024-09-11T06:42:56.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:42:56.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:56.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_42 2024-09-11T06:42:56.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=started 2024-09-11T06:42:56.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local status_result 2024-09-11T06:42:56.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=stopped 2024-09-11T06:42:56.706 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_42 2024-09-11T06:42:56.706 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -i state 2024-09-11T06:42:56.706 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_instance=cluster1 2024-09-11T06:42:56.706 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:42:56.706 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:42:56.706 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance=cluster1 2024-09-11T06:42:56.706 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local loop_instance 2024-09-11T06:42:56.707 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:42:56.707 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sleep 0 2024-09-11T06:42:56.707 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:42:56.708 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ seq 0 1 2024-09-11T06:42:56.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:42:56.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_42 2024-09-11T06:42:56.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1:0 2024-09-11T06:42:56.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:42:56.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:42:56.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:42:56.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:42:56.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=cluster 2024-09-11T06:42:56.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:42:56.711 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:42:56.711 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:42:56.711 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:42:56.711 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval cluster=cluster1 2024-09-11T06:42:56.711 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:42:56.711 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:42:56.711 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:42:56.711 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:42:56.711 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local local_cluster=cluster1:0 2024-09-11T06:42:56.711 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster=cluster1 2024-09-11T06:42:56.711 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance 2024-09-11T06:42:56.711 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:42:56.712 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local val=cluster1:0 2024-09-11T06:42:56.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T06:42:56.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance_var_name=instance 2024-09-11T06:42:56.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:42:56.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:42:56.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T06:42:56.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval local_cluster=cluster1 2024-09-11T06:42:56.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ local_cluster=cluster1 2024-09-11T06:42:56.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval instance=0 2024-09-11T06:42:56.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ instance=0 2024-09-11T06:42:56.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:42:56.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:42:56.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:42:56.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:42:56.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_42 2024-09-11T06:42:56.862 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:Invalid command: unused arguments: ['mirror/image_42'] 2024-09-11T06:42:56.863 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T06:42:56.863 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:admin_socket: invalid command 2024-09-11T06:42:56.865 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:42:56.865 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_42 2024-09-11T06:42:56.865 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1:1 2024-09-11T06:42:56.865 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:42:56.865 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:42:56.865 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-11T06:42:56.865 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:1 2024-09-11T06:42:56.865 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=cluster 2024-09-11T06:42:56.865 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:42:56.865 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:42:56.865 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=1 2024-09-11T06:42:56.865 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T06:42:56.865 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval cluster=cluster1 2024-09-11T06:42:56.866 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:42:56.866 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=1 2024-09-11T06:42:56.866 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=1 2024-09-11T06:42:56.867 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-11T06:42:56.867 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local local_cluster=cluster1:1 2024-09-11T06:42:56.867 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster=cluster1 2024-09-11T06:42:56.867 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance 2024-09-11T06:42:56.867 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T06:42:56.867 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local val=cluster1:1 2024-09-11T06:42:56.867 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T06:42:56.868 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance_var_name=instance 2024-09-11T06:42:56.868 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:42:56.868 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=1 2024-09-11T06:42:56.868 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-11T06:42:56.868 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval local_cluster=cluster1 2024-09-11T06:42:56.868 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ local_cluster=cluster1 2024-09-11T06:42:56.868 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval instance=1 2024-09-11T06:42:56.868 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ instance=1 2024-09-11T06:42:56.868 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T06:42:56.890 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:42:56.891 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:42:56.891 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:42:56.891 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_42 2024-09-11T06:42:57.032 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ return 0 2024-09-11T06:42:57.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T06:42:57.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ echo ' "state": "Replaying"' 2024-09-11T06:42:57.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ grep -i Replaying 2024-09-11T06:42:57.035 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: "state": "Replaying" 2024-09-11T06:42:57.037 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_state=started 2024-09-11T06:42:57.037 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test started = started 2024-09-11T06:42:57.038 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:42:57.038 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_42 2024-09-11T06:42:57.038 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:42:57.038 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:42:57.038 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:57.038 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_42 2024-09-11T06:42:57.038 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' journal = journal ']' 2024-09-11T06:42:57.038 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_42 2024-09-11T06:42:57.038 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:42:57.038 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:42:57.038 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:57.038 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_42 2024-09-11T06:42:57.038 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-11T06:42:57.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-11T06:42:57.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:42:57.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:42:57.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:42:57.239 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror image_42 2024-09-11T06:42:57.239 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:42:57.239 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:57.239 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_42 2024-09-11T06:42:57.239 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:42:57.239 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n image_42 ']' 2024-09-11T06:42:57.239 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/image_42' 2024-09-11T06:42:57.239 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_42 2024-09-11T06:42:57.239 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:42:57.239 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:42:57.239 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:42:57.240 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:42:57.240 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:42:57.240 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:42:57.240 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:42:57.240 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:42:57.240 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:42:57.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:42:57.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_42 2024-09-11T06:42:57.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:42:57.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:42:57.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:42:57.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:42:57.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:42:57.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:42:57.242 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:42:57.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:42:57.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:42:57.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:42:57.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:42:57.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:42:57.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:42:57.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:42:57.244 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:42:57.244 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:42:57.244 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:42:57.244 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:42:57.245 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:42:57.245 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:42:57.245 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:42:57.245 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:42:57.245 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:42:57.245 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:42:57.245 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:42:57.245 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:42:57.245 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:42:57.245 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:42:57.245 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:42:57.245 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:42:57.269 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:42:57.269 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:42:57.269 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:42:57.269 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_42 2024-09-11T06:42:57.394 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:Invalid command: unused arguments: ['mirror/image_42'] 2024-09-11T06:42:57.394 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd mirror flush : flush rbd mirror 2024-09-11T06:42:57.394 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:admin_socket: invalid command 2024-09-11T06:42:57.396 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:42:57.396 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_42 2024-09-11T06:42:57.397 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:1 2024-09-11T06:42:57.397 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:42:57.397 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:42:57.397 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-11T06:42:57.397 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:1 2024-09-11T06:42:57.397 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:42:57.397 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:42:57.397 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:42:57.397 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=1 2024-09-11T06:42:57.397 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-11T06:42:57.397 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:42:57.397 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:42:57.398 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=1 2024-09-11T06:42:57.398 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=1 2024-09-11T06:42:57.399 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-11T06:42:57.399 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:1 2024-09-11T06:42:57.399 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:42:57.399 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:42:57.399 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T06:42:57.399 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:1 2024-09-11T06:42:57.399 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:42:57.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:42:57.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:42:57.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=1 2024-09-11T06:42:57.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T06:42:57.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:42:57.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:42:57.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=1 2024-09-11T06:42:57.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=1 2024-09-11T06:42:57.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T06:42:57.422 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:42:57.422 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:42:57.422 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:42:57.423 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_42 2024-09-11T06:42:57.559 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:42:57.560 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror image_42 2024-09-11T06:42:57.561 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:42:57.561 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:42:57.561 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_42 2024-09-11T06:42:57.561 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror image_42 '' 2024-09-11T06:42:57.561 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:42:57.561 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:42:57.561 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_42 2024-09-11T06:42:57.561 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:42:57.562 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-image_42.status 2024-09-11T06:42:57.562 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-image_42.status 2024-09-11T06:42:57.562 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:42:57.563 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_42.status 2024-09-11T06:42:57.563 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_42 2024-09-11T06:42:57.563 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_42.status 2024-09-11T06:42:57.614 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 2 2024-09-11T06:42:57.614 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 2 2024-09-11T06:42:57.614 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:42:57.614 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=3, entry_tid=1], [object_number=8, tag_tid=3, entry_tid=0], [object_number=11, tag_tid=2, entry_tid=3], [object_number=10, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:42:57.614 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=9, tag_tid=3, entry_tid=1], [object_number=8, tag_tid=3, entry_tid=0], [object_number=11, tag_tid=2, entry_tid=3], [object_number=10, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:42:57.618 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_42.status 2024-09-11T06:42:57.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=9, tag_tid=3, entry_tid=1' 2024-09-11T06:42:57.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror image_42 2024-09-11T06:42:57.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:42:57.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:42:57.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_42 2024-09-11T06:42:57.621 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror image_42 '..*' 2024-09-11T06:42:57.621 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:42:57.621 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:42:57.621 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_42 2024-09-11T06:42:57.621 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:42:57.621 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-image_42.status 2024-09-11T06:42:57.621 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-image_42.status 2024-09-11T06:42:57.621 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:42:57.621 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_42.status 2024-09-11T06:42:57.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_42 2024-09-11T06:42:57.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_42.status 2024-09-11T06:42:57.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 2 2024-09-11T06:42:57.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 2 2024-09-11T06:42:57.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:42:57.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=3, entry_tid=1], [object_number=8, tag_tid=3, entry_tid=0], [object_number=11, tag_tid=2, entry_tid=3], [object_number=10, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:42:57.668 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=9, tag_tid=3, entry_tid=1], [object_number=8, tag_tid=3, entry_tid=0], [object_number=11, tag_tid=2, entry_tid=3], [object_number=10, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:42:57.671 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_42.status 2024-09-11T06:42:57.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=9, tag_tid=3, entry_tid=1' 2024-09-11T06:42:57.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=9, tag_tid=3, entry_tid=1' -a 'object_number=9, tag_tid=3, entry_tid=1' = 'object_number=9, tag_tid=3, entry_tid=1' 2024-09-11T06:42:57.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:42:57.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_snap_present cluster1 mirror image_42 snap 2024-09-11T06:42:57.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:42:57.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:57.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_42 2024-09-11T06:42:57.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:42:57.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:42:57.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T06:42:57.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:42:58.674 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 info mirror/image_42@snap 2024-09-11T06:42:58.739 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:rbd image 'image_42': 2024-09-11T06:42:58.739 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: size 128 MiB in 32 objects 2024-09-11T06:42:58.739 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: order 22 (4 MiB objects) 2024-09-11T06:42:58.739 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: snapshot_count: 1 2024-09-11T06:42:58.739 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: id: 10a96d2220eb 2024-09-11T06:42:58.739 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: block_name_prefix: rbd_data.10a96d2220eb 2024-09-11T06:42:58.739 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: format: 2 2024-09-11T06:42:58.740 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: features: layering, exclusive-lock, journaling 2024-09-11T06:42:58.740 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: op_features: 2024-09-11T06:42:58.740 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: flags: 2024-09-11T06:42:58.740 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: create_timestamp: Wed Sep 11 06:37:15 2024 2024-09-11T06:42:58.740 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: access_timestamp: Wed Sep 11 06:37:15 2024 2024-09-11T06:42:58.740 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: modify_timestamp: Wed Sep 11 06:39:48 2024 2024-09-11T06:42:58.740 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: protected: False 2024-09-11T06:42:58.740 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: journal: 10a96d2220eb 2024-09-11T06:42:58.740 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring state: enabled 2024-09-11T06:42:58.740 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring mode: journal 2024-09-11T06:42:58.740 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring global id: 40cde00f-e413-44b1-9854-cbb977983778 2024-09-11T06:42:58.740 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring primary: false 2024-09-11T06:42:58.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:42:58.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ compare_image_snaps mirror image_42 snap 2024-09-11T06:42:58.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:58.745 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_42 2024-09-11T06:42:58.745 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:42:58.745 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local ret=0 2024-09-11T06:42:58.745 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local rmt_export=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_42.export 2024-09-11T06:42:58.745 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loc_export=/tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_42.export 2024-09-11T06:42:58.745 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_42.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_42.export 2024-09-11T06:42:58.745 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror export image_42@snap /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_42.export 2024-09-11T06:42:58.970 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T06:42:58.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 -p mirror export image_42@snap /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_42.export 2024-09-11T06:42:59.392 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T06:42:59.407 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmp /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_42.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_42.export 2024-09-11T06:42:59.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_42.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_42.export 2024-09-11T06:42:59.530 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:42:59.530 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:42:59.530 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_43 2024-09-11T06:42:59.530 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_snap cluster2 mirror image_43 snap 2024-09-11T06:42:59.530 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:42:59.530 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:42:59.530 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_43 2024-09-11T06:42:59.530 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:42:59.530 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror snap create image_43@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T06:43:00.708 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_started cluster1 mirror image_43 2024-09-11T06:43:00.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:43:00.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:00.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_43 2024-09-11T06:43:00.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_state cluster1 mirror image_43 started 2024-09-11T06:43:00.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:43:00.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:00.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_43 2024-09-11T06:43:00.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=started 2024-09-11T06:43:00.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:43:00.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:43:00.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:43:01.711 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_replay_state cluster1 mirror image_43 started 2024-09-11T06:43:01.711 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:43:01.711 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:01.711 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_43 2024-09-11T06:43:01.711 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=started 2024-09-11T06:43:01.711 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local status_result 2024-09-11T06:43:01.711 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=stopped 2024-09-11T06:43:01.712 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_43 2024-09-11T06:43:01.712 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_instance=cluster1 2024-09-11T06:43:01.712 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:43:01.712 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:43:01.712 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance=cluster1 2024-09-11T06:43:01.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local loop_instance 2024-09-11T06:43:01.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -i state 2024-09-11T06:43:01.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:43:01.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sleep 0 2024-09-11T06:43:01.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:43:01.713 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ seq 0 1 2024-09-11T06:43:01.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:43:01.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_43 2024-09-11T06:43:01.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1:0 2024-09-11T06:43:01.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:43:01.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:43:01.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:43:01.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:43:01.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=cluster 2024-09-11T06:43:01.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:43:01.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:43:01.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:43:01.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:43:01.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval cluster=cluster1 2024-09-11T06:43:01.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:43:01.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:43:01.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:43:01.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:43:01.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local local_cluster=cluster1:0 2024-09-11T06:43:01.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster=cluster1 2024-09-11T06:43:01.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance 2024-09-11T06:43:01.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:43:01.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local val=cluster1:0 2024-09-11T06:43:01.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T06:43:01.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance_var_name=instance 2024-09-11T06:43:01.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:43:01.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:43:01.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T06:43:01.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval local_cluster=cluster1 2024-09-11T06:43:01.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ local_cluster=cluster1 2024-09-11T06:43:01.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval instance=0 2024-09-11T06:43:01.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ instance=0 2024-09-11T06:43:01.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:43:01.740 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:43:01.740 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:43:01.740 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:43:01.741 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_43 2024-09-11T06:43:01.881 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ return 0 2024-09-11T06:43:01.882 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T06:43:01.883 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ echo ' "state": "Replaying"' 2024-09-11T06:43:01.883 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ grep -i Replaying 2024-09-11T06:43:01.884 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: "state": "Replaying" 2024-09-11T06:43:01.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_state=started 2024-09-11T06:43:01.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test started = started 2024-09-11T06:43:01.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:43:01.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_43 2024-09-11T06:43:01.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:43:01.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:43:01.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:01.887 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_43 2024-09-11T06:43:01.887 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' journal = journal ']' 2024-09-11T06:43:01.887 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_43 2024-09-11T06:43:01.887 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:43:01.887 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:43:01.887 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:01.887 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_43 2024-09-11T06:43:01.887 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-11T06:43:01.887 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-11T06:43:01.887 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:43:01.887 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:43:01.887 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:43:02.087 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror image_43 2024-09-11T06:43:02.088 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:43:02.088 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:02.088 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_43 2024-09-11T06:43:02.088 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:43:02.088 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n image_43 ']' 2024-09-11T06:43:02.088 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/image_43' 2024-09-11T06:43:02.088 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_43 2024-09-11T06:43:02.088 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:43:02.088 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:43:02.088 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:43:02.088 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:43:02.088 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:43:02.089 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:43:02.089 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:43:02.090 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:43:02.090 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:43:02.090 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:43:02.091 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_43 2024-09-11T06:43:02.091 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:43:02.091 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:43:02.091 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:43:02.091 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:43:02.091 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:43:02.091 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:43:02.091 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:43:02.091 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:43:02.091 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:43:02.091 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:43:02.092 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:43:02.092 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:43:02.092 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:43:02.092 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:43:02.093 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:43:02.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:43:02.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:43:02.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:43:02.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:43:02.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:43:02.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:43:02.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:43:02.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:43:02.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:43:02.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:43:02.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:43:02.094 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:43:02.095 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:43:02.095 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:43:02.095 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:43:02.118 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:43:02.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:43:02.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:43:02.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_43 2024-09-11T06:43:02.261 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:43:02.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror image_43 2024-09-11T06:43:02.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:43:02.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:43:02.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_43 2024-09-11T06:43:02.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror image_43 '' 2024-09-11T06:43:02.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:43:02.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:43:02.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_43 2024-09-11T06:43:02.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:43:02.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-image_43.status 2024-09-11T06:43:02.264 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-image_43.status 2024-09-11T06:43:02.264 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:43:02.265 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_43.status 2024-09-11T06:43:02.265 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_43 2024-09-11T06:43:02.266 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_43.status 2024-09-11T06:43:02.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 2 2024-09-11T06:43:02.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 2 2024-09-11T06:43:02.310 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:43:02.310 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=3, entry_tid=1], [object_number=8, tag_tid=3, entry_tid=0], [object_number=11, tag_tid=2, entry_tid=3], [object_number=10, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:43:02.310 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=9, tag_tid=3, entry_tid=1], [object_number=8, tag_tid=3, entry_tid=0], [object_number=11, tag_tid=2, entry_tid=3], [object_number=10, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:43:02.313 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_43.status 2024-09-11T06:43:02.315 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=9, tag_tid=3, entry_tid=1' 2024-09-11T06:43:02.315 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror image_43 2024-09-11T06:43:02.315 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:43:02.315 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:43:02.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_43 2024-09-11T06:43:02.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror image_43 '..*' 2024-09-11T06:43:02.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:43:02.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:43:02.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_43 2024-09-11T06:43:02.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:43:02.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-image_43.status 2024-09-11T06:43:02.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-image_43.status 2024-09-11T06:43:02.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:43:02.317 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_43.status 2024-09-11T06:43:02.317 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_43 2024-09-11T06:43:02.317 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_43.status 2024-09-11T06:43:02.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 2 2024-09-11T06:43:02.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 2 2024-09-11T06:43:02.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:43:02.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=3, entry_tid=1], [object_number=8, tag_tid=3, entry_tid=0], [object_number=11, tag_tid=2, entry_tid=3], [object_number=10, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:43:02.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=9, tag_tid=3, entry_tid=1], [object_number=8, tag_tid=3, entry_tid=0], [object_number=11, tag_tid=2, entry_tid=3], [object_number=10, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:43:02.364 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_43.status 2024-09-11T06:43:02.366 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=9, tag_tid=3, entry_tid=1' 2024-09-11T06:43:02.366 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=9, tag_tid=3, entry_tid=1' -a 'object_number=9, tag_tid=3, entry_tid=1' = 'object_number=9, tag_tid=3, entry_tid=1' 2024-09-11T06:43:02.366 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:43:02.366 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_snap_present cluster1 mirror image_43 snap 2024-09-11T06:43:02.366 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:43:02.367 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:02.367 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_43 2024-09-11T06:43:02.367 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:43:02.367 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:43:02.367 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T06:43:02.367 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:43:03.367 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 info mirror/image_43@snap 2024-09-11T06:43:03.434 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:rbd image 'image_43': 2024-09-11T06:43:03.434 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: size 128 MiB in 32 objects 2024-09-11T06:43:03.434 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: order 22 (4 MiB objects) 2024-09-11T06:43:03.434 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: snapshot_count: 1 2024-09-11T06:43:03.434 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: id: 10a6f4591736 2024-09-11T06:43:03.435 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: block_name_prefix: rbd_data.10a6f4591736 2024-09-11T06:43:03.435 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: format: 2 2024-09-11T06:43:03.435 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: features: layering, exclusive-lock, journaling 2024-09-11T06:43:03.435 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: op_features: 2024-09-11T06:43:03.435 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: flags: 2024-09-11T06:43:03.435 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: create_timestamp: Wed Sep 11 06:37:31 2024 2024-09-11T06:43:03.435 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: access_timestamp: Wed Sep 11 06:37:31 2024 2024-09-11T06:43:03.435 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: modify_timestamp: Wed Sep 11 06:40:07 2024 2024-09-11T06:43:03.435 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: protected: False 2024-09-11T06:43:03.435 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: journal: 10a6f4591736 2024-09-11T06:43:03.435 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring state: enabled 2024-09-11T06:43:03.435 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring mode: journal 2024-09-11T06:43:03.435 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring global id: a0b0ee92-8aab-4d85-80b9-3178d96d315b 2024-09-11T06:43:03.436 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring primary: false 2024-09-11T06:43:03.439 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:43:03.439 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ compare_image_snaps mirror image_43 snap 2024-09-11T06:43:03.439 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:03.439 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_43 2024-09-11T06:43:03.439 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:43:03.439 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local ret=0 2024-09-11T06:43:03.439 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local rmt_export=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_43.export 2024-09-11T06:43:03.439 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loc_export=/tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_43.export 2024-09-11T06:43:03.439 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_43.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_43.export 2024-09-11T06:43:03.440 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror export image_43@snap /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_43.export 2024-09-11T06:43:03.632 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T06:43:03.640 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 -p mirror export image_43@snap /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_43.export 2024-09-11T06:43:04.161 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T06:43:04.175 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmp /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_43.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_43.export 2024-09-11T06:43:04.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_43.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_43.export 2024-09-11T06:43:04.294 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:43:04.294 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:43:04.294 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_44 2024-09-11T06:43:04.294 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_snap cluster2 mirror image_44 snap 2024-09-11T06:43:04.294 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:43:04.294 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:04.294 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_44 2024-09-11T06:43:04.294 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:43:04.294 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror snap create image_44@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T06:43:04.729 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_started cluster1 mirror image_44 2024-09-11T06:43:04.730 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:43:04.730 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:04.730 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_44 2024-09-11T06:43:04.730 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_state cluster1 mirror image_44 started 2024-09-11T06:43:04.730 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:43:04.730 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:04.730 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_44 2024-09-11T06:43:04.730 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=started 2024-09-11T06:43:04.730 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:43:04.730 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:43:04.730 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:43:05.731 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_replay_state cluster1 mirror image_44 started 2024-09-11T06:43:05.732 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:43:05.732 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:05.732 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_44 2024-09-11T06:43:05.732 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=started 2024-09-11T06:43:05.732 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local status_result 2024-09-11T06:43:05.732 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=stopped 2024-09-11T06:43:05.734 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_44 2024-09-11T06:43:05.734 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_instance=cluster1 2024-09-11T06:43:05.734 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:43:05.734 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:43:05.734 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance=cluster1 2024-09-11T06:43:05.735 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local loop_instance 2024-09-11T06:43:05.735 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:43:05.735 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sleep 0 2024-09-11T06:43:05.735 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -i state 2024-09-11T06:43:05.735 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:43:05.741 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ seq 0 1 2024-09-11T06:43:05.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:43:05.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_44 2024-09-11T06:43:05.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1:0 2024-09-11T06:43:05.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:43:05.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:43:05.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:43:05.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:43:05.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=cluster 2024-09-11T06:43:05.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:43:05.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:43:05.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:43:05.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:43:05.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval cluster=cluster1 2024-09-11T06:43:05.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:43:05.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:43:05.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:43:05.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:43:05.745 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local local_cluster=cluster1:0 2024-09-11T06:43:05.745 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster=cluster1 2024-09-11T06:43:05.745 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance 2024-09-11T06:43:05.745 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:43:05.745 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local val=cluster1:0 2024-09-11T06:43:05.745 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T06:43:05.746 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance_var_name=instance 2024-09-11T06:43:05.746 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:43:05.746 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:43:05.746 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T06:43:05.746 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval local_cluster=cluster1 2024-09-11T06:43:05.746 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ local_cluster=cluster1 2024-09-11T06:43:05.747 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval instance=0 2024-09-11T06:43:05.747 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ instance=0 2024-09-11T06:43:05.747 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:43:05.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:43:05.764 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:43:05.764 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:43:05.764 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_44 2024-09-11T06:43:05.890 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:Invalid command: unused arguments: ['mirror/image_44'] 2024-09-11T06:43:05.890 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T06:43:05.890 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:admin_socket: invalid command 2024-09-11T06:43:05.893 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:43:05.893 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_44 2024-09-11T06:43:05.893 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1:1 2024-09-11T06:43:05.893 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:43:05.893 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:43:05.893 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-11T06:43:05.893 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:1 2024-09-11T06:43:05.893 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=cluster 2024-09-11T06:43:05.893 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:43:05.893 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:43:05.893 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=1 2024-09-11T06:43:05.893 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T06:43:05.894 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval cluster=cluster1 2024-09-11T06:43:05.894 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:43:05.894 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=1 2024-09-11T06:43:05.894 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=1 2024-09-11T06:43:05.895 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-11T06:43:05.895 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local local_cluster=cluster1:1 2024-09-11T06:43:05.895 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster=cluster1 2024-09-11T06:43:05.895 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance 2024-09-11T06:43:05.895 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T06:43:05.895 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local val=cluster1:1 2024-09-11T06:43:05.896 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T06:43:05.896 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance_var_name=instance 2024-09-11T06:43:05.896 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:43:05.896 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=1 2024-09-11T06:43:05.896 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-11T06:43:05.896 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval local_cluster=cluster1 2024-09-11T06:43:05.896 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ local_cluster=cluster1 2024-09-11T06:43:05.896 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval instance=1 2024-09-11T06:43:05.896 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ instance=1 2024-09-11T06:43:05.896 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T06:43:05.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:43:05.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:43:05.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:43:05.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_44 2024-09-11T06:43:06.057 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ return 0 2024-09-11T06:43:06.057 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T06:43:06.058 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ echo ' "state": "Replaying"' 2024-09-11T06:43:06.058 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ grep -i Replaying 2024-09-11T06:43:06.060 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: "state": "Replaying" 2024-09-11T06:43:06.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_state=started 2024-09-11T06:43:06.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test started = started 2024-09-11T06:43:06.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:43:06.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_44 2024-09-11T06:43:06.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:43:06.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:43:06.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:06.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_44 2024-09-11T06:43:06.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' journal = journal ']' 2024-09-11T06:43:06.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_44 2024-09-11T06:43:06.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:43:06.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:43:06.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:06.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_44 2024-09-11T06:43:06.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-11T06:43:06.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-11T06:43:06.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:43:06.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:43:06.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:43:06.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror image_44 2024-09-11T06:43:06.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:43:06.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:06.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_44 2024-09-11T06:43:06.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:43:06.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n image_44 ']' 2024-09-11T06:43:06.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/image_44' 2024-09-11T06:43:06.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_44 2024-09-11T06:43:06.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:43:06.264 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:43:06.264 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:43:06.264 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:43:06.264 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:43:06.264 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:43:06.264 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:43:06.264 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:43:06.265 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:43:06.266 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:43:06.266 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_44 2024-09-11T06:43:06.266 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:43:06.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:43:06.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:43:06.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:43:06.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:43:06.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:43:06.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:43:06.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:43:06.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:43:06.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:43:06.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:43:06.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:43:06.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:43:06.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:43:06.269 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:43:06.269 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:43:06.269 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:43:06.269 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:43:06.269 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:43:06.269 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:43:06.269 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:43:06.269 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:43:06.269 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:43:06.269 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:43:06.269 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:43:06.269 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:43:06.270 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:43:06.270 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:43:06.270 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:43:06.270 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:43:06.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:43:06.293 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:43:06.293 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:43:06.293 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_44 2024-09-11T06:43:06.398 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:43:06.393+0000 7efd718c5640 -1 reset not still connected to 0x560b3a5f0750 2024-09-11T06:43:06.427 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:Invalid command: unused arguments: ['mirror/image_44'] 2024-09-11T06:43:06.427 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd mirror flush : flush rbd mirror 2024-09-11T06:43:06.427 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:admin_socket: invalid command 2024-09-11T06:43:06.430 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:43:06.430 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_44 2024-09-11T06:43:06.430 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:1 2024-09-11T06:43:06.430 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:43:06.430 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:43:06.430 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-11T06:43:06.430 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:1 2024-09-11T06:43:06.430 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:43:06.430 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:43:06.431 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:43:06.431 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=1 2024-09-11T06:43:06.431 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-11T06:43:06.431 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:43:06.431 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:43:06.431 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=1 2024-09-11T06:43:06.431 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=1 2024-09-11T06:43:06.432 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-11T06:43:06.433 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:1 2024-09-11T06:43:06.433 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:43:06.433 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:43:06.433 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T06:43:06.433 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:1 2024-09-11T06:43:06.433 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:43:06.433 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:43:06.433 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:43:06.433 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=1 2024-09-11T06:43:06.433 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T06:43:06.433 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:43:06.433 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:43:06.433 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=1 2024-09-11T06:43:06.433 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=1 2024-09-11T06:43:06.434 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T06:43:06.456 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:43:06.456 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:43:06.456 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:43:06.456 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_44 2024-09-11T06:43:06.594 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:43:06.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror image_44 2024-09-11T06:43:06.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:43:06.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:43:06.595 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_44 2024-09-11T06:43:06.596 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror image_44 '' 2024-09-11T06:43:06.596 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:43:06.596 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:43:06.596 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_44 2024-09-11T06:43:06.596 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:43:06.596 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-image_44.status 2024-09-11T06:43:06.596 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-image_44.status 2024-09-11T06:43:06.596 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:43:06.598 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_44.status 2024-09-11T06:43:06.599 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_44 2024-09-11T06:43:06.599 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_44.status 2024-09-11T06:43:06.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 1 2024-09-11T06:43:06.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 1 2024-09-11T06:43:06.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:43:06.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:43:06.644 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:43:06.647 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_44.status 2024-09-11T06:43:06.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-09-11T06:43:06.649 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror image_44 2024-09-11T06:43:06.649 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:43:06.650 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:43:06.650 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_44 2024-09-11T06:43:06.650 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror image_44 '..*' 2024-09-11T06:43:06.650 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:43:06.650 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:43:06.650 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_44 2024-09-11T06:43:06.650 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:43:06.650 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-image_44.status 2024-09-11T06:43:06.650 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-image_44.status 2024-09-11T06:43:06.650 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:43:06.651 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_44.status 2024-09-11T06:43:06.651 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_44 2024-09-11T06:43:06.651 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_44.status 2024-09-11T06:43:06.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 1 2024-09-11T06:43:06.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 1 2024-09-11T06:43:06.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:43:06.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:43:06.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:43:06.702 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_44.status 2024-09-11T06:43:06.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-09-11T06:43:06.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=3, entry_tid=1' 2024-09-11T06:43:06.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:43:06.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_snap_present cluster1 mirror image_44 snap 2024-09-11T06:43:06.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:43:06.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:06.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_44 2024-09-11T06:43:06.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:43:06.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:43:06.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T06:43:06.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:43:07.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 info mirror/image_44@snap 2024-09-11T06:43:07.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:rbd image 'image_44': 2024-09-11T06:43:07.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: size 128 MiB in 32 objects 2024-09-11T06:43:07.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: order 22 (4 MiB objects) 2024-09-11T06:43:07.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: snapshot_count: 1 2024-09-11T06:43:07.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: id: 10a933177537 2024-09-11T06:43:07.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: block_name_prefix: rbd_data.10a933177537 2024-09-11T06:43:07.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: format: 2 2024-09-11T06:43:07.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: features: layering, exclusive-lock, journaling 2024-09-11T06:43:07.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: op_features: 2024-09-11T06:43:07.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: flags: 2024-09-11T06:43:07.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: create_timestamp: Wed Sep 11 06:37:44 2024 2024-09-11T06:43:07.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: access_timestamp: Wed Sep 11 06:37:44 2024 2024-09-11T06:43:07.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: modify_timestamp: Wed Sep 11 06:39:03 2024 2024-09-11T06:43:07.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: protected: False 2024-09-11T06:43:07.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: journal: 10a933177537 2024-09-11T06:43:07.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring state: enabled 2024-09-11T06:43:07.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring mode: journal 2024-09-11T06:43:07.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring global id: b71b944f-365f-4640-866c-e87fd45af096 2024-09-11T06:43:07.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring primary: false 2024-09-11T06:43:07.775 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:43:07.775 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ compare_image_snaps mirror image_44 snap 2024-09-11T06:43:07.775 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:07.775 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_44 2024-09-11T06:43:07.775 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:43:07.775 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local ret=0 2024-09-11T06:43:07.775 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local rmt_export=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_44.export 2024-09-11T06:43:07.775 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loc_export=/tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_44.export 2024-09-11T06:43:07.775 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_44.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_44.export 2024-09-11T06:43:07.776 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror export image_44@snap /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_44.export 2024-09-11T06:43:07.929 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T06:43:07.937 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 -p mirror export image_44@snap /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_44.export 2024-09-11T06:43:08.444 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T06:43:08.454 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmp /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_44.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_44.export 2024-09-11T06:43:08.549 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_44.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_44.export 2024-09-11T06:43:08.586 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:43:08.587 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:43:08.587 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_45 2024-09-11T06:43:08.587 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_snap cluster2 mirror image_45 snap 2024-09-11T06:43:08.587 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:43:08.587 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:08.587 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_45 2024-09-11T06:43:08.587 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:43:08.587 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror snap create image_45@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T06:43:09.751 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_started cluster1 mirror image_45 2024-09-11T06:43:09.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:43:09.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:09.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_45 2024-09-11T06:43:09.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_state cluster1 mirror image_45 started 2024-09-11T06:43:09.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:43:09.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:09.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_45 2024-09-11T06:43:09.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=started 2024-09-11T06:43:09.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:43:09.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:43:09.753 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:43:10.753 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_replay_state cluster1 mirror image_45 started 2024-09-11T06:43:10.753 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:43:10.753 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:10.754 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_45 2024-09-11T06:43:10.754 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=started 2024-09-11T06:43:10.754 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local status_result 2024-09-11T06:43:10.754 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=stopped 2024-09-11T06:43:10.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_45 2024-09-11T06:43:10.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -i state 2024-09-11T06:43:10.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_instance=cluster1 2024-09-11T06:43:10.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:43:10.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:43:10.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance=cluster1 2024-09-11T06:43:10.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local loop_instance 2024-09-11T06:43:10.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:43:10.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sleep 0 2024-09-11T06:43:10.756 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:43:10.756 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ seq 0 1 2024-09-11T06:43:10.757 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:43:10.758 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_45 2024-09-11T06:43:10.758 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1:0 2024-09-11T06:43:10.758 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:43:10.758 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:43:10.758 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:43:10.758 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:43:10.758 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=cluster 2024-09-11T06:43:10.758 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:43:10.758 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:43:10.758 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:43:10.758 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:43:10.758 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval cluster=cluster1 2024-09-11T06:43:10.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:43:10.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:43:10.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:43:10.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:43:10.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local local_cluster=cluster1:0 2024-09-11T06:43:10.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster=cluster1 2024-09-11T06:43:10.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance 2024-09-11T06:43:10.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:43:10.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local val=cluster1:0 2024-09-11T06:43:10.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T06:43:10.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance_var_name=instance 2024-09-11T06:43:10.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:43:10.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:43:10.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T06:43:10.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval local_cluster=cluster1 2024-09-11T06:43:10.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ local_cluster=cluster1 2024-09-11T06:43:10.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval instance=0 2024-09-11T06:43:10.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ instance=0 2024-09-11T06:43:10.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:43:10.783 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:43:10.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:43:10.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:43:10.784 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_45 2024-09-11T06:43:10.922 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ return 0 2024-09-11T06:43:10.923 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T06:43:10.923 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ echo ' "state": "Replaying"' 2024-09-11T06:43:10.924 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ grep -i Replaying 2024-09-11T06:43:10.925 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: "state": "Replaying" 2024-09-11T06:43:10.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_state=started 2024-09-11T06:43:10.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test started = started 2024-09-11T06:43:10.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:43:10.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_45 2024-09-11T06:43:10.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:43:10.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:43:10.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:10.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_45 2024-09-11T06:43:10.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' journal = journal ']' 2024-09-11T06:43:10.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_45 2024-09-11T06:43:10.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:43:10.928 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:43:10.928 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:10.928 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_45 2024-09-11T06:43:10.928 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-11T06:43:10.928 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-11T06:43:10.928 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:43:10.928 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:43:10.928 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:43:11.128 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror image_45 2024-09-11T06:43:11.128 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:43:11.128 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:11.128 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_45 2024-09-11T06:43:11.128 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:43:11.128 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n image_45 ']' 2024-09-11T06:43:11.129 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/image_45' 2024-09-11T06:43:11.129 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_45 2024-09-11T06:43:11.129 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:43:11.129 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:43:11.129 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:43:11.129 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:43:11.129 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:43:11.129 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:43:11.129 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:43:11.130 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:43:11.130 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:43:11.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:43:11.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_45 2024-09-11T06:43:11.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:43:11.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:43:11.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:43:11.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:43:11.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:43:11.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:43:11.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:43:11.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:43:11.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:43:11.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:43:11.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:43:11.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:43:11.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:43:11.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:43:11.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:43:11.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:43:11.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:43:11.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:43:11.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:43:11.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:43:11.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:43:11.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:43:11.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:43:11.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:43:11.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:43:11.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:43:11.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:43:11.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:43:11.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:43:11.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:43:11.159 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:43:11.159 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:43:11.159 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:43:11.159 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_45 2024-09-11T06:43:11.302 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:43:11.303 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror image_45 2024-09-11T06:43:11.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:43:11.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:43:11.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_45 2024-09-11T06:43:11.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror image_45 '' 2024-09-11T06:43:11.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:43:11.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:43:11.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_45 2024-09-11T06:43:11.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:43:11.305 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-image_45.status 2024-09-11T06:43:11.305 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-image_45.status 2024-09-11T06:43:11.305 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:43:11.306 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_45.status 2024-09-11T06:43:11.306 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_45 2024-09-11T06:43:11.307 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_45.status 2024-09-11T06:43:11.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 2 2024-09-11T06:43:11.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 2 2024-09-11T06:43:11.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:43:11.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=3, entry_tid=1], [object_number=8, tag_tid=3, entry_tid=0], [object_number=11, tag_tid=2, entry_tid=3], [object_number=10, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:43:11.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=9, tag_tid=3, entry_tid=1], [object_number=8, tag_tid=3, entry_tid=0], [object_number=11, tag_tid=2, entry_tid=3], [object_number=10, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:43:11.363 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_45.status 2024-09-11T06:43:11.366 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=9, tag_tid=3, entry_tid=1' 2024-09-11T06:43:11.366 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror image_45 2024-09-11T06:43:11.366 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:43:11.366 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:43:11.366 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_45 2024-09-11T06:43:11.366 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror image_45 '..*' 2024-09-11T06:43:11.366 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:43:11.366 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:43:11.366 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_45 2024-09-11T06:43:11.366 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:43:11.366 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-image_45.status 2024-09-11T06:43:11.366 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-image_45.status 2024-09-11T06:43:11.366 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:43:11.367 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_45.status 2024-09-11T06:43:11.367 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_45 2024-09-11T06:43:11.367 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_45.status 2024-09-11T06:43:11.413 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 2 2024-09-11T06:43:11.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 2 2024-09-11T06:43:11.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:43:11.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=3, entry_tid=1], [object_number=8, tag_tid=3, entry_tid=0], [object_number=11, tag_tid=2, entry_tid=3], [object_number=10, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:43:11.414 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=9, tag_tid=3, entry_tid=1], [object_number=8, tag_tid=3, entry_tid=0], [object_number=11, tag_tid=2, entry_tid=3], [object_number=10, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:43:11.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_45.status 2024-09-11T06:43:11.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=9, tag_tid=3, entry_tid=1' 2024-09-11T06:43:11.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=9, tag_tid=3, entry_tid=1' -a 'object_number=9, tag_tid=3, entry_tid=1' = 'object_number=9, tag_tid=3, entry_tid=1' 2024-09-11T06:43:11.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:43:11.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_snap_present cluster1 mirror image_45 snap 2024-09-11T06:43:11.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:43:11.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:11.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_45 2024-09-11T06:43:11.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:43:11.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:43:11.420 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T06:43:11.420 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:43:12.420 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 info mirror/image_45@snap 2024-09-11T06:43:12.483 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:rbd image 'image_45': 2024-09-11T06:43:12.483 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: size 128 MiB in 32 objects 2024-09-11T06:43:12.483 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: order 22 (4 MiB objects) 2024-09-11T06:43:12.483 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: snapshot_count: 1 2024-09-11T06:43:12.483 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: id: 10a6fda0e885 2024-09-11T06:43:12.483 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: block_name_prefix: rbd_data.10a6fda0e885 2024-09-11T06:43:12.483 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: format: 2 2024-09-11T06:43:12.483 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: features: layering, exclusive-lock, journaling 2024-09-11T06:43:12.483 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: op_features: 2024-09-11T06:43:12.483 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: flags: 2024-09-11T06:43:12.483 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: create_timestamp: Wed Sep 11 06:37:47 2024 2024-09-11T06:43:12.484 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: access_timestamp: Wed Sep 11 06:37:47 2024 2024-09-11T06:43:12.484 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: modify_timestamp: Wed Sep 11 06:39:59 2024 2024-09-11T06:43:12.484 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: protected: False 2024-09-11T06:43:12.484 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: journal: 10a6fda0e885 2024-09-11T06:43:12.484 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring state: enabled 2024-09-11T06:43:12.484 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring mode: journal 2024-09-11T06:43:12.484 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring global id: 6c5f85d9-7b3e-49a2-b42c-d35c6616bc15 2024-09-11T06:43:12.484 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring primary: false 2024-09-11T06:43:12.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:43:12.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ compare_image_snaps mirror image_45 snap 2024-09-11T06:43:12.487 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:12.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_45 2024-09-11T06:43:12.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:43:12.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local ret=0 2024-09-11T06:43:12.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local rmt_export=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_45.export 2024-09-11T06:43:12.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loc_export=/tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_45.export 2024-09-11T06:43:12.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_45.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_45.export 2024-09-11T06:43:12.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror export image_45@snap /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_45.export 2024-09-11T06:43:12.681 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T06:43:12.689 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 -p mirror export image_45@snap /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_45.export 2024-09-11T06:43:13.051 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T06:43:13.063 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmp /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_45.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_45.export 2024-09-11T06:43:13.139 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_45.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_45.export 2024-09-11T06:43:13.175 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:43:13.175 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:43:13.175 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_46 2024-09-11T06:43:13.175 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_snap cluster2 mirror image_46 snap 2024-09-11T06:43:13.175 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:43:13.176 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:13.176 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_46 2024-09-11T06:43:13.176 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:43:13.176 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror snap create image_46@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T06:43:13.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_started cluster1 mirror image_46 2024-09-11T06:43:13.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:43:13.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:13.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_46 2024-09-11T06:43:13.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_state cluster1 mirror image_46 started 2024-09-11T06:43:13.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:43:13.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:13.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_46 2024-09-11T06:43:13.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=started 2024-09-11T06:43:13.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:43:13.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:43:13.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:43:14.768 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_replay_state cluster1 mirror image_46 started 2024-09-11T06:43:14.768 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:43:14.768 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:14.768 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_46 2024-09-11T06:43:14.768 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=started 2024-09-11T06:43:14.768 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local status_result 2024-09-11T06:43:14.768 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=stopped 2024-09-11T06:43:14.769 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_46 2024-09-11T06:43:14.769 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_instance=cluster1 2024-09-11T06:43:14.769 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:43:14.769 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:43:14.769 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance=cluster1 2024-09-11T06:43:14.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local loop_instance 2024-09-11T06:43:14.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:43:14.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sleep 0 2024-09-11T06:43:14.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -i state 2024-09-11T06:43:14.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:43:14.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ seq 0 1 2024-09-11T06:43:14.773 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:43:14.773 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_46 2024-09-11T06:43:14.773 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1:0 2024-09-11T06:43:14.773 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:43:14.773 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:43:14.773 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:43:14.773 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:43:14.773 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=cluster 2024-09-11T06:43:14.773 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:43:14.773 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:43:14.774 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:43:14.774 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:43:14.774 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval cluster=cluster1 2024-09-11T06:43:14.774 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:43:14.774 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:43:14.774 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:43:14.776 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:43:14.776 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local local_cluster=cluster1:0 2024-09-11T06:43:14.776 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster=cluster1 2024-09-11T06:43:14.776 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance 2024-09-11T06:43:14.776 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:43:14.776 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local val=cluster1:0 2024-09-11T06:43:14.776 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T06:43:14.776 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance_var_name=instance 2024-09-11T06:43:14.776 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:43:14.776 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:43:14.776 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T06:43:14.777 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval local_cluster=cluster1 2024-09-11T06:43:14.777 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ local_cluster=cluster1 2024-09-11T06:43:14.777 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval instance=0 2024-09-11T06:43:14.777 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ instance=0 2024-09-11T06:43:14.777 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:43:14.795 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:43:14.796 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:43:14.796 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:43:14.796 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_46 2024-09-11T06:43:14.954 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:Invalid command: unused arguments: ['mirror/image_46'] 2024-09-11T06:43:14.954 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T06:43:14.955 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:admin_socket: invalid command 2024-09-11T06:43:14.958 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:43:14.958 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_46 2024-09-11T06:43:14.958 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1:1 2024-09-11T06:43:14.958 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:43:14.958 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:43:14.958 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-11T06:43:14.958 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:1 2024-09-11T06:43:14.958 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=cluster 2024-09-11T06:43:14.958 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:43:14.958 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:43:14.959 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=1 2024-09-11T06:43:14.959 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T06:43:14.959 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval cluster=cluster1 2024-09-11T06:43:14.959 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:43:14.959 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=1 2024-09-11T06:43:14.959 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=1 2024-09-11T06:43:14.959 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-11T06:43:14.959 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local local_cluster=cluster1:1 2024-09-11T06:43:14.959 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster=cluster1 2024-09-11T06:43:14.959 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance 2024-09-11T06:43:14.959 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T06:43:14.959 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local val=cluster1:1 2024-09-11T06:43:14.959 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T06:43:14.960 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance_var_name=instance 2024-09-11T06:43:14.960 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:43:14.960 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=1 2024-09-11T06:43:14.960 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-11T06:43:14.960 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval local_cluster=cluster1 2024-09-11T06:43:14.960 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ local_cluster=cluster1 2024-09-11T06:43:14.960 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval instance=1 2024-09-11T06:43:14.960 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ instance=1 2024-09-11T06:43:14.960 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T06:43:14.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:43:14.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:43:14.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:43:14.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_46 2024-09-11T06:43:15.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ return 0 2024-09-11T06:43:15.112 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T06:43:15.113 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ echo ' "state": "Replaying"' 2024-09-11T06:43:15.113 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ grep -i Replaying 2024-09-11T06:43:15.115 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: "state": "Replaying" 2024-09-11T06:43:15.116 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_state=started 2024-09-11T06:43:15.116 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test started = started 2024-09-11T06:43:15.116 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:43:15.116 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_46 2024-09-11T06:43:15.116 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:43:15.116 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:43:15.116 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:15.116 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_46 2024-09-11T06:43:15.116 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' journal = journal ']' 2024-09-11T06:43:15.116 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_46 2024-09-11T06:43:15.116 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:43:15.117 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:43:15.117 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:15.117 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_46 2024-09-11T06:43:15.117 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-11T06:43:15.117 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-11T06:43:15.117 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:43:15.117 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:43:15.117 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:43:15.317 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror image_46 2024-09-11T06:43:15.317 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:43:15.318 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:15.318 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_46 2024-09-11T06:43:15.318 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:43:15.318 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n image_46 ']' 2024-09-11T06:43:15.318 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/image_46' 2024-09-11T06:43:15.318 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_46 2024-09-11T06:43:15.318 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:43:15.318 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:43:15.318 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:43:15.318 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:43:15.318 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:43:15.319 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:43:15.319 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:43:15.319 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:43:15.319 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:43:15.322 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:43:15.322 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_46 2024-09-11T06:43:15.322 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:43:15.322 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:43:15.323 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:43:15.323 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:43:15.323 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:43:15.323 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:43:15.323 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:43:15.323 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:43:15.323 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:43:15.323 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:43:15.323 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:43:15.323 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:43:15.323 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:43:15.323 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:43:15.323 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:43:15.324 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:43:15.324 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:43:15.324 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:43:15.324 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:43:15.324 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:43:15.324 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:43:15.324 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:43:15.324 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:43:15.324 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:43:15.324 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:43:15.324 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:43:15.324 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:43:15.324 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:43:15.324 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:43:15.325 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:43:15.349 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:43:15.349 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:43:15.349 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:43:15.349 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_46 2024-09-11T06:43:15.468 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:Invalid command: unused arguments: ['mirror/image_46'] 2024-09-11T06:43:15.468 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd mirror flush : flush rbd mirror 2024-09-11T06:43:15.468 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:admin_socket: invalid command 2024-09-11T06:43:15.469 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:43:15.469 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_46 2024-09-11T06:43:15.470 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:1 2024-09-11T06:43:15.470 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:43:15.470 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:43:15.470 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-11T06:43:15.470 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:1 2024-09-11T06:43:15.470 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:43:15.470 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:43:15.470 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:43:15.470 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=1 2024-09-11T06:43:15.470 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-11T06:43:15.471 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:43:15.471 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:43:15.471 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=1 2024-09-11T06:43:15.471 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=1 2024-09-11T06:43:15.472 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-11T06:43:15.472 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:1 2024-09-11T06:43:15.472 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:43:15.472 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:43:15.473 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T06:43:15.473 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:1 2024-09-11T06:43:15.473 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:43:15.473 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:43:15.473 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:43:15.473 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=1 2024-09-11T06:43:15.473 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T06:43:15.473 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:43:15.473 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:43:15.473 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=1 2024-09-11T06:43:15.473 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=1 2024-09-11T06:43:15.473 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T06:43:15.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:43:15.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:43:15.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:43:15.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_46 2024-09-11T06:43:15.631 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:43:15.632 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror image_46 2024-09-11T06:43:15.632 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:43:15.632 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:43:15.632 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_46 2024-09-11T06:43:15.632 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror image_46 '' 2024-09-11T06:43:15.632 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:43:15.632 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:43:15.633 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_46 2024-09-11T06:43:15.633 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:43:15.633 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-image_46.status 2024-09-11T06:43:15.633 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-image_46.status 2024-09-11T06:43:15.633 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:43:15.635 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_46.status 2024-09-11T06:43:15.635 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_46 2024-09-11T06:43:15.635 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_46.status 2024-09-11T06:43:15.890 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 1 2024-09-11T06:43:15.890 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 1 2024-09-11T06:43:15.890 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:43:15.890 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:43:15.890 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:43:15.894 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_46.status 2024-09-11T06:43:15.895 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-09-11T06:43:15.896 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror image_46 2024-09-11T06:43:15.896 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:43:15.897 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:43:15.897 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_46 2024-09-11T06:43:15.897 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror image_46 '..*' 2024-09-11T06:43:15.897 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:43:15.897 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:43:15.897 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_46 2024-09-11T06:43:15.897 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:43:15.897 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-image_46.status 2024-09-11T06:43:15.897 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-image_46.status 2024-09-11T06:43:15.898 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:43:15.898 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_46.status 2024-09-11T06:43:15.898 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_46 2024-09-11T06:43:15.899 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_46.status 2024-09-11T06:43:15.949 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 1 2024-09-11T06:43:15.949 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 1 2024-09-11T06:43:15.949 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:43:15.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:43:15.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:43:15.953 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_46.status 2024-09-11T06:43:15.955 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-09-11T06:43:15.955 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=3, entry_tid=1' 2024-09-11T06:43:15.956 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:43:15.956 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_snap_present cluster1 mirror image_46 snap 2024-09-11T06:43:15.956 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:43:15.956 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:15.956 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_46 2024-09-11T06:43:15.956 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:43:15.956 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:43:15.956 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T06:43:15.956 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:43:16.956 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 info mirror/image_46@snap 2024-09-11T06:43:17.024 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:rbd image 'image_46': 2024-09-11T06:43:17.025 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: size 128 MiB in 32 objects 2024-09-11T06:43:17.025 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: order 22 (4 MiB objects) 2024-09-11T06:43:17.025 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: snapshot_count: 1 2024-09-11T06:43:17.025 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: id: 10a91af4a233 2024-09-11T06:43:17.025 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: block_name_prefix: rbd_data.10a91af4a233 2024-09-11T06:43:17.025 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: format: 2 2024-09-11T06:43:17.025 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: features: layering, exclusive-lock, journaling 2024-09-11T06:43:17.025 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: op_features: 2024-09-11T06:43:17.025 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: flags: 2024-09-11T06:43:17.025 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: create_timestamp: Wed Sep 11 06:38:01 2024 2024-09-11T06:43:17.025 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: access_timestamp: Wed Sep 11 06:38:01 2024 2024-09-11T06:43:17.026 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: modify_timestamp: Wed Sep 11 06:39:03 2024 2024-09-11T06:43:17.026 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: protected: False 2024-09-11T06:43:17.026 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: journal: 10a91af4a233 2024-09-11T06:43:17.026 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring state: enabled 2024-09-11T06:43:17.026 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring mode: journal 2024-09-11T06:43:17.026 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring global id: cbb33d98-e015-4042-8184-f6f673dbf890 2024-09-11T06:43:17.026 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring primary: false 2024-09-11T06:43:17.029 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:43:17.029 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ compare_image_snaps mirror image_46 snap 2024-09-11T06:43:17.029 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:17.029 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_46 2024-09-11T06:43:17.029 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:43:17.029 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local ret=0 2024-09-11T06:43:17.029 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local rmt_export=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_46.export 2024-09-11T06:43:17.029 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loc_export=/tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_46.export 2024-09-11T06:43:17.030 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_46.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_46.export 2024-09-11T06:43:17.030 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror export image_46@snap /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_46.export 2024-09-11T06:43:17.187 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T06:43:17.196 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 -p mirror export image_46@snap /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_46.export 2024-09-11T06:43:17.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T06:43:17.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmp /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_46.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_46.export 2024-09-11T06:43:17.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_46.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_46.export 2024-09-11T06:43:17.538 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:43:17.539 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:43:17.539 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_47 2024-09-11T06:43:17.539 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_snap cluster2 mirror image_47 snap 2024-09-11T06:43:17.539 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:43:17.539 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:17.539 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_47 2024-09-11T06:43:17.539 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:43:17.539 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror snap create image_47@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T06:43:17.757 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_started cluster1 mirror image_47 2024-09-11T06:43:17.758 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:43:17.758 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:17.758 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_47 2024-09-11T06:43:17.758 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_state cluster1 mirror image_47 started 2024-09-11T06:43:17.758 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:43:17.758 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:17.758 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_47 2024-09-11T06:43:17.758 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=started 2024-09-11T06:43:17.758 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:43:17.758 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:43:17.758 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:43:18.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_replay_state cluster1 mirror image_47 started 2024-09-11T06:43:18.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:43:18.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:18.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_47 2024-09-11T06:43:18.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=started 2024-09-11T06:43:18.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local status_result 2024-09-11T06:43:18.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=stopped 2024-09-11T06:43:18.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_47 2024-09-11T06:43:18.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -i state 2024-09-11T06:43:18.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_instance=cluster1 2024-09-11T06:43:18.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:43:18.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:43:18.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance=cluster1 2024-09-11T06:43:18.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local loop_instance 2024-09-11T06:43:18.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:43:18.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sleep 0 2024-09-11T06:43:18.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:43:18.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ seq 0 1 2024-09-11T06:43:18.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:43:18.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_47 2024-09-11T06:43:18.768 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1:0 2024-09-11T06:43:18.768 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:43:18.768 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:43:18.768 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:43:18.768 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:43:18.768 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=cluster 2024-09-11T06:43:18.769 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:43:18.769 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:43:18.769 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:43:18.769 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:43:18.769 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval cluster=cluster1 2024-09-11T06:43:18.769 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:43:18.769 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:43:18.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:43:18.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:43:18.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local local_cluster=cluster1:0 2024-09-11T06:43:18.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster=cluster1 2024-09-11T06:43:18.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance 2024-09-11T06:43:18.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:43:18.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local val=cluster1:0 2024-09-11T06:43:18.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T06:43:18.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance_var_name=instance 2024-09-11T06:43:18.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:43:18.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:43:18.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T06:43:18.772 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval local_cluster=cluster1 2024-09-11T06:43:18.772 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ local_cluster=cluster1 2024-09-11T06:43:18.772 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval instance=0 2024-09-11T06:43:18.772 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ instance=0 2024-09-11T06:43:18.772 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:43:18.788 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:43:18.788 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:43:18.788 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:43:18.788 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_47 2024-09-11T06:43:18.928 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ return 0 2024-09-11T06:43:18.929 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T06:43:18.929 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ echo ' "state": "Replaying"' 2024-09-11T06:43:18.929 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ grep -i Replaying 2024-09-11T06:43:18.930 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: "state": "Replaying" 2024-09-11T06:43:18.932 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_state=started 2024-09-11T06:43:18.932 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test started = started 2024-09-11T06:43:18.932 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:43:18.932 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_47 2024-09-11T06:43:18.932 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:43:18.932 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:43:18.932 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:18.932 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_47 2024-09-11T06:43:18.932 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' journal = journal ']' 2024-09-11T06:43:18.932 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_47 2024-09-11T06:43:18.932 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:43:18.932 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:43:18.933 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:18.933 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_47 2024-09-11T06:43:18.933 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-11T06:43:18.933 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-11T06:43:18.933 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:43:18.933 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:43:18.933 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:43:19.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror image_47 2024-09-11T06:43:19.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:43:19.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:19.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_47 2024-09-11T06:43:19.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:43:19.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n image_47 ']' 2024-09-11T06:43:19.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/image_47' 2024-09-11T06:43:19.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_47 2024-09-11T06:43:19.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:43:19.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:43:19.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:43:19.134 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:43:19.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:43:19.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:43:19.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:43:19.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:43:19.135 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:43:19.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:43:19.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_47 2024-09-11T06:43:19.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:43:19.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:43:19.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:43:19.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:43:19.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:43:19.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:43:19.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:43:19.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:43:19.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:43:19.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:43:19.139 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:43:19.139 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:43:19.139 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:43:19.139 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:43:19.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:43:19.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:43:19.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:43:19.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:43:19.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:43:19.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:43:19.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:43:19.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:43:19.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:43:19.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:43:19.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:43:19.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:43:19.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:43:19.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:43:19.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:43:19.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:43:19.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:43:19.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:43:19.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:43:19.166 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_47 2024-09-11T06:43:19.307 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:43:19.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror image_47 2024-09-11T06:43:19.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:43:19.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:43:19.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_47 2024-09-11T06:43:19.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror image_47 '' 2024-09-11T06:43:19.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:43:19.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:43:19.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_47 2024-09-11T06:43:19.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:43:19.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-image_47.status 2024-09-11T06:43:19.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-image_47.status 2024-09-11T06:43:19.309 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:43:19.311 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_47.status 2024-09-11T06:43:19.312 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_47 2024-09-11T06:43:19.312 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_47.status 2024-09-11T06:43:19.363 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 1 2024-09-11T06:43:19.364 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 1 2024-09-11T06:43:19.364 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:43:19.364 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:43:19.364 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:43:19.367 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_47.status 2024-09-11T06:43:19.369 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-09-11T06:43:19.369 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror image_47 2024-09-11T06:43:19.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:43:19.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:43:19.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_47 2024-09-11T06:43:19.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror image_47 '..*' 2024-09-11T06:43:19.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:43:19.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:43:19.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_47 2024-09-11T06:43:19.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:43:19.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-image_47.status 2024-09-11T06:43:19.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-image_47.status 2024-09-11T06:43:19.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:43:19.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_47.status 2024-09-11T06:43:19.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_47 2024-09-11T06:43:19.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_47.status 2024-09-11T06:43:19.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 1 2024-09-11T06:43:19.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 1 2024-09-11T06:43:19.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:43:19.418 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:43:19.418 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:43:19.421 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_47.status 2024-09-11T06:43:19.422 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-09-11T06:43:19.423 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=3, entry_tid=1' 2024-09-11T06:43:19.423 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:43:19.423 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_snap_present cluster1 mirror image_47 snap 2024-09-11T06:43:19.423 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:43:19.423 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:19.423 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_47 2024-09-11T06:43:19.423 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:43:19.423 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:43:19.423 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T06:43:19.423 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:43:20.423 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 info mirror/image_47@snap 2024-09-11T06:43:20.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:rbd image 'image_47': 2024-09-11T06:43:20.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: size 128 MiB in 32 objects 2024-09-11T06:43:20.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: order 22 (4 MiB objects) 2024-09-11T06:43:20.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: snapshot_count: 1 2024-09-11T06:43:20.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: id: 10a6639a6c24 2024-09-11T06:43:20.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: block_name_prefix: rbd_data.10a6639a6c24 2024-09-11T06:43:20.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: format: 2 2024-09-11T06:43:20.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: features: layering, exclusive-lock, journaling 2024-09-11T06:43:20.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: op_features: 2024-09-11T06:43:20.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: flags: 2024-09-11T06:43:20.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: create_timestamp: Wed Sep 11 06:38:07 2024 2024-09-11T06:43:20.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: access_timestamp: Wed Sep 11 06:38:07 2024 2024-09-11T06:43:20.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: modify_timestamp: Wed Sep 11 06:39:14 2024 2024-09-11T06:43:20.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: protected: False 2024-09-11T06:43:20.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: journal: 10a6639a6c24 2024-09-11T06:43:20.498 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring state: enabled 2024-09-11T06:43:20.498 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring mode: journal 2024-09-11T06:43:20.498 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring global id: 7e6c0e78-5a4c-436d-8876-7a94bad10115 2024-09-11T06:43:20.498 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring primary: false 2024-09-11T06:43:20.501 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:43:20.501 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ compare_image_snaps mirror image_47 snap 2024-09-11T06:43:20.502 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:20.502 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_47 2024-09-11T06:43:20.502 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:43:20.502 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local ret=0 2024-09-11T06:43:20.502 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local rmt_export=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_47.export 2024-09-11T06:43:20.502 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loc_export=/tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_47.export 2024-09-11T06:43:20.502 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_47.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_47.export 2024-09-11T06:43:20.502 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror export image_47@snap /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_47.export 2024-09-11T06:43:20.696 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T06:43:20.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 -p mirror export image_47@snap /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_47.export 2024-09-11T06:43:21.067 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T06:43:21.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmp /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_47.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_47.export 2024-09-11T06:43:21.159 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_47.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_47.export 2024-09-11T06:43:21.202 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:43:21.202 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:43:21.202 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_48 2024-09-11T06:43:21.203 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_snap cluster2 mirror image_48 snap 2024-09-11T06:43:21.203 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:43:21.203 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:21.203 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_48 2024-09-11T06:43:21.203 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:43:21.203 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror snap create image_48@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T06:43:21.473 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_started cluster1 mirror image_48 2024-09-11T06:43:21.473 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:43:21.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:21.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_48 2024-09-11T06:43:21.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_state cluster1 mirror image_48 started 2024-09-11T06:43:21.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:43:21.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:21.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_48 2024-09-11T06:43:21.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=started 2024-09-11T06:43:21.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:43:21.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:43:21.474 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:43:22.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_replay_state cluster1 mirror image_48 started 2024-09-11T06:43:22.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:43:22.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:22.475 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_48 2024-09-11T06:43:22.476 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=started 2024-09-11T06:43:22.476 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local status_result 2024-09-11T06:43:22.476 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=stopped 2024-09-11T06:43:22.477 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_48 2024-09-11T06:43:22.477 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_instance=cluster1 2024-09-11T06:43:22.477 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -i state 2024-09-11T06:43:22.477 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:43:22.477 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:43:22.477 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance=cluster1 2024-09-11T06:43:22.477 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local loop_instance 2024-09-11T06:43:22.477 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:43:22.478 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sleep 0 2024-09-11T06:43:22.478 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:43:22.478 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ seq 0 1 2024-09-11T06:43:22.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:43:22.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_48 2024-09-11T06:43:22.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1:0 2024-09-11T06:43:22.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:43:22.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:43:22.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:43:22.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:43:22.481 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=cluster 2024-09-11T06:43:22.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:43:22.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:43:22.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:43:22.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:43:22.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval cluster=cluster1 2024-09-11T06:43:22.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:43:22.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:43:22.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:43:22.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:43:22.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local local_cluster=cluster1:0 2024-09-11T06:43:22.484 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster=cluster1 2024-09-11T06:43:22.484 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance 2024-09-11T06:43:22.484 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:43:22.484 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local val=cluster1:0 2024-09-11T06:43:22.484 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T06:43:22.484 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance_var_name=instance 2024-09-11T06:43:22.484 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:43:22.484 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:43:22.484 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T06:43:22.485 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval local_cluster=cluster1 2024-09-11T06:43:22.485 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ local_cluster=cluster1 2024-09-11T06:43:22.485 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval instance=0 2024-09-11T06:43:22.485 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ instance=0 2024-09-11T06:43:22.485 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:43:22.507 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:43:22.507 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:43:22.507 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:43:22.507 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_48 2024-09-11T06:43:22.637 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:Invalid command: unused arguments: ['mirror/image_48'] 2024-09-11T06:43:22.637 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T06:43:22.637 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:admin_socket: invalid command 2024-09-11T06:43:22.640 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:43:22.640 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_48 2024-09-11T06:43:22.640 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1:1 2024-09-11T06:43:22.640 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:43:22.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:43:22.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-11T06:43:22.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:1 2024-09-11T06:43:22.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=cluster 2024-09-11T06:43:22.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:43:22.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:43:22.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=1 2024-09-11T06:43:22.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T06:43:22.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval cluster=cluster1 2024-09-11T06:43:22.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:43:22.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=1 2024-09-11T06:43:22.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=1 2024-09-11T06:43:22.641 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-11T06:43:22.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local local_cluster=cluster1:1 2024-09-11T06:43:22.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster=cluster1 2024-09-11T06:43:22.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance 2024-09-11T06:43:22.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T06:43:22.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local val=cluster1:1 2024-09-11T06:43:22.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T06:43:22.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance_var_name=instance 2024-09-11T06:43:22.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:43:22.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=1 2024-09-11T06:43:22.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-11T06:43:22.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval local_cluster=cluster1 2024-09-11T06:43:22.642 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ local_cluster=cluster1 2024-09-11T06:43:22.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval instance=1 2024-09-11T06:43:22.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ instance=1 2024-09-11T06:43:22.643 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T06:43:22.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:43:22.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:43:22.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:43:22.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_48 2024-09-11T06:43:22.800 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ return 0 2024-09-11T06:43:22.801 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T06:43:22.801 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ echo ' "state": "Replaying"' 2024-09-11T06:43:22.801 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ grep -i Replaying 2024-09-11T06:43:22.803 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: "state": "Replaying" 2024-09-11T06:43:22.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_state=started 2024-09-11T06:43:22.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test started = started 2024-09-11T06:43:22.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:43:22.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_48 2024-09-11T06:43:22.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:43:22.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:43:22.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:22.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_48 2024-09-11T06:43:22.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' journal = journal ']' 2024-09-11T06:43:22.806 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_48 2024-09-11T06:43:22.806 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:43:22.806 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:43:22.806 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:22.806 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_48 2024-09-11T06:43:22.806 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-11T06:43:22.806 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-11T06:43:22.806 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:43:22.806 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:43:22.806 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:43:23.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror image_48 2024-09-11T06:43:23.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:43:23.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:23.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_48 2024-09-11T06:43:23.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:43:23.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n image_48 ']' 2024-09-11T06:43:23.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/image_48' 2024-09-11T06:43:23.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_48 2024-09-11T06:43:23.007 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:43:23.007 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:43:23.007 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:43:23.007 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:43:23.007 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:43:23.007 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:43:23.007 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:43:23.007 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:43:23.008 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:43:23.011 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:43:23.012 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_48 2024-09-11T06:43:23.012 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:43:23.012 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:43:23.012 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:43:23.012 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:43:23.012 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:43:23.012 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:43:23.012 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:43:23.012 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:43:23.012 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:43:23.013 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:43:23.013 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:43:23.013 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:43:23.013 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:43:23.013 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:43:23.013 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:43:23.013 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:43:23.013 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:43:23.013 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:43:23.013 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:43:23.013 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:43:23.013 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:43:23.013 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:43:23.014 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:43:23.014 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:43:23.014 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:43:23.014 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:43:23.014 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:43:23.014 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:43:23.014 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:43:23.014 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:43:23.037 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:43:23.038 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:43:23.038 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:43:23.038 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_48 2024-09-11T06:43:23.159 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:Invalid command: unused arguments: ['mirror/image_48'] 2024-09-11T06:43:23.159 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd mirror flush : flush rbd mirror 2024-09-11T06:43:23.159 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:admin_socket: invalid command 2024-09-11T06:43:23.163 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:43:23.163 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_48 2024-09-11T06:43:23.163 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:1 2024-09-11T06:43:23.163 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:43:23.164 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:43:23.164 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-11T06:43:23.164 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:1 2024-09-11T06:43:23.164 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:43:23.164 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:43:23.164 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:43:23.164 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=1 2024-09-11T06:43:23.164 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-11T06:43:23.164 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:43:23.164 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:43:23.164 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=1 2024-09-11T06:43:23.164 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=1 2024-09-11T06:43:23.164 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-11T06:43:23.164 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:1 2024-09-11T06:43:23.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:43:23.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:43:23.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T06:43:23.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:1 2024-09-11T06:43:23.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:43:23.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:43:23.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:43:23.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=1 2024-09-11T06:43:23.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T06:43:23.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:43:23.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:43:23.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=1 2024-09-11T06:43:23.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=1 2024-09-11T06:43:23.166 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T06:43:23.187 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:43:23.187 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:43:23.187 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:43:23.187 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_48 2024-09-11T06:43:23.319 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:43:23.321 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror image_48 2024-09-11T06:43:23.321 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:43:23.321 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:43:23.321 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_48 2024-09-11T06:43:23.321 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror image_48 '' 2024-09-11T06:43:23.321 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:43:23.321 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:43:23.321 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_48 2024-09-11T06:43:23.321 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:43:23.322 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-image_48.status 2024-09-11T06:43:23.322 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-image_48.status 2024-09-11T06:43:23.322 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:43:23.323 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_48.status 2024-09-11T06:43:23.323 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_48 2024-09-11T06:43:23.323 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_48.status 2024-09-11T06:43:23.375 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 1 2024-09-11T06:43:23.375 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 1 2024-09-11T06:43:23.375 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:43:23.375 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:43:23.376 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:43:23.378 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_48.status 2024-09-11T06:43:23.380 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-09-11T06:43:23.381 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror image_48 2024-09-11T06:43:23.381 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:43:23.381 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:43:23.381 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_48 2024-09-11T06:43:23.381 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror image_48 '..*' 2024-09-11T06:43:23.381 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:43:23.381 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:43:23.381 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_48 2024-09-11T06:43:23.382 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:43:23.382 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-image_48.status 2024-09-11T06:43:23.382 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-image_48.status 2024-09-11T06:43:23.382 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:43:23.382 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_48.status 2024-09-11T06:43:23.382 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_48 2024-09-11T06:43:23.383 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_48.status 2024-09-11T06:43:23.428 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 1 2024-09-11T06:43:23.428 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 1 2024-09-11T06:43:23.428 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:43:23.428 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:43:23.428 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:43:23.432 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_48.status 2024-09-11T06:43:23.434 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-09-11T06:43:23.434 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=3, entry_tid=1' 2024-09-11T06:43:23.434 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:43:23.434 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_snap_present cluster1 mirror image_48 snap 2024-09-11T06:43:23.434 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:43:23.434 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:23.434 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_48 2024-09-11T06:43:23.434 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:43:23.434 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:43:23.434 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T06:43:23.434 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:43:24.434 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 info mirror/image_48@snap 2024-09-11T06:43:24.502 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:rbd image 'image_48': 2024-09-11T06:43:24.502 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: size 128 MiB in 32 objects 2024-09-11T06:43:24.503 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: order 22 (4 MiB objects) 2024-09-11T06:43:24.503 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: snapshot_count: 1 2024-09-11T06:43:24.503 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: id: 10a9e8794998 2024-09-11T06:43:24.503 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: block_name_prefix: rbd_data.10a9e8794998 2024-09-11T06:43:24.503 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: format: 2 2024-09-11T06:43:24.503 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: features: layering, exclusive-lock, journaling 2024-09-11T06:43:24.503 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: op_features: 2024-09-11T06:43:24.503 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: flags: 2024-09-11T06:43:24.503 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: create_timestamp: Wed Sep 11 06:38:15 2024 2024-09-11T06:43:24.503 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: access_timestamp: Wed Sep 11 06:38:15 2024 2024-09-11T06:43:24.503 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: modify_timestamp: Wed Sep 11 06:39:16 2024 2024-09-11T06:43:24.503 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: protected: False 2024-09-11T06:43:24.503 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: journal: 10a9e8794998 2024-09-11T06:43:24.503 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring state: enabled 2024-09-11T06:43:24.504 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring mode: journal 2024-09-11T06:43:24.504 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring global id: 19132f0f-4159-4858-8e68-08476f9def6b 2024-09-11T06:43:24.504 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring primary: false 2024-09-11T06:43:24.507 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:43:24.507 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ compare_image_snaps mirror image_48 snap 2024-09-11T06:43:24.507 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:24.507 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_48 2024-09-11T06:43:24.507 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:43:24.507 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local ret=0 2024-09-11T06:43:24.508 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local rmt_export=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_48.export 2024-09-11T06:43:24.508 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loc_export=/tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_48.export 2024-09-11T06:43:24.508 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_48.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_48.export 2024-09-11T06:43:24.508 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror export image_48@snap /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_48.export 2024-09-11T06:43:24.712 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T06:43:24.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 -p mirror export image_48@snap /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_48.export 2024-09-11T06:43:25.032 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T06:43:25.041 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmp /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_48.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_48.export 2024-09-11T06:43:25.116 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_48.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_48.export 2024-09-11T06:43:25.159 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:43:25.159 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:43:25.159 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_49 2024-09-11T06:43:25.159 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_snap cluster2 mirror image_49 snap 2024-09-11T06:43:25.159 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:43:25.159 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:25.159 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_49 2024-09-11T06:43:25.159 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:43:25.159 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror snap create image_49@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T06:43:25.655 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_started cluster1 mirror image_49 2024-09-11T06:43:25.655 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:43:25.655 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:25.655 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_49 2024-09-11T06:43:25.656 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_state cluster1 mirror image_49 started 2024-09-11T06:43:25.656 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:43:25.656 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:25.656 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_49 2024-09-11T06:43:25.656 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=started 2024-09-11T06:43:25.656 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:43:25.656 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:43:25.656 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:43:26.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_replay_state cluster1 mirror image_49 started 2024-09-11T06:43:26.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:43:26.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:26.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_49 2024-09-11T06:43:26.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=started 2024-09-11T06:43:26.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local status_result 2024-09-11T06:43:26.658 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=stopped 2024-09-11T06:43:26.659 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_49 2024-09-11T06:43:26.659 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -i state 2024-09-11T06:43:26.659 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_instance=cluster1 2024-09-11T06:43:26.659 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:43:26.659 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:43:26.659 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance=cluster1 2024-09-11T06:43:26.659 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local loop_instance 2024-09-11T06:43:26.659 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:43:26.659 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sleep 0 2024-09-11T06:43:26.660 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:43:26.660 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ seq 0 1 2024-09-11T06:43:26.663 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:43:26.663 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_49 2024-09-11T06:43:26.663 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1:0 2024-09-11T06:43:26.663 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:43:26.664 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:43:26.664 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:43:26.664 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:43:26.664 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=cluster 2024-09-11T06:43:26.664 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:43:26.664 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:43:26.664 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:43:26.664 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:43:26.664 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval cluster=cluster1 2024-09-11T06:43:26.664 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:43:26.664 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:43:26.664 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:43:26.665 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:43:26.665 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local local_cluster=cluster1:0 2024-09-11T06:43:26.665 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster=cluster1 2024-09-11T06:43:26.665 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance 2024-09-11T06:43:26.665 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:43:26.665 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local val=cluster1:0 2024-09-11T06:43:26.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T06:43:26.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance_var_name=instance 2024-09-11T06:43:26.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:43:26.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:43:26.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T06:43:26.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval local_cluster=cluster1 2024-09-11T06:43:26.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ local_cluster=cluster1 2024-09-11T06:43:26.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval instance=0 2024-09-11T06:43:26.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ instance=0 2024-09-11T06:43:26.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:43:26.689 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:43:26.689 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:43:26.689 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:43:26.689 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_49 2024-09-11T06:43:26.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ return 0 2024-09-11T06:43:26.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T06:43:26.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ echo ' "state": "Replaying"' 2024-09-11T06:43:26.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ grep -i Replaying 2024-09-11T06:43:26.835 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: "state": "Replaying" 2024-09-11T06:43:26.836 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_state=started 2024-09-11T06:43:26.836 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test started = started 2024-09-11T06:43:26.836 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:43:26.836 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_49 2024-09-11T06:43:26.836 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:43:26.836 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:43:26.836 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:26.836 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_49 2024-09-11T06:43:26.836 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' journal = journal ']' 2024-09-11T06:43:26.836 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_49 2024-09-11T06:43:26.836 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:43:26.836 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:43:26.837 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:26.837 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_49 2024-09-11T06:43:26.837 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-11T06:43:26.837 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-11T06:43:26.837 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:43:26.837 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:43:26.837 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:43:27.037 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror image_49 2024-09-11T06:43:27.037 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:43:27.037 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:27.038 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_49 2024-09-11T06:43:27.038 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:43:27.038 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n image_49 ']' 2024-09-11T06:43:27.038 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/image_49' 2024-09-11T06:43:27.038 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_49 2024-09-11T06:43:27.038 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:43:27.038 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:43:27.038 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:43:27.038 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:43:27.038 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:43:27.038 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:43:27.038 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:43:27.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:43:27.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:43:27.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:43:27.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_49 2024-09-11T06:43:27.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:43:27.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:43:27.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:43:27.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:43:27.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:43:27.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:43:27.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:43:27.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:43:27.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:43:27.043 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:43:27.043 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:43:27.043 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:43:27.043 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:43:27.043 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:43:27.043 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:43:27.043 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:43:27.043 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:43:27.043 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:43:27.043 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:43:27.043 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:43:27.043 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:43:27.043 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:43:27.044 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:43:27.044 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:43:27.044 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:43:27.044 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:43:27.044 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:43:27.044 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:43:27.044 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:43:27.044 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:43:27.068 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:43:27.068 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:43:27.068 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:43:27.069 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_49 2024-09-11T06:43:27.209 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:43:27.210 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror image_49 2024-09-11T06:43:27.210 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:43:27.210 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:43:27.210 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_49 2024-09-11T06:43:27.210 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror image_49 '' 2024-09-11T06:43:27.211 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:43:27.211 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:43:27.211 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_49 2024-09-11T06:43:27.211 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:43:27.211 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-image_49.status 2024-09-11T06:43:27.211 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-image_49.status 2024-09-11T06:43:27.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:43:27.213 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_49.status 2024-09-11T06:43:27.214 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_49.status 2024-09-11T06:43:27.214 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_49 2024-09-11T06:43:27.264 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 1 2024-09-11T06:43:27.264 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 1 2024-09-11T06:43:27.265 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:43:27.265 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:43:27.265 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:43:27.268 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_49.status 2024-09-11T06:43:27.269 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-09-11T06:43:27.270 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror image_49 2024-09-11T06:43:27.270 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:43:27.270 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:43:27.271 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_49 2024-09-11T06:43:27.271 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror image_49 '..*' 2024-09-11T06:43:27.271 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:43:27.271 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:43:27.271 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_49 2024-09-11T06:43:27.271 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:43:27.271 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-image_49.status 2024-09-11T06:43:27.271 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-image_49.status 2024-09-11T06:43:27.271 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:43:27.272 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_49.status 2024-09-11T06:43:27.272 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_49 2024-09-11T06:43:27.272 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_49.status 2024-09-11T06:43:27.318 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 1 2024-09-11T06:43:27.318 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 1 2024-09-11T06:43:27.318 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:43:27.318 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:43:27.318 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=5, tag_tid=3, entry_tid=1], [object_number=4, tag_tid=3, entry_tid=0], [object_number=7, tag_tid=2, entry_tid=3], [object_number=6, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:43:27.321 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_49.status 2024-09-11T06:43:27.323 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=5, tag_tid=3, entry_tid=1' 2024-09-11T06:43:27.323 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=5, tag_tid=3, entry_tid=1' -a 'object_number=5, tag_tid=3, entry_tid=1' = 'object_number=5, tag_tid=3, entry_tid=1' 2024-09-11T06:43:27.323 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:43:27.324 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_snap_present cluster1 mirror image_49 snap 2024-09-11T06:43:27.324 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:43:27.324 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:27.324 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_49 2024-09-11T06:43:27.324 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:43:27.324 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:43:27.324 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T06:43:27.324 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:43:28.324 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 info mirror/image_49@snap 2024-09-11T06:43:28.396 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:rbd image 'image_49': 2024-09-11T06:43:28.396 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: size 128 MiB in 32 objects 2024-09-11T06:43:28.396 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: order 22 (4 MiB objects) 2024-09-11T06:43:28.396 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: snapshot_count: 1 2024-09-11T06:43:28.397 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: id: 10a63df4a184 2024-09-11T06:43:28.397 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: block_name_prefix: rbd_data.10a63df4a184 2024-09-11T06:43:28.397 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: format: 2 2024-09-11T06:43:28.397 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: features: layering, exclusive-lock, journaling 2024-09-11T06:43:28.397 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: op_features: 2024-09-11T06:43:28.397 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: flags: 2024-09-11T06:43:28.397 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: create_timestamp: Wed Sep 11 06:38:25 2024 2024-09-11T06:43:28.397 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: access_timestamp: Wed Sep 11 06:38:25 2024 2024-09-11T06:43:28.397 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: modify_timestamp: Wed Sep 11 06:39:25 2024 2024-09-11T06:43:28.397 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: protected: False 2024-09-11T06:43:28.397 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: journal: 10a63df4a184 2024-09-11T06:43:28.397 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring state: enabled 2024-09-11T06:43:28.397 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring mode: journal 2024-09-11T06:43:28.398 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring global id: 0075d621-210a-4e4b-9736-bc19fcc1224f 2024-09-11T06:43:28.398 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring primary: false 2024-09-11T06:43:28.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:43:28.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ compare_image_snaps mirror image_49 snap 2024-09-11T06:43:28.402 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:28.403 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_49 2024-09-11T06:43:28.403 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:43:28.403 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local ret=0 2024-09-11T06:43:28.403 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local rmt_export=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_49.export 2024-09-11T06:43:28.403 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loc_export=/tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_49.export 2024-09-11T06:43:28.403 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_49.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_49.export 2024-09-11T06:43:28.403 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror export image_49@snap /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_49.export 2024-09-11T06:43:28.581 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T06:43:28.589 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 -p mirror export image_49@snap /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_49.export 2024-09-11T06:43:28.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T06:43:28.899 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmp /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_49.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_49.export 2024-09-11T06:43:28.979 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_49.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_49.export 2024-09-11T06:43:29.024 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:43:29.025 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:43:29.025 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_50 2024-09-11T06:43:29.025 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_snap cluster2 mirror image_50 snap 2024-09-11T06:43:29.025 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:43:29.025 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:29.025 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_50 2024-09-11T06:43:29.025 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:43:29.025 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror snap create image_50@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T06:43:29.652 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_started cluster1 mirror image_50 2024-09-11T06:43:29.652 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:43:29.652 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:29.652 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_50 2024-09-11T06:43:29.652 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_replay_state cluster1 mirror image_50 started 2024-09-11T06:43:29.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:43:29.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:29.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_50 2024-09-11T06:43:29.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=started 2024-09-11T06:43:29.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:43:29.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:43:29.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:43:30.654 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_replay_state cluster1 mirror image_50 started 2024-09-11T06:43:30.654 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:43:30.654 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:30.655 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_50 2024-09-11T06:43:30.655 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=started 2024-09-11T06:43:30.655 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local status_result 2024-09-11T06:43:30.655 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=stopped 2024-09-11T06:43:30.656 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_50 2024-09-11T06:43:30.656 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -i state 2024-09-11T06:43:30.656 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_instance=cluster1 2024-09-11T06:43:30.656 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:43:30.656 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:43:30.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance=cluster1 2024-09-11T06:43:30.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local loop_instance 2024-09-11T06:43:30.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:43:30.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sleep 0 2024-09-11T06:43:30.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:43:30.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ seq 0 1 2024-09-11T06:43:30.661 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:43:30.661 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_50 2024-09-11T06:43:30.661 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1:0 2024-09-11T06:43:30.661 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:43:30.661 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:43:30.661 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:43:30.661 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:43:30.661 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=cluster 2024-09-11T06:43:30.661 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:43:30.661 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:43:30.661 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:43:30.662 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:43:30.662 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval cluster=cluster1 2024-09-11T06:43:30.662 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:43:30.662 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:43:30.662 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:43:30.662 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:43:30.662 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local local_cluster=cluster1:0 2024-09-11T06:43:30.662 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster=cluster1 2024-09-11T06:43:30.662 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance 2024-09-11T06:43:30.662 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:43:30.662 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local val=cluster1:0 2024-09-11T06:43:30.662 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T06:43:30.662 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance_var_name=instance 2024-09-11T06:43:30.663 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:43:30.663 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:43:30.663 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T06:43:30.663 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval local_cluster=cluster1 2024-09-11T06:43:30.663 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ local_cluster=cluster1 2024-09-11T06:43:30.663 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval instance=0 2024-09-11T06:43:30.663 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ instance=0 2024-09-11T06:43:30.663 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:43:30.685 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:43:30.686 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:43:30.687 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:43:30.688 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_50 2024-09-11T06:43:30.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:Invalid command: unused arguments: ['mirror/image_50'] 2024-09-11T06:43:30.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T06:43:30.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:admin_socket: invalid command 2024-09-11T06:43:30.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:43:30.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_50 2024-09-11T06:43:30.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1:1 2024-09-11T06:43:30.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ shift 2024-09-11T06:43:30.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:43:30.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-11T06:43:30.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:1 2024-09-11T06:43:30.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=cluster 2024-09-11T06:43:30.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:43:30.811 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:43:30.812 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=1 2024-09-11T06:43:30.812 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T06:43:30.812 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval cluster=cluster1 2024-09-11T06:43:30.812 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:43:30.812 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=1 2024-09-11T06:43:30.812 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=1 2024-09-11T06:43:30.813 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-11T06:43:30.813 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local local_cluster=cluster1:1 2024-09-11T06:43:30.813 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster=cluster1 2024-09-11T06:43:30.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance 2024-09-11T06:43:30.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T06:43:30.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local val=cluster1:1 2024-09-11T06:43:30.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T06:43:30.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance_var_name=instance 2024-09-11T06:43:30.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:43:30.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=1 2024-09-11T06:43:30.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-11T06:43:30.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval local_cluster=cluster1 2024-09-11T06:43:30.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ local_cluster=cluster1 2024-09-11T06:43:30.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval instance=1 2024-09-11T06:43:30.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ instance=1 2024-09-11T06:43:30.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T06:43:30.836 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:43:30.836 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:43:30.836 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:43:30.836 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_50 2024-09-11T06:43:30.973 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ return 0 2024-09-11T06:43:30.973 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T06:43:30.974 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ echo ' "state": "Replaying"' 2024-09-11T06:43:30.974 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ grep -i Replaying 2024-09-11T06:43:30.976 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: "state": "Replaying" 2024-09-11T06:43:30.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_state=started 2024-09-11T06:43:30.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test started = started 2024-09-11T06:43:30.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:43:30.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_50 2024-09-11T06:43:30.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:43:30.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:43:30.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:30.979 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_50 2024-09-11T06:43:30.979 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' journal = journal ']' 2024-09-11T06:43:30.979 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror image_50 2024-09-11T06:43:30.979 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local local_cluster=cluster1 2024-09-11T06:43:30.979 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:43:30.979 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:30.979 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_50 2024-09-11T06:43:30.979 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s master_pos mirror_pos last_mirror_pos 2024-09-11T06:43:30.979 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2024-09-11T06:43:30.979 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:43:30.979 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T06:43:30.979 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.2 2024-09-11T06:43:31.179 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ flush cluster1 mirror image_50 2024-09-11T06:43:31.179 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:43:31.179 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:31.179 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_50 2024-09-11T06:43:31.179 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local 'cmd=rbd mirror flush' 2024-09-11T06:43:31.179 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n image_50 ']' 2024-09-11T06:43:31.180 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmd='rbd mirror flush mirror/image_50' 2024-09-11T06:43:31.180 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/image_50 2024-09-11T06:43:31.180 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_instance=cluster1 2024-09-11T06:43:31.180 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:43:31.180 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:43:31.180 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance=cluster1 2024-09-11T06:43:31.180 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loop_instance 2024-09-11T06:43:31.180 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T06:43:31.181 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:43:31.183 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 '!=' cluster1 ']' 2024-09-11T06:43:31.184 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:43:31.184 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:43:31.184 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/image_50 2024-09-11T06:43:31.184 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:43:31.184 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:43:31.184 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:43:31.184 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:43:31.184 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:0 2024-09-11T06:43:31.184 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:43:31.184 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:43:31.184 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:43:31.184 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:43:31.184 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T06:43:31.184 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:43:31.185 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:43:31.185 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:43:31.185 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:43:31.185 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:43:31.185 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:43:31.185 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:43:31.185 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:43:31.185 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:43:31.185 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:43:31.185 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:43:31.185 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:43:31.185 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:43:31.185 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:43:31.186 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:43:31.186 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:43:31.186 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:43:31.186 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:43:31.186 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:43:31.186 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:43:31.210 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:43:31.210 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:43:31.211 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:43:31.211 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/image_50 2024-09-11T06:43:31.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:Invalid command: unused arguments: ['mirror/image_50'] 2024-09-11T06:43:31.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd mirror flush : flush rbd mirror 2024-09-11T06:43:31.336 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:admin_socket: invalid command 2024-09-11T06:43:31.339 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:43:31.339 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/image_50 2024-09-11T06:43:31.339 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:1 2024-09-11T06:43:31.339 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:43:31.339 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:43:31.339 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-11T06:43:31.339 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1:1 2024-09-11T06:43:31.339 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:43:31.339 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:43:31.339 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:43:31.339 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=1 2024-09-11T06:43:31.339 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-11T06:43:31.340 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:43:31.340 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:43:31.340 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=1 2024-09-11T06:43:31.340 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=1 2024-09-11T06:43:31.341 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-11T06:43:31.341 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:1 2024-09-11T06:43:31.341 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:43:31.341 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:43:31.341 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T06:43:31.341 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:1 2024-09-11T06:43:31.341 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:43:31.341 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:43:31.341 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:43:31.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=1 2024-09-11T06:43:31.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T06:43:31.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:43:31.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:43:31.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=1 2024-09-11T06:43:31.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=1 2024-09-11T06:43:31.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T06:43:31.359 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:43:31.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:43:31.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:43:31.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/image_50 2024-09-11T06:43:31.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:43:31.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_master_journal_position cluster2 mirror image_50 2024-09-11T06:43:31.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:43:31.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:43:31.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_50 2024-09-11T06:43:31.505 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror image_50 '' 2024-09-11T06:43:31.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:43:31.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:43:31.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_50 2024-09-11T06:43:31.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local id_regexp= 2024-09-11T06:43:31.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-image_50.status 2024-09-11T06:43:31.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:43:31.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-image_50.status 2024-09-11T06:43:31.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_50.status 2024-09-11T06:43:31.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_50 2024-09-11T06:43:31.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_50.status 2024-09-11T06:43:31.550 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 2 2024-09-11T06:43:31.550 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 2 2024-09-11T06:43:31.550 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:43:31.550 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=3, entry_tid=1], [object_number=8, tag_tid=3, entry_tid=0], [object_number=11, tag_tid=2, entry_tid=3], [object_number=10, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:43:31.550 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=9, tag_tid=3, entry_tid=1], [object_number=8, tag_tid=3, entry_tid=0], [object_number=11, tag_tid=2, entry_tid=3], [object_number=10, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:43:31.553 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_50.status 2024-09-11T06:43:31.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ master_pos='object_number=9, tag_tid=3, entry_tid=1' 2024-09-11T06:43:31.555 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_mirror_journal_position cluster2 mirror image_50 2024-09-11T06:43:31.556 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:43:31.556 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:43:31.556 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_50 2024-09-11T06:43:31.556 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_journal_position cluster2 mirror image_50 '..*' 2024-09-11T06:43:31.556 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:43:31.556 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:43:31.556 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_50 2024-09-11T06:43:31.556 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local 'id_regexp=..*' 2024-09-11T06:43:31.556 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ mkfname cluster2-mirror-image_50.status 2024-09-11T06:43:31.556 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo cluster2-mirror-image_50.status 2024-09-11T06:43:31.556 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ sed -e 's|[/ ]|_|g' 2024-09-11T06:43:31.557 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local status_log=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_50.status 2024-09-11T06:43:31.557 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster2 journal status --image mirror/image_50 2024-09-11T06:43:31.557 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ tee /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_50.status 2024-09-11T06:43:31.604 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:minimum_set: 2 2024-09-11T06:43:31.604 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:active_set: 2 2024-09-11T06:43:31.604 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:registered clients: 2024-09-11T06:43:31.604 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=, commit_position=[positions=[[object_number=9, tag_tid=3, entry_tid=1], [object_number=8, tag_tid=3, entry_tid=0], [object_number=11, tag_tid=2, entry_tid=3], [object_number=10, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:43:31.605 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: [id=3f3c044c-6c0b-4a55-ac91-32704217502c, commit_position=[positions=[[object_number=9, tag_tid=3, entry_tid=1], [object_number=8, tag_tid=3, entry_tid=0], [object_number=11, tag_tid=2, entry_tid=3], [object_number=10, tag_tid=2, entry_tid=2]]], state=connected] 2024-09-11T06:43:31.608 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -nEe 's/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p' /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_50.status 2024-09-11T06:43:31.610 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ mirror_pos='object_number=9, tag_tid=3, entry_tid=1' 2024-09-11T06:43:31.610 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 'object_number=9, tag_tid=3, entry_tid=1' -a 'object_number=9, tag_tid=3, entry_tid=1' = 'object_number=9, tag_tid=3, entry_tid=1' 2024-09-11T06:43:31.610 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:43:31.610 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_snap_present cluster1 mirror image_50 snap 2024-09-11T06:43:31.610 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:43:31.610 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:31.610 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_50 2024-09-11T06:43:31.610 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:43:31.610 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:43:31.610 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T06:43:31.610 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:43:32.611 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 info mirror/image_50@snap 2024-09-11T06:43:32.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:rbd image 'image_50': 2024-09-11T06:43:32.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: size 128 MiB in 32 objects 2024-09-11T06:43:32.673 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: order 22 (4 MiB objects) 2024-09-11T06:43:32.674 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: snapshot_count: 1 2024-09-11T06:43:32.674 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: id: 10a9ed304620 2024-09-11T06:43:32.674 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: block_name_prefix: rbd_data.10a9ed304620 2024-09-11T06:43:32.674 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: format: 2 2024-09-11T06:43:32.674 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: features: layering, exclusive-lock, journaling 2024-09-11T06:43:32.674 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: op_features: 2024-09-11T06:43:32.674 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: flags: 2024-09-11T06:43:32.674 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: create_timestamp: Wed Sep 11 06:38:34 2024 2024-09-11T06:43:32.674 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: access_timestamp: Wed Sep 11 06:38:34 2024 2024-09-11T06:43:32.674 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: modify_timestamp: Wed Sep 11 06:39:35 2024 2024-09-11T06:43:32.674 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: protected: False 2024-09-11T06:43:32.675 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: journal: 10a9ed304620 2024-09-11T06:43:32.675 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring state: enabled 2024-09-11T06:43:32.675 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring mode: journal 2024-09-11T06:43:32.675 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring global id: a6ef6301-0d5a-4507-bfa5-32ba0465e618 2024-09-11T06:43:32.675 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring primary: false 2024-09-11T06:43:32.678 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:43:32.678 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ compare_image_snaps mirror image_50 snap 2024-09-11T06:43:32.678 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:32.678 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_50 2024-09-11T06:43:32.678 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap_name=snap 2024-09-11T06:43:32.678 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local ret=0 2024-09-11T06:43:32.678 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local rmt_export=/tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_50.export 2024-09-11T06:43:32.678 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local loc_export=/tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_50.export 2024-09-11T06:43:32.678 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_50.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_50.export 2024-09-11T06:43:32.679 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 -p mirror export image_50@snap /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_50.export 2024-09-11T06:43:32.868 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T06:43:32.876 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster1 -p mirror export image_50@snap /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_50.export 2024-09-11T06:43:33.284 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T06:43:33.297 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cmp /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_50.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_50.export 2024-09-11T06:43:33.375 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /tmp/tmp.Dxwd1ESqKa/cluster2-mirror-image_50.export /tmp/tmp.Dxwd1ESqKa/cluster1-mirror-image_50.export 2024-09-11T06:43:33.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:43:33.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ testlog 'TEST: delete many images' 2024-09-11T06:43:33.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ tee -a /tmp/tmp.Dxwd1ESqKa/rbd-mirror.test.log 2024-09-11T06:43:33.418 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ date '+%F %T' 2024-09-11T06:43:33.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ echo 2024-09-11 06:43:33 TEST: delete many images 2024-09-11T06:43:33.420 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:2024-09-11 06:43:33 TEST: delete many images 2024-09-11T06:43:33.420 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 1 50 2024-09-11T06:43:33.421 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:43:33.422 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_1 2024-09-11T06:43:33.422 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_snapshot cluster2 mirror image_1 snap 2024-09-11T06:43:33.422 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:43:33.422 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:33.422 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_1 2024-09-11T06:43:33.422 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap=snap 2024-09-11T06:43:33.422 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 snap rm mirror/image_1@snap 2024-09-11T06:43:33.672 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing snap: 100% complete...done. 2024-09-11T06:43:33.690 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image_retry cluster2 mirror image_1 2024-09-11T06:43:33.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:43:33.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:33.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_1 2024-09-11T06:43:33.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T06:43:33.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:43:33.691 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image cluster2 mirror image_1 2024-09-11T06:43:33.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:43:33.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:33.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_1 2024-09-11T06:43:33.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_1 2024-09-11T06:43:33.739 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 rm mirror/image_1 2024-09-11T06:43:33.949 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T06:43:33.953 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:43:33.953 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:43:33.953 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_2 2024-09-11T06:43:33.953 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_snapshot cluster2 mirror image_2 snap 2024-09-11T06:43:33.953 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:43:33.953 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:33.953 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_2 2024-09-11T06:43:33.954 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap=snap 2024-09-11T06:43:33.954 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 snap rm mirror/image_2@snap 2024-09-11T06:43:34.702 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing snap: 100% complete...done. 2024-09-11T06:43:34.727 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image_retry cluster2 mirror image_2 2024-09-11T06:43:34.727 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:43:34.727 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:34.727 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_2 2024-09-11T06:43:34.727 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T06:43:34.727 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:43:34.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image cluster2 mirror image_2 2024-09-11T06:43:34.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:43:34.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:34.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_2 2024-09-11T06:43:34.728 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_2 2024-09-11T06:43:34.780 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 rm mirror/image_2 2024-09-11T06:43:35.056 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T06:43:35.060 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:43:35.060 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:43:35.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_3 2024-09-11T06:43:35.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_snapshot cluster2 mirror image_3 snap 2024-09-11T06:43:35.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:43:35.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:35.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_3 2024-09-11T06:43:35.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap=snap 2024-09-11T06:43:35.061 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 snap rm mirror/image_3@snap 2024-09-11T06:43:35.693 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing snap: 100% complete...done. 2024-09-11T06:43:35.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image_retry cluster2 mirror image_3 2024-09-11T06:43:35.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:43:35.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:35.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_3 2024-09-11T06:43:35.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T06:43:35.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:43:35.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image cluster2 mirror image_3 2024-09-11T06:43:35.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:43:35.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:35.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_3 2024-09-11T06:43:35.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_3 2024-09-11T06:43:35.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 rm mirror/image_3 2024-09-11T06:43:35.995 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T06:43:36.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:43:36.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:43:36.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_4 2024-09-11T06:43:36.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_snapshot cluster2 mirror image_4 snap 2024-09-11T06:43:36.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:43:36.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:36.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_4 2024-09-11T06:43:36.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap=snap 2024-09-11T06:43:36.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 snap rm mirror/image_4@snap 2024-09-11T06:43:36.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing snap: 100% complete...done. 2024-09-11T06:43:36.748 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image_retry cluster2 mirror image_4 2024-09-11T06:43:36.749 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:43:36.749 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:36.749 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_4 2024-09-11T06:43:36.749 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T06:43:36.749 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:43:36.749 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image cluster2 mirror image_4 2024-09-11T06:43:36.750 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:43:36.750 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:36.750 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_4 2024-09-11T06:43:36.750 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_4 2024-09-11T06:43:36.802 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 rm mirror/image_4 2024-09-11T06:43:37.043 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T06:43:37.047 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:43:37.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:43:37.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_5 2024-09-11T06:43:37.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_snapshot cluster2 mirror image_5 snap 2024-09-11T06:43:37.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:43:37.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:37.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_5 2024-09-11T06:43:37.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap=snap 2024-09-11T06:43:37.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 snap rm mirror/image_5@snap 2024-09-11T06:43:37.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing snap: 100% complete...done. 2024-09-11T06:43:37.737 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image_retry cluster2 mirror image_5 2024-09-11T06:43:37.737 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:43:37.737 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:37.737 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_5 2024-09-11T06:43:37.737 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T06:43:37.737 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:43:37.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image cluster2 mirror image_5 2024-09-11T06:43:37.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:43:37.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:37.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_5 2024-09-11T06:43:37.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_5 2024-09-11T06:43:37.790 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 rm mirror/image_5 2024-09-11T06:43:38.019 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T06:43:38.023 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:43:38.024 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:43:38.024 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_6 2024-09-11T06:43:38.024 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_snapshot cluster2 mirror image_6 snap 2024-09-11T06:43:38.024 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:43:38.024 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:38.024 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_6 2024-09-11T06:43:38.024 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap=snap 2024-09-11T06:43:38.024 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 snap rm mirror/image_6@snap 2024-09-11T06:43:38.090 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:43:38.088+0000 7efd718c5640 -1 reset not still connected to 0x560b14da5d40 2024-09-11T06:43:38.091 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:43:38.088+0000 7efd718c5640 -1 reset not still connected to 0x560b14e4e820 2024-09-11T06:43:38.091 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:43:38.088+0000 7efd718c5640 -1 reset not still connected to 0x560b159635f0 2024-09-11T06:43:38.091 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:43:38.088+0000 7efd718c5640 -1 reset not still connected to 0x560b15a708f0 2024-09-11T06:43:38.091 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:43:38.088+0000 7efd718c5640 -1 reset not still connected to 0x560b15a70dd0 2024-09-11T06:43:38.091 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:43:38.088+0000 7efd718c5640 -1 reset not still connected to 0x560b15a71520 2024-09-11T06:43:38.734 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing snap: 100% complete...done. 2024-09-11T06:43:38.753 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image_retry cluster2 mirror image_6 2024-09-11T06:43:38.753 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:43:38.753 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:38.753 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_6 2024-09-11T06:43:38.753 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T06:43:38.754 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:43:38.754 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image cluster2 mirror image_6 2024-09-11T06:43:38.754 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:43:38.754 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:38.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_6 2024-09-11T06:43:38.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_6 2024-09-11T06:43:38.806 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 rm mirror/image_6 2024-09-11T06:43:39.047 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T06:43:39.052 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:43:39.052 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:43:39.052 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_7 2024-09-11T06:43:39.053 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_snapshot cluster2 mirror image_7 snap 2024-09-11T06:43:39.053 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:43:39.053 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:39.053 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_7 2024-09-11T06:43:39.053 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap=snap 2024-09-11T06:43:39.053 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 snap rm mirror/image_7@snap 2024-09-11T06:43:39.736 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing snap: 100% complete...done. 2024-09-11T06:43:39.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image_retry cluster2 mirror image_7 2024-09-11T06:43:39.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:43:39.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:39.768 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_7 2024-09-11T06:43:39.768 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T06:43:39.768 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:43:39.769 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image cluster2 mirror image_7 2024-09-11T06:43:39.769 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:43:39.769 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:39.769 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_7 2024-09-11T06:43:39.769 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_7 2024-09-11T06:43:39.820 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 rm mirror/image_7 2024-09-11T06:43:39.949 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:43:39.941+0000 7fb45318f640 -1 reset not still connected to 0x561330ed0410 2024-09-11T06:43:39.949 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:43:39.945+0000 7fb45318f640 -1 reset not still connected to 0x561341f69450 2024-09-11T06:43:39.955 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:43:39.949+0000 7fb45318f640 -1 reset not still connected to 0x5613480e41a0 2024-09-11T06:43:39.955 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:43:39.949+0000 7fb45318f640 -1 reset not still connected to 0x5613488b0820 2024-09-11T06:43:39.955 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:43:39.949+0000 7fb45318f640 -1 reset not still connected to 0x5613488b0d00 2024-09-11T06:43:39.955 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:43:39.949+0000 7fb45318f640 -1 reset not still connected to 0x561354d77110 2024-09-11T06:43:39.957 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:43:39.953+0000 7fb45318f640 -1 reset not still connected to 0x561355040c30 2024-09-11T06:43:39.975 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:43:39.969+0000 7fb45318f640 -1 reset not still connected to 0x561315ba4270 2024-09-11T06:43:39.975 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:43:39.969+0000 7fb45318f640 -1 reset not still connected to 0x561315ba5520 2024-09-11T06:43:39.975 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:43:39.969+0000 7fb45318f640 -1 reset not still connected to 0x56131a1ffe10 2024-09-11T06:43:39.975 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:43:39.969+0000 7fb45318f640 -1 reset not still connected to 0x561330ed0410 2024-09-11T06:43:39.975 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:43:39.969+0000 7fb45318f640 -1 reset not still connected to 0x5613480e41a0 2024-09-11T06:43:39.975 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:43:39.969+0000 7fb45318f640 -1 reset not still connected to 0x5613488b0820 2024-09-11T06:43:39.977 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:43:39.973+0000 7fb45318f640 -1 reset not still connected to 0x561354d77110 2024-09-11T06:43:39.977 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:43:39.973+0000 7fb45318f640 -1 reset not still connected to 0x561355040c30 2024-09-11T06:43:40.295 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T06:43:40.299 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:43:40.300 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:43:40.300 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_8 2024-09-11T06:43:40.300 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_snapshot cluster2 mirror image_8 snap 2024-09-11T06:43:40.300 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:43:40.300 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:40.300 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_8 2024-09-11T06:43:40.300 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap=snap 2024-09-11T06:43:40.300 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 snap rm mirror/image_8@snap 2024-09-11T06:43:40.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing snap: 100% complete...done. 2024-09-11T06:43:40.764 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image_retry cluster2 mirror image_8 2024-09-11T06:43:40.764 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:43:40.764 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:40.764 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_8 2024-09-11T06:43:40.764 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T06:43:40.764 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:43:40.765 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image cluster2 mirror image_8 2024-09-11T06:43:40.765 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:43:40.765 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:40.765 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_8 2024-09-11T06:43:40.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_8 2024-09-11T06:43:40.817 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 rm mirror/image_8 2024-09-11T06:43:41.053 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T06:43:41.057 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:43:41.057 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:43:41.057 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_9 2024-09-11T06:43:41.058 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_snapshot cluster2 mirror image_9 snap 2024-09-11T06:43:41.058 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:43:41.058 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:41.058 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_9 2024-09-11T06:43:41.058 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap=snap 2024-09-11T06:43:41.058 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 snap rm mirror/image_9@snap 2024-09-11T06:43:41.745 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing snap: 100% complete...done. 2024-09-11T06:43:41.780 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image_retry cluster2 mirror image_9 2024-09-11T06:43:41.781 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:43:41.781 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:41.781 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_9 2024-09-11T06:43:41.781 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T06:43:41.781 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:43:41.782 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image cluster2 mirror image_9 2024-09-11T06:43:41.782 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:43:41.782 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:41.782 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_9 2024-09-11T06:43:41.782 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_9 2024-09-11T06:43:41.836 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 rm mirror/image_9 2024-09-11T06:43:42.079 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T06:43:42.084 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:43:42.084 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:43:42.084 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_10 2024-09-11T06:43:42.084 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_snapshot cluster2 mirror image_10 snap 2024-09-11T06:43:42.084 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:43:42.084 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:42.084 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_10 2024-09-11T06:43:42.084 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap=snap 2024-09-11T06:43:42.084 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 snap rm mirror/image_10@snap 2024-09-11T06:43:42.636 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing snap: 100% complete...done. 2024-09-11T06:43:42.659 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image_retry cluster2 mirror image_10 2024-09-11T06:43:42.659 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:43:42.659 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:42.659 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_10 2024-09-11T06:43:42.659 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T06:43:42.659 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:43:42.660 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image cluster2 mirror image_10 2024-09-11T06:43:42.660 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:43:42.660 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:42.660 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_10 2024-09-11T06:43:42.660 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_10 2024-09-11T06:43:42.710 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 rm mirror/image_10 2024-09-11T06:43:42.916 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T06:43:42.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:43:42.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:43:42.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_11 2024-09-11T06:43:42.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_snapshot cluster2 mirror image_11 snap 2024-09-11T06:43:42.921 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:43:42.921 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:42.921 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_11 2024-09-11T06:43:42.921 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap=snap 2024-09-11T06:43:42.921 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 snap rm mirror/image_11@snap 2024-09-11T06:43:43.768 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing snap: 100% complete...done. 2024-09-11T06:43:43.792 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image_retry cluster2 mirror image_11 2024-09-11T06:43:43.792 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:43:43.792 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:43.792 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_11 2024-09-11T06:43:43.792 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T06:43:43.792 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:43:43.793 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image cluster2 mirror image_11 2024-09-11T06:43:43.793 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:43:43.793 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:43.793 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_11 2024-09-11T06:43:43.793 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_11 2024-09-11T06:43:43.847 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 rm mirror/image_11 2024-09-11T06:43:44.096 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T06:43:44.100 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:43:44.101 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:43:44.101 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_12 2024-09-11T06:43:44.101 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_snapshot cluster2 mirror image_12 snap 2024-09-11T06:43:44.101 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:43:44.101 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:44.101 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_12 2024-09-11T06:43:44.101 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap=snap 2024-09-11T06:43:44.101 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 snap rm mirror/image_12@snap 2024-09-11T06:43:44.749 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing snap: 100% complete...done. 2024-09-11T06:43:44.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image_retry cluster2 mirror image_12 2024-09-11T06:43:44.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:43:44.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:44.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_12 2024-09-11T06:43:44.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T06:43:44.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:43:44.772 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image cluster2 mirror image_12 2024-09-11T06:43:44.772 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:43:44.772 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:44.772 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_12 2024-09-11T06:43:44.772 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_12 2024-09-11T06:43:44.822 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 rm mirror/image_12 2024-09-11T06:43:45.102 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T06:43:45.107 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:43:45.107 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:43:45.107 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_13 2024-09-11T06:43:45.107 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_snapshot cluster2 mirror image_13 snap 2024-09-11T06:43:45.107 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:43:45.107 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:45.107 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_13 2024-09-11T06:43:45.107 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap=snap 2024-09-11T06:43:45.107 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 snap rm mirror/image_13@snap 2024-09-11T06:43:45.777 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing snap: 100% complete...done. 2024-09-11T06:43:45.812 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image_retry cluster2 mirror image_13 2024-09-11T06:43:45.812 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:43:45.812 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:45.812 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_13 2024-09-11T06:43:45.812 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T06:43:45.812 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:43:45.813 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image cluster2 mirror image_13 2024-09-11T06:43:45.813 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:43:45.813 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:45.813 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_13 2024-09-11T06:43:45.813 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_13 2024-09-11T06:43:45.863 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 rm mirror/image_13 2024-09-11T06:43:46.709 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T06:43:46.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:43:46.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:43:46.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_14 2024-09-11T06:43:46.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_snapshot cluster2 mirror image_14 snap 2024-09-11T06:43:46.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:43:46.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:46.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_14 2024-09-11T06:43:46.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap=snap 2024-09-11T06:43:46.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 snap rm mirror/image_14@snap 2024-09-11T06:43:47.782 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing snap: 100% complete...done. 2024-09-11T06:43:47.799 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image_retry cluster2 mirror image_14 2024-09-11T06:43:47.800 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:43:47.800 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:47.800 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_14 2024-09-11T06:43:47.800 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T06:43:47.800 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:43:47.800 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image cluster2 mirror image_14 2024-09-11T06:43:47.801 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:43:47.801 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:47.801 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_14 2024-09-11T06:43:47.801 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_14 2024-09-11T06:43:47.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 rm mirror/image_14 2024-09-11T06:43:48.110 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T06:43:48.114 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:43:48.114 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:43:48.114 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_15 2024-09-11T06:43:48.114 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_snapshot cluster2 mirror image_15 snap 2024-09-11T06:43:48.115 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:43:48.115 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:48.115 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_15 2024-09-11T06:43:48.115 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap=snap 2024-09-11T06:43:48.115 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 snap rm mirror/image_15@snap 2024-09-11T06:43:48.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing snap: 100% complete...done. 2024-09-11T06:43:48.799 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image_retry cluster2 mirror image_15 2024-09-11T06:43:48.799 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:43:48.799 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:48.799 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_15 2024-09-11T06:43:48.799 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T06:43:48.799 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:43:48.800 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image cluster2 mirror image_15 2024-09-11T06:43:48.800 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:43:48.800 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:48.800 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_15 2024-09-11T06:43:48.800 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_15 2024-09-11T06:43:48.854 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 rm mirror/image_15 2024-09-11T06:43:49.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T06:43:49.074 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:43:49.074 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:43:49.074 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_16 2024-09-11T06:43:49.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_snapshot cluster2 mirror image_16 snap 2024-09-11T06:43:49.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:43:49.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:49.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_16 2024-09-11T06:43:49.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap=snap 2024-09-11T06:43:49.075 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 snap rm mirror/image_16@snap 2024-09-11T06:43:49.782 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing snap: 100% complete...done. 2024-09-11T06:43:49.808 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image_retry cluster2 mirror image_16 2024-09-11T06:43:49.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:43:49.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:49.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_16 2024-09-11T06:43:49.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T06:43:49.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:43:49.810 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image cluster2 mirror image_16 2024-09-11T06:43:49.810 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:43:49.810 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:49.810 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_16 2024-09-11T06:43:49.810 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_16 2024-09-11T06:43:49.875 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 rm mirror/image_16 2024-09-11T06:43:50.097 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T06:43:50.101 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:43:50.102 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:43:50.102 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_17 2024-09-11T06:43:50.102 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_snapshot cluster2 mirror image_17 snap 2024-09-11T06:43:50.102 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:43:50.102 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:50.102 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_17 2024-09-11T06:43:50.102 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap=snap 2024-09-11T06:43:50.102 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 snap rm mirror/image_17@snap 2024-09-11T06:43:50.791 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing snap: 100% complete...done. 2024-09-11T06:43:50.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image_retry cluster2 mirror image_17 2024-09-11T06:43:50.822 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:43:50.822 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:50.822 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_17 2024-09-11T06:43:50.822 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T06:43:50.822 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:43:50.823 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image cluster2 mirror image_17 2024-09-11T06:43:50.823 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:43:50.823 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:50.823 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_17 2024-09-11T06:43:50.823 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_17 2024-09-11T06:43:50.873 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 rm mirror/image_17 2024-09-11T06:43:51.087 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T06:43:51.092 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:43:51.092 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:43:51.092 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_18 2024-09-11T06:43:51.092 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_snapshot cluster2 mirror image_18 snap 2024-09-11T06:43:51.092 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:43:51.093 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:51.093 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_18 2024-09-11T06:43:51.093 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap=snap 2024-09-11T06:43:51.093 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 snap rm mirror/image_18@snap 2024-09-11T06:43:51.192 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:43:51.196+0000 7efd718c5640 -1 reset not still connected to 0x560b2fc524e0 2024-09-11T06:43:51.192 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:43:51.196+0000 7efd718c5640 -1 reset not still connected to 0x560b34cafee0 2024-09-11T06:43:51.350 INFO:tasks.ceph.cluster1.osd.0.smithi006.stderr:2024-09-11T06:43:51.333+0000 7fb45318f640 -1 reset not still connected to 0x5613551fd110 2024-09-11T06:43:51.807 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing snap: 100% complete...done. 2024-09-11T06:43:51.826 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image_retry cluster2 mirror image_18 2024-09-11T06:43:51.826 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:43:51.826 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:51.826 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_18 2024-09-11T06:43:51.826 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T06:43:51.826 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:43:51.827 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image cluster2 mirror image_18 2024-09-11T06:43:51.827 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:43:51.827 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:51.827 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_18 2024-09-11T06:43:51.827 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_18 2024-09-11T06:43:51.875 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 rm mirror/image_18 2024-09-11T06:43:52.114 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T06:43:52.118 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:43:52.118 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:43:52.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_19 2024-09-11T06:43:52.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_snapshot cluster2 mirror image_19 snap 2024-09-11T06:43:52.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:43:52.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:52.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_19 2024-09-11T06:43:52.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap=snap 2024-09-11T06:43:52.119 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 snap rm mirror/image_19@snap 2024-09-11T06:43:52.636 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing snap: 100% complete...done. 2024-09-11T06:43:52.652 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image_retry cluster2 mirror image_19 2024-09-11T06:43:52.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:43:52.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:52.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_19 2024-09-11T06:43:52.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T06:43:52.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:43:52.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image cluster2 mirror image_19 2024-09-11T06:43:52.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:43:52.654 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:52.654 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_19 2024-09-11T06:43:52.654 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_19 2024-09-11T06:43:52.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 rm mirror/image_19 2024-09-11T06:43:52.940 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T06:43:52.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:43:52.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:43:52.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_20 2024-09-11T06:43:52.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_snapshot cluster2 mirror image_20 snap 2024-09-11T06:43:52.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:43:52.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:52.946 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_20 2024-09-11T06:43:52.946 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap=snap 2024-09-11T06:43:52.946 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 snap rm mirror/image_20@snap 2024-09-11T06:43:53.812 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing snap: 100% complete...done. 2024-09-11T06:43:53.844 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image_retry cluster2 mirror image_20 2024-09-11T06:43:53.844 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:43:53.844 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:53.844 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_20 2024-09-11T06:43:53.844 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T06:43:53.845 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:43:53.845 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image cluster2 mirror image_20 2024-09-11T06:43:53.845 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:43:53.845 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:53.845 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_20 2024-09-11T06:43:53.845 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_20 2024-09-11T06:43:53.895 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 rm mirror/image_20 2024-09-11T06:43:54.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T06:43:54.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:43:54.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:43:54.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_21 2024-09-11T06:43:54.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_snapshot cluster2 mirror image_21 snap 2024-09-11T06:43:54.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:43:54.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:54.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_21 2024-09-11T06:43:54.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap=snap 2024-09-11T06:43:54.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 snap rm mirror/image_21@snap 2024-09-11T06:43:54.389 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:43:54.381+0000 7fc41084f640 -1 reset not still connected to 0x55eb15565d40 2024-09-11T06:43:54.389 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:43:54.381+0000 7fc41084f640 -1 reset not still connected to 0x55eb15806dd0 2024-09-11T06:43:54.394 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:43:54.385+0000 7fc41084f640 -1 reset not still connected to 0x55eb16561450 2024-09-11T06:43:54.394 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:43:54.385+0000 7fc41084f640 -1 reset not still connected to 0x55eb1fefeb60 2024-09-11T06:43:54.394 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:43:54.385+0000 7fc41084f640 -1 reset not still connected to 0x55eb2458e680 2024-09-11T06:43:54.394 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:43:54.385+0000 7fc41084f640 -1 reset not still connected to 0x55eb2601e820 2024-09-11T06:43:54.394 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:43:54.385+0000 7fc41084f640 -1 reset not still connected to 0x55eb2da77ba0 2024-09-11T06:43:54.395 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:43:54.385+0000 7fc41084f640 -1 reset not still connected to 0x55eb345e5ad0 2024-09-11T06:43:54.395 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:43:54.385+0000 7fc41084f640 -1 reset not still connected to 0x55eb34898a90 2024-09-11T06:43:54.395 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:43:54.385+0000 7fc41084f640 -1 reset not still connected to 0x55eb35243040 2024-09-11T06:43:54.395 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:43:54.385+0000 7fc41084f640 -1 reset not still connected to 0x55eb3add8270 2024-09-11T06:43:54.395 INFO:tasks.ceph.cluster1.osd.1.smithi006.stderr:2024-09-11T06:43:54.385+0000 7fc41084f640 -1 reset not still connected to 0x55eb4480b930 2024-09-11T06:43:54.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing snap: 100% complete...done. 2024-09-11T06:43:54.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image_retry cluster2 mirror image_21 2024-09-11T06:43:54.854 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:43:54.854 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:54.854 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_21 2024-09-11T06:43:54.854 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T06:43:54.854 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:43:54.854 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image cluster2 mirror image_21 2024-09-11T06:43:54.855 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:43:54.855 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:54.855 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_21 2024-09-11T06:43:54.855 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_21 2024-09-11T06:43:54.916 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 rm mirror/image_21 2024-09-11T06:43:55.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T06:43:55.145 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:43:55.145 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:43:55.145 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_22 2024-09-11T06:43:55.145 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_snapshot cluster2 mirror image_22 snap 2024-09-11T06:43:55.145 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:43:55.145 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:55.145 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_22 2024-09-11T06:43:55.145 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap=snap 2024-09-11T06:43:55.145 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 snap rm mirror/image_22@snap 2024-09-11T06:43:55.823 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing snap: 100% complete...done. 2024-09-11T06:43:55.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image_retry cluster2 mirror image_22 2024-09-11T06:43:55.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:43:55.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:55.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_22 2024-09-11T06:43:55.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T06:43:55.852 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:43:55.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image cluster2 mirror image_22 2024-09-11T06:43:55.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:43:55.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:55.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_22 2024-09-11T06:43:55.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_22 2024-09-11T06:43:55.902 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 rm mirror/image_22 2024-09-11T06:43:56.102 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T06:43:56.107 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:43:56.107 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:43:56.107 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_23 2024-09-11T06:43:56.107 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_snapshot cluster2 mirror image_23 snap 2024-09-11T06:43:56.107 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:43:56.107 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:56.107 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_23 2024-09-11T06:43:56.107 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap=snap 2024-09-11T06:43:56.108 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 snap rm mirror/image_23@snap 2024-09-11T06:43:56.841 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing snap: 100% complete...done. 2024-09-11T06:43:56.861 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image_retry cluster2 mirror image_23 2024-09-11T06:43:56.861 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:43:56.861 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:56.861 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_23 2024-09-11T06:43:56.861 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T06:43:56.862 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:43:56.862 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image cluster2 mirror image_23 2024-09-11T06:43:56.862 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:43:56.862 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:56.862 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_23 2024-09-11T06:43:56.862 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_23 2024-09-11T06:43:56.912 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 rm mirror/image_23 2024-09-11T06:43:57.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T06:43:57.169 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:43:57.169 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:43:57.169 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_24 2024-09-11T06:43:57.169 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_snapshot cluster2 mirror image_24 snap 2024-09-11T06:43:57.169 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:43:57.169 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:57.170 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_24 2024-09-11T06:43:57.170 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap=snap 2024-09-11T06:43:57.170 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 snap rm mirror/image_24@snap 2024-09-11T06:43:57.841 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing snap: 100% complete...done. 2024-09-11T06:43:57.864 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image_retry cluster2 mirror image_24 2024-09-11T06:43:57.864 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:43:57.864 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:57.864 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_24 2024-09-11T06:43:57.864 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T06:43:57.864 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:43:57.865 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image cluster2 mirror image_24 2024-09-11T06:43:57.865 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:43:57.866 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:57.866 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_24 2024-09-11T06:43:57.866 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_24 2024-09-11T06:43:57.915 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 rm mirror/image_24 2024-09-11T06:43:58.169 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T06:43:58.174 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:43:58.174 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:43:58.174 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_25 2024-09-11T06:43:58.175 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_snapshot cluster2 mirror image_25 snap 2024-09-11T06:43:58.175 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:43:58.175 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:58.175 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_25 2024-09-11T06:43:58.175 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap=snap 2024-09-11T06:43:58.175 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 snap rm mirror/image_25@snap 2024-09-11T06:43:58.840 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing snap: 100% complete...done. 2024-09-11T06:43:58.873 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image_retry cluster2 mirror image_25 2024-09-11T06:43:58.873 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:43:58.873 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:58.874 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_25 2024-09-11T06:43:58.874 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T06:43:58.874 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:43:58.874 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image cluster2 mirror image_25 2024-09-11T06:43:58.874 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:43:58.874 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:58.874 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_25 2024-09-11T06:43:58.875 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_25 2024-09-11T06:43:58.925 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 rm mirror/image_25 2024-09-11T06:43:59.187 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T06:43:59.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:43:59.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:43:59.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_26 2024-09-11T06:43:59.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_snapshot cluster2 mirror image_26 snap 2024-09-11T06:43:59.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:43:59.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:59.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_26 2024-09-11T06:43:59.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap=snap 2024-09-11T06:43:59.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 snap rm mirror/image_26@snap 2024-09-11T06:43:59.870 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing snap: 100% complete...done. 2024-09-11T06:43:59.893 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image_retry cluster2 mirror image_26 2024-09-11T06:43:59.893 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:43:59.893 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:59.893 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_26 2024-09-11T06:43:59.893 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T06:43:59.893 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:43:59.894 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image cluster2 mirror image_26 2024-09-11T06:43:59.894 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:43:59.894 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:43:59.894 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_26 2024-09-11T06:43:59.894 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_26 2024-09-11T06:43:59.944 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 rm mirror/image_26 2024-09-11T06:44:00.168 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T06:44:00.173 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:44:00.173 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:44:00.173 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_27 2024-09-11T06:44:00.173 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_snapshot cluster2 mirror image_27 snap 2024-09-11T06:44:00.173 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:44:00.173 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:00.173 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_27 2024-09-11T06:44:00.173 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap=snap 2024-09-11T06:44:00.173 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 snap rm mirror/image_27@snap 2024-09-11T06:44:00.855 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing snap: 100% complete...done. 2024-09-11T06:44:00.889 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image_retry cluster2 mirror image_27 2024-09-11T06:44:00.889 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:44:00.889 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:00.889 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_27 2024-09-11T06:44:00.889 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T06:44:00.890 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:44:00.890 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image cluster2 mirror image_27 2024-09-11T06:44:00.890 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:44:00.890 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:00.890 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_27 2024-09-11T06:44:00.890 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_27 2024-09-11T06:44:00.943 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 rm mirror/image_27 2024-09-11T06:44:01.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T06:44:01.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:44:01.199 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:44:01.199 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_28 2024-09-11T06:44:01.199 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_snapshot cluster2 mirror image_28 snap 2024-09-11T06:44:01.199 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:44:01.199 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:01.199 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_28 2024-09-11T06:44:01.199 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap=snap 2024-09-11T06:44:01.199 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 snap rm mirror/image_28@snap 2024-09-11T06:44:01.872 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing snap: 100% complete...done. 2024-09-11T06:44:01.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image_retry cluster2 mirror image_28 2024-09-11T06:44:01.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:44:01.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:01.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_28 2024-09-11T06:44:01.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T06:44:01.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:44:01.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image cluster2 mirror image_28 2024-09-11T06:44:01.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:44:01.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:01.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_28 2024-09-11T06:44:01.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_28 2024-09-11T06:44:01.956 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 rm mirror/image_28 2024-09-11T06:44:02.180 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T06:44:02.184 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:44:02.184 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:44:02.184 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_29 2024-09-11T06:44:02.185 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_snapshot cluster2 mirror image_29 snap 2024-09-11T06:44:02.185 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:44:02.185 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:02.185 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_29 2024-09-11T06:44:02.185 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap=snap 2024-09-11T06:44:02.185 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 snap rm mirror/image_29@snap 2024-09-11T06:44:02.654 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing snap: 100% complete...done. 2024-09-11T06:44:02.668 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image_retry cluster2 mirror image_29 2024-09-11T06:44:02.668 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:44:02.668 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:02.668 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_29 2024-09-11T06:44:02.668 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T06:44:02.668 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:44:02.669 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image cluster2 mirror image_29 2024-09-11T06:44:02.669 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:44:02.669 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:02.669 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_29 2024-09-11T06:44:02.669 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_29 2024-09-11T06:44:02.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 rm mirror/image_29 2024-09-11T06:44:02.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T06:44:02.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:44:02.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:44:02.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_30 2024-09-11T06:44:02.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_snapshot cluster2 mirror image_30 snap 2024-09-11T06:44:02.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:44:02.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:02.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_30 2024-09-11T06:44:02.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap=snap 2024-09-11T06:44:02.950 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 snap rm mirror/image_30@snap 2024-09-11T06:44:03.901 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing snap: 100% complete...done. 2024-09-11T06:44:03.926 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image_retry cluster2 mirror image_30 2024-09-11T06:44:03.926 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:44:03.926 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:03.926 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_30 2024-09-11T06:44:03.926 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T06:44:03.926 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:44:03.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image cluster2 mirror image_30 2024-09-11T06:44:03.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:44:03.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:03.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_30 2024-09-11T06:44:03.927 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_30 2024-09-11T06:44:03.979 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 rm mirror/image_30 2024-09-11T06:44:04.212 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T06:44:04.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:44:04.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:44:04.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_31 2024-09-11T06:44:04.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_snapshot cluster2 mirror image_31 snap 2024-09-11T06:44:04.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:44:04.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:04.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_31 2024-09-11T06:44:04.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap=snap 2024-09-11T06:44:04.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 snap rm mirror/image_31@snap 2024-09-11T06:44:04.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing snap: 100% complete...done. 2024-09-11T06:44:04.921 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image_retry cluster2 mirror image_31 2024-09-11T06:44:04.921 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:44:04.921 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:04.921 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_31 2024-09-11T06:44:04.921 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T06:44:04.921 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:44:04.922 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image cluster2 mirror image_31 2024-09-11T06:44:04.922 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:44:04.922 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:04.922 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_31 2024-09-11T06:44:04.922 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_31 2024-09-11T06:44:04.969 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 rm mirror/image_31 2024-09-11T06:44:05.209 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T06:44:05.213 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:44:05.213 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:44:05.213 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_32 2024-09-11T06:44:05.213 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_snapshot cluster2 mirror image_32 snap 2024-09-11T06:44:05.213 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:44:05.216 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:05.216 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_32 2024-09-11T06:44:05.216 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap=snap 2024-09-11T06:44:05.216 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 snap rm mirror/image_32@snap 2024-09-11T06:44:05.889 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing snap: 100% complete...done. 2024-09-11T06:44:05.909 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image_retry cluster2 mirror image_32 2024-09-11T06:44:05.909 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:44:05.909 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:05.909 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_32 2024-09-11T06:44:05.909 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T06:44:05.909 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:44:05.910 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image cluster2 mirror image_32 2024-09-11T06:44:05.910 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:44:05.910 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:05.910 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_32 2024-09-11T06:44:05.910 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_32 2024-09-11T06:44:05.960 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 rm mirror/image_32 2024-09-11T06:44:06.167 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T06:44:06.171 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:44:06.171 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:44:06.171 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_33 2024-09-11T06:44:06.171 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_snapshot cluster2 mirror image_33 snap 2024-09-11T06:44:06.171 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:44:06.171 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:06.171 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_33 2024-09-11T06:44:06.171 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap=snap 2024-09-11T06:44:06.172 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 snap rm mirror/image_33@snap 2024-09-11T06:44:06.891 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing snap: 100% complete...done. 2024-09-11T06:44:06.916 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image_retry cluster2 mirror image_33 2024-09-11T06:44:06.916 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:44:06.916 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:06.916 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_33 2024-09-11T06:44:06.916 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T06:44:06.916 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:44:06.917 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image cluster2 mirror image_33 2024-09-11T06:44:06.917 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:44:06.917 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:06.917 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_33 2024-09-11T06:44:06.917 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_33 2024-09-11T06:44:06.966 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 rm mirror/image_33 2024-09-11T06:44:07.216 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T06:44:07.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:44:07.221 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:44:07.221 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_34 2024-09-11T06:44:07.221 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_snapshot cluster2 mirror image_34 snap 2024-09-11T06:44:07.221 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:44:07.221 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:07.221 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_34 2024-09-11T06:44:07.221 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap=snap 2024-09-11T06:44:07.221 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 snap rm mirror/image_34@snap 2024-09-11T06:44:07.923 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing snap: 100% complete...done. 2024-09-11T06:44:07.944 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image_retry cluster2 mirror image_34 2024-09-11T06:44:07.944 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:44:07.944 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:07.944 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_34 2024-09-11T06:44:07.944 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T06:44:07.944 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:44:07.945 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image cluster2 mirror image_34 2024-09-11T06:44:07.946 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:44:07.946 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:07.946 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_34 2024-09-11T06:44:07.946 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_34 2024-09-11T06:44:07.995 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 rm mirror/image_34 2024-09-11T06:44:08.248 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T06:44:08.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:44:08.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:44:08.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_35 2024-09-11T06:44:08.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_snapshot cluster2 mirror image_35 snap 2024-09-11T06:44:08.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:44:08.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:08.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_35 2024-09-11T06:44:08.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap=snap 2024-09-11T06:44:08.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 snap rm mirror/image_35@snap 2024-09-11T06:44:08.906 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing snap: 100% complete...done. 2024-09-11T06:44:08.930 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image_retry cluster2 mirror image_35 2024-09-11T06:44:08.931 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:44:08.931 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:08.931 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_35 2024-09-11T06:44:08.931 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T06:44:08.931 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:44:08.931 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image cluster2 mirror image_35 2024-09-11T06:44:08.931 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:44:08.932 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:08.932 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_35 2024-09-11T06:44:08.932 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_35 2024-09-11T06:44:08.982 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 rm mirror/image_35 2024-09-11T06:44:09.221 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T06:44:09.225 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:44:09.226 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:44:09.226 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_36 2024-09-11T06:44:09.226 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_snapshot cluster2 mirror image_36 snap 2024-09-11T06:44:09.226 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:44:09.226 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:09.226 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_36 2024-09-11T06:44:09.226 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap=snap 2024-09-11T06:44:09.226 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 snap rm mirror/image_36@snap 2024-09-11T06:44:09.900 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing snap: 100% complete...done. 2024-09-11T06:44:09.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image_retry cluster2 mirror image_36 2024-09-11T06:44:09.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:44:09.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:09.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_36 2024-09-11T06:44:09.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T06:44:09.919 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:44:09.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image cluster2 mirror image_36 2024-09-11T06:44:09.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:44:09.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:09.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_36 2024-09-11T06:44:09.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_36 2024-09-11T06:44:09.966 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 rm mirror/image_36 2024-09-11T06:44:10.218 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T06:44:10.222 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:44:10.223 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:44:10.223 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_37 2024-09-11T06:44:10.223 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_snapshot cluster2 mirror image_37 snap 2024-09-11T06:44:10.223 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:44:10.223 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:10.223 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_37 2024-09-11T06:44:10.223 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap=snap 2024-09-11T06:44:10.223 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 snap rm mirror/image_37@snap 2024-09-11T06:44:10.946 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing snap: 100% complete...done. 2024-09-11T06:44:10.977 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image_retry cluster2 mirror image_37 2024-09-11T06:44:10.977 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:44:10.977 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:10.977 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_37 2024-09-11T06:44:10.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T06:44:10.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:44:10.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image cluster2 mirror image_37 2024-09-11T06:44:10.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:44:10.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:10.979 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_37 2024-09-11T06:44:10.979 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_37 2024-09-11T06:44:11.028 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 rm mirror/image_37 2024-09-11T06:44:11.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T06:44:11.255 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:44:11.255 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:44:11.256 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_38 2024-09-11T06:44:11.256 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_snapshot cluster2 mirror image_38 snap 2024-09-11T06:44:11.256 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:44:11.256 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:11.256 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_38 2024-09-11T06:44:11.256 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap=snap 2024-09-11T06:44:11.256 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 snap rm mirror/image_38@snap 2024-09-11T06:44:11.943 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing snap: 100% complete...done. 2024-09-11T06:44:11.971 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image_retry cluster2 mirror image_38 2024-09-11T06:44:11.971 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:44:11.971 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:11.971 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_38 2024-09-11T06:44:11.972 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T06:44:11.972 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:44:11.972 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image cluster2 mirror image_38 2024-09-11T06:44:11.972 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:44:11.973 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:11.973 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_38 2024-09-11T06:44:11.973 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_38 2024-09-11T06:44:12.024 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 rm mirror/image_38 2024-09-11T06:44:12.236 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T06:44:12.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:44:12.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:44:12.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_39 2024-09-11T06:44:12.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_snapshot cluster2 mirror image_39 snap 2024-09-11T06:44:12.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:44:12.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:12.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_39 2024-09-11T06:44:12.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap=snap 2024-09-11T06:44:12.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 snap rm mirror/image_39@snap 2024-09-11T06:44:12.629 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing snap: 100% complete...done. 2024-09-11T06:44:12.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image_retry cluster2 mirror image_39 2024-09-11T06:44:12.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:44:12.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:12.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_39 2024-09-11T06:44:12.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T06:44:12.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:44:12.658 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image cluster2 mirror image_39 2024-09-11T06:44:12.658 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:44:12.658 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:12.658 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_39 2024-09-11T06:44:12.658 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_39 2024-09-11T06:44:12.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 rm mirror/image_39 2024-09-11T06:44:12.963 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T06:44:12.967 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:44:12.967 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:44:12.967 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_40 2024-09-11T06:44:12.968 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_snapshot cluster2 mirror image_40 snap 2024-09-11T06:44:12.968 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:44:12.968 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:12.968 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_40 2024-09-11T06:44:12.968 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap=snap 2024-09-11T06:44:12.968 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 snap rm mirror/image_40@snap 2024-09-11T06:44:13.959 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing snap: 100% complete...done. 2024-09-11T06:44:13.981 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image_retry cluster2 mirror image_40 2024-09-11T06:44:13.981 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:44:13.981 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:13.981 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_40 2024-09-11T06:44:13.981 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T06:44:13.981 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:44:13.982 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image cluster2 mirror image_40 2024-09-11T06:44:13.982 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:44:13.983 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:13.983 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_40 2024-09-11T06:44:13.983 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_40 2024-09-11T06:44:14.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 rm mirror/image_40 2024-09-11T06:44:14.296 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T06:44:14.300 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:44:14.300 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:44:14.301 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_41 2024-09-11T06:44:14.301 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_snapshot cluster2 mirror image_41 snap 2024-09-11T06:44:14.301 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:44:14.301 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:14.301 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_41 2024-09-11T06:44:14.301 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap=snap 2024-09-11T06:44:14.301 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 snap rm mirror/image_41@snap 2024-09-11T06:44:14.924 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing snap: 100% complete...done. 2024-09-11T06:44:14.954 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image_retry cluster2 mirror image_41 2024-09-11T06:44:14.954 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:44:14.954 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:14.954 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_41 2024-09-11T06:44:14.954 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T06:44:14.954 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:44:14.955 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image cluster2 mirror image_41 2024-09-11T06:44:14.955 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:44:14.955 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:14.955 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_41 2024-09-11T06:44:14.955 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_41 2024-09-11T06:44:15.002 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 rm mirror/image_41 2024-09-11T06:44:15.162 INFO:tasks.ceph.cluster2.osd.0.smithi112.stderr:2024-09-11T06:44:15.163+0000 7f3fa0017640 -1 reset not still connected to 0x563d195b4c30 2024-09-11T06:44:15.227 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T06:44:15.231 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:44:15.231 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:44:15.231 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_42 2024-09-11T06:44:15.231 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_snapshot cluster2 mirror image_42 snap 2024-09-11T06:44:15.231 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:44:15.231 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:15.232 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_42 2024-09-11T06:44:15.232 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap=snap 2024-09-11T06:44:15.232 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 snap rm mirror/image_42@snap 2024-09-11T06:44:15.940 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing snap: 100% complete...done. 2024-09-11T06:44:15.961 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image_retry cluster2 mirror image_42 2024-09-11T06:44:15.962 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:44:15.962 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:15.962 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_42 2024-09-11T06:44:15.962 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T06:44:15.962 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:44:15.963 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image cluster2 mirror image_42 2024-09-11T06:44:15.963 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:44:15.963 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:15.963 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_42 2024-09-11T06:44:15.963 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_42 2024-09-11T06:44:16.008 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 rm mirror/image_42 2024-09-11T06:44:16.259 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T06:44:16.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:44:16.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:44:16.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_43 2024-09-11T06:44:16.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_snapshot cluster2 mirror image_43 snap 2024-09-11T06:44:16.263 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:44:16.264 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:16.264 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_43 2024-09-11T06:44:16.264 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap=snap 2024-09-11T06:44:16.264 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 snap rm mirror/image_43@snap 2024-09-11T06:44:16.954 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing snap: 100% complete...done. 2024-09-11T06:44:16.977 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image_retry cluster2 mirror image_43 2024-09-11T06:44:16.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:44:16.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:16.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_43 2024-09-11T06:44:16.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T06:44:16.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:44:16.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image cluster2 mirror image_43 2024-09-11T06:44:16.979 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:44:16.979 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:16.979 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_43 2024-09-11T06:44:16.979 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_43 2024-09-11T06:44:17.027 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 rm mirror/image_43 2024-09-11T06:44:17.273 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T06:44:17.277 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:44:17.278 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:44:17.278 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_44 2024-09-11T06:44:17.278 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_snapshot cluster2 mirror image_44 snap 2024-09-11T06:44:17.278 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:44:17.278 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:17.278 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_44 2024-09-11T06:44:17.278 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap=snap 2024-09-11T06:44:17.278 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 snap rm mirror/image_44@snap 2024-09-11T06:44:17.979 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing snap: 100% complete...done. 2024-09-11T06:44:18.005 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image_retry cluster2 mirror image_44 2024-09-11T06:44:18.005 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:44:18.005 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:18.005 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_44 2024-09-11T06:44:18.005 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T06:44:18.005 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:44:18.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image cluster2 mirror image_44 2024-09-11T06:44:18.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:44:18.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:18.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_44 2024-09-11T06:44:18.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_44 2024-09-11T06:44:18.056 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 rm mirror/image_44 2024-09-11T06:44:18.273 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T06:44:18.278 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:44:18.278 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:44:18.278 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_45 2024-09-11T06:44:18.278 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_snapshot cluster2 mirror image_45 snap 2024-09-11T06:44:18.278 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:44:18.278 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:18.278 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_45 2024-09-11T06:44:18.278 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap=snap 2024-09-11T06:44:18.278 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 snap rm mirror/image_45@snap 2024-09-11T06:44:18.991 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing snap: 100% complete...done. 2024-09-11T06:44:19.014 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image_retry cluster2 mirror image_45 2024-09-11T06:44:19.014 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:44:19.014 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:19.015 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_45 2024-09-11T06:44:19.015 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T06:44:19.015 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:44:19.015 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image cluster2 mirror image_45 2024-09-11T06:44:19.015 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:44:19.016 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:19.016 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_45 2024-09-11T06:44:19.016 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_45 2024-09-11T06:44:19.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 rm mirror/image_45 2024-09-11T06:44:19.279 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T06:44:19.283 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:44:19.283 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:44:19.283 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_46 2024-09-11T06:44:19.284 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_snapshot cluster2 mirror image_46 snap 2024-09-11T06:44:19.284 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:44:19.284 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:19.284 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_46 2024-09-11T06:44:19.284 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap=snap 2024-09-11T06:44:19.284 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 snap rm mirror/image_46@snap 2024-09-11T06:44:19.966 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing snap: 100% complete...done. 2024-09-11T06:44:19.989 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image_retry cluster2 mirror image_46 2024-09-11T06:44:19.989 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:44:19.989 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:19.990 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_46 2024-09-11T06:44:19.990 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T06:44:19.990 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:44:19.990 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image cluster2 mirror image_46 2024-09-11T06:44:19.990 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:44:19.990 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:19.991 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_46 2024-09-11T06:44:19.991 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_46 2024-09-11T06:44:20.037 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 rm mirror/image_46 2024-09-11T06:44:20.232 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T06:44:20.236 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:44:20.236 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:44:20.237 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_47 2024-09-11T06:44:20.237 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_snapshot cluster2 mirror image_47 snap 2024-09-11T06:44:20.237 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:44:20.237 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:20.237 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_47 2024-09-11T06:44:20.237 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap=snap 2024-09-11T06:44:20.237 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 snap rm mirror/image_47@snap 2024-09-11T06:44:20.987 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing snap: 100% complete...done. 2024-09-11T06:44:21.037 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image_retry cluster2 mirror image_47 2024-09-11T06:44:21.037 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:44:21.037 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:21.037 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_47 2024-09-11T06:44:21.038 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T06:44:21.038 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:44:21.038 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image cluster2 mirror image_47 2024-09-11T06:44:21.038 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:44:21.038 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:21.038 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_47 2024-09-11T06:44:21.039 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_47 2024-09-11T06:44:21.091 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 rm mirror/image_47 2024-09-11T06:44:21.312 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T06:44:21.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:44:21.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:44:21.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_48 2024-09-11T06:44:21.317 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_snapshot cluster2 mirror image_48 snap 2024-09-11T06:44:21.317 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:44:21.317 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:21.317 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_48 2024-09-11T06:44:21.317 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap=snap 2024-09-11T06:44:21.317 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 snap rm mirror/image_48@snap 2024-09-11T06:44:22.002 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing snap: 100% complete...done. 2024-09-11T06:44:22.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image_retry cluster2 mirror image_48 2024-09-11T06:44:22.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:44:22.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:22.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_48 2024-09-11T06:44:22.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T06:44:22.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:44:22.035 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image cluster2 mirror image_48 2024-09-11T06:44:22.035 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:44:22.035 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:22.036 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_48 2024-09-11T06:44:22.036 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_48 2024-09-11T06:44:22.085 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 rm mirror/image_48 2024-09-11T06:44:22.316 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T06:44:22.320 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:44:22.320 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:44:22.320 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_49 2024-09-11T06:44:22.320 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_snapshot cluster2 mirror image_49 snap 2024-09-11T06:44:22.320 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:44:22.320 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:22.320 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_49 2024-09-11T06:44:22.321 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap=snap 2024-09-11T06:44:22.321 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 snap rm mirror/image_49@snap 2024-09-11T06:44:22.636 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing snap: 100% complete...done. 2024-09-11T06:44:22.664 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image_retry cluster2 mirror image_49 2024-09-11T06:44:22.664 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:44:22.664 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:22.664 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_49 2024-09-11T06:44:22.664 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T06:44:22.664 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:44:22.665 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image cluster2 mirror image_49 2024-09-11T06:44:22.665 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:44:22.665 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:22.665 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_49 2024-09-11T06:44:22.665 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_49 2024-09-11T06:44:22.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 rm mirror/image_49 2024-09-11T06:44:22.947 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T06:44:22.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:44:22.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:44:22.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_50 2024-09-11T06:44:22.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_snapshot cluster2 mirror image_50 snap 2024-09-11T06:44:22.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:44:22.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:22.951 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_50 2024-09-11T06:44:22.952 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local snap=snap 2024-09-11T06:44:22.952 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 snap rm mirror/image_50@snap 2024-09-11T06:44:24.022 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing snap: 100% complete...done. 2024-09-11T06:44:24.046 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image_retry cluster2 mirror image_50 2024-09-11T06:44:24.046 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:44:24.046 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:24.046 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_50 2024-09-11T06:44:24.046 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T06:44:24.046 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:44:24.047 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image cluster2 mirror image_50 2024-09-11T06:44:24.047 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:44:24.047 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:24.047 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_50 2024-09-11T06:44:24.047 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_50 2024-09-11T06:44:24.095 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 rm mirror/image_50 2024-09-11T06:44:24.318 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T06:44:24.323 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:44:24.323 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ testlog 'TEST: image deletions should propagate' 2024-09-11T06:44:24.323 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ tee -a /tmp/tmp.Dxwd1ESqKa/rbd-mirror.test.log 2024-09-11T06:44:24.323 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ date '+%F %T' 2024-09-11T06:44:24.325 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ echo 2024-09-11 06:44:24 TEST: image deletions should propagate 2024-09-11T06:44:24.325 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:2024-09-11 06:44:24 TEST: image deletions should propagate 2024-09-11T06:44:24.325 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_pool_images cluster1 mirror 0 2024-09-11T06:44:24.325 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:44:24.325 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:24.325 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_count=0 2024-09-11T06:44:24.325 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:44:24.325 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local count 2024-09-11T06:44:24.326 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local last_count=0 2024-09-11T06:44:24.326 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ true 2024-09-11T06:44:24.326 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 1 40 2024-09-11T06:44:24.326 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in `seq 1 40` 2024-09-11T06:44:24.326 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 1 -ne 1 2024-09-11T06:44:24.327 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2024-09-11T06:44:24.327 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep 'images: ' 2024-09-11T06:44:24.327 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut '-d ' -f 2 2024-09-11T06:44:24.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: failed to query services: (13) Permission denied 2024-09-11T06:44:24.374 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ count=0 2024-09-11T06:44:24.375 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 0 = 0 2024-09-11T06:44:24.375 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:44:24.375 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_pool_healthy cluster1 mirror 0 2024-09-11T06:44:24.375 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:44:24.375 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:24.375 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:44:24.375 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state 2024-09-11T06:44:24.375 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 1 40 2024-09-11T06:44:24.376 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in `seq 1 40` 2024-09-11T06:44:24.376 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 1 -ne 1 2024-09-11T06:44:24.376 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2024-09-11T06:44:24.376 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep 'image health:' 2024-09-11T06:44:24.376 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cut '-d ' -f 3 2024-09-11T06:44:24.416 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: failed to query services: (13) Permission denied 2024-09-11T06:44:24.420 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ state=OK 2024-09-11T06:44:24.420 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test OK = ERROR 2024-09-11T06:44:24.420 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test OK = OK 2024-09-11T06:44:24.420 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:44:24.421 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 1 50 2024-09-11T06:44:24.423 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:44:24.424 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_1 2024-09-11T06:44:24.424 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_present cluster1 mirror image_1 deleted 2024-09-11T06:44:24.424 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:44:24.424 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:24.424 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_1 2024-09-11T06:44:24.425 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=deleted 2024-09-11T06:44:24.425 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:44:24.425 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:44:24.425 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:44:24.425 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_1 2024-09-11T06:44:24.425 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:44:24.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:44:24.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_1 2024-09-11T06:44:24.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-09-11T06:44:24.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:44:24.502 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_1: (2) No such file or directory 2024-09-11T06:44:24.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image_id= 2024-09-11T06:44:24.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T06:44:24.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.1 2024-09-11T06:44:24.607 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_present cluster1 mirror image_1 deleted '' 2024-09-11T06:44:24.608 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:44:24.608 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:24.608 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_1 2024-09-11T06:44:24.608 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=deleted 2024-09-11T06:44:24.608 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:44:24.608 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=deleted 2024-09-11T06:44:24.608 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_image_id 2024-09-11T06:44:24.609 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_1 2024-09-11T06:44:24.609 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:44:24.609 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:44:24.609 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_1 2024-09-11T06:44:24.609 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-09-11T06:44:24.609 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:44:24.650 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_1: (2) No such file or directory 2024-09-11T06:44:24.656 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_image_id= 2024-09-11T06:44:24.656 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:44:24.656 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test deleted = deleted 2024-09-11T06:44:24.656 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:44:24.656 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:44:24.656 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_2 2024-09-11T06:44:24.656 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_present cluster1 mirror image_2 deleted 2024-09-11T06:44:24.656 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:44:24.656 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:24.656 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_2 2024-09-11T06:44:24.656 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=deleted 2024-09-11T06:44:24.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:44:24.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:44:24.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:44:24.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_2 2024-09-11T06:44:24.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:44:24.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:44:24.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_2 2024-09-11T06:44:24.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-09-11T06:44:24.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:44:24.692 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_2: (2) No such file or directory 2024-09-11T06:44:24.696 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image_id= 2024-09-11T06:44:24.696 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T06:44:24.696 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.1 2024-09-11T06:44:24.798 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_present cluster1 mirror image_2 deleted '' 2024-09-11T06:44:24.798 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:44:24.798 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:24.798 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_2 2024-09-11T06:44:24.798 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=deleted 2024-09-11T06:44:24.798 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:44:24.798 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=deleted 2024-09-11T06:44:24.799 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_image_id 2024-09-11T06:44:24.799 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_2 2024-09-11T06:44:24.799 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:44:24.799 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:44:24.799 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_2 2024-09-11T06:44:24.799 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-09-11T06:44:24.800 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:44:24.844 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_2: (2) No such file or directory 2024-09-11T06:44:24.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_image_id= 2024-09-11T06:44:24.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:44:24.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test deleted = deleted 2024-09-11T06:44:24.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:44:24.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:44:24.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_3 2024-09-11T06:44:24.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_present cluster1 mirror image_3 deleted 2024-09-11T06:44:24.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:44:24.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:24.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_3 2024-09-11T06:44:24.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=deleted 2024-09-11T06:44:24.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:44:24.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:44:24.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:44:24.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_3 2024-09-11T06:44:24.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:44:24.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:44:24.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_3 2024-09-11T06:44:24.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-09-11T06:44:24.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:44:24.889 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_3: (2) No such file or directory 2024-09-11T06:44:24.893 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image_id= 2024-09-11T06:44:24.893 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T06:44:24.893 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.1 2024-09-11T06:44:24.995 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_present cluster1 mirror image_3 deleted '' 2024-09-11T06:44:24.995 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:44:24.995 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:24.995 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_3 2024-09-11T06:44:24.995 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=deleted 2024-09-11T06:44:24.995 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:44:24.995 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=deleted 2024-09-11T06:44:24.995 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_image_id 2024-09-11T06:44:24.996 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_3 2024-09-11T06:44:24.996 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:44:24.996 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:44:24.996 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_3 2024-09-11T06:44:24.996 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-09-11T06:44:24.996 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:44:25.043 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_3: (2) No such file or directory 2024-09-11T06:44:25.047 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_image_id= 2024-09-11T06:44:25.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:44:25.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test deleted = deleted 2024-09-11T06:44:25.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:44:25.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:44:25.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_4 2024-09-11T06:44:25.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_present cluster1 mirror image_4 deleted 2024-09-11T06:44:25.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:44:25.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:25.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_4 2024-09-11T06:44:25.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=deleted 2024-09-11T06:44:25.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:44:25.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:44:25.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:44:25.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_4 2024-09-11T06:44:25.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:44:25.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:44:25.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_4 2024-09-11T06:44:25.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-09-11T06:44:25.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:44:25.089 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_4: (2) No such file or directory 2024-09-11T06:44:25.093 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image_id= 2024-09-11T06:44:25.093 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T06:44:25.093 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.1 2024-09-11T06:44:25.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_present cluster1 mirror image_4 deleted '' 2024-09-11T06:44:25.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:44:25.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:25.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_4 2024-09-11T06:44:25.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=deleted 2024-09-11T06:44:25.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:44:25.196 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=deleted 2024-09-11T06:44:25.196 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_image_id 2024-09-11T06:44:25.196 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_4 2024-09-11T06:44:25.196 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:44:25.196 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:44:25.196 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_4 2024-09-11T06:44:25.196 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-09-11T06:44:25.196 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:44:25.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_4: (2) No such file or directory 2024-09-11T06:44:25.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_image_id= 2024-09-11T06:44:25.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:44:25.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test deleted = deleted 2024-09-11T06:44:25.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:44:25.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:44:25.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_5 2024-09-11T06:44:25.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_present cluster1 mirror image_5 deleted 2024-09-11T06:44:25.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:44:25.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:25.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_5 2024-09-11T06:44:25.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=deleted 2024-09-11T06:44:25.248 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:44:25.248 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:44:25.248 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:44:25.248 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_5 2024-09-11T06:44:25.248 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:44:25.248 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:44:25.248 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_5 2024-09-11T06:44:25.248 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-09-11T06:44:25.248 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:44:25.286 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_5: (2) No such file or directory 2024-09-11T06:44:25.290 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image_id= 2024-09-11T06:44:25.290 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T06:44:25.290 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.1 2024-09-11T06:44:25.391 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_present cluster1 mirror image_5 deleted '' 2024-09-11T06:44:25.391 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:44:25.391 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:25.391 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_5 2024-09-11T06:44:25.392 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=deleted 2024-09-11T06:44:25.392 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:44:25.392 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=deleted 2024-09-11T06:44:25.392 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_image_id 2024-09-11T06:44:25.392 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_5 2024-09-11T06:44:25.393 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:44:25.393 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:44:25.393 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_5 2024-09-11T06:44:25.393 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-09-11T06:44:25.393 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:44:25.437 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_5: (2) No such file or directory 2024-09-11T06:44:25.442 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_image_id= 2024-09-11T06:44:25.442 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:44:25.442 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test deleted = deleted 2024-09-11T06:44:25.443 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:44:25.443 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:44:25.443 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_6 2024-09-11T06:44:25.443 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_present cluster1 mirror image_6 deleted 2024-09-11T06:44:25.443 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:44:25.443 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:25.443 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_6 2024-09-11T06:44:25.443 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=deleted 2024-09-11T06:44:25.443 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:44:25.443 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:44:25.443 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:44:25.443 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_6 2024-09-11T06:44:25.444 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:44:25.444 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:44:25.444 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_6 2024-09-11T06:44:25.444 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-09-11T06:44:25.444 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:44:25.513 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_6: (2) No such file or directory 2024-09-11T06:44:25.518 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image_id= 2024-09-11T06:44:25.518 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T06:44:25.518 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.1 2024-09-11T06:44:25.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_present cluster1 mirror image_6 deleted '' 2024-09-11T06:44:25.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:44:25.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:25.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_6 2024-09-11T06:44:25.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=deleted 2024-09-11T06:44:25.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:44:25.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=deleted 2024-09-11T06:44:25.621 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_image_id 2024-09-11T06:44:25.621 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_6 2024-09-11T06:44:25.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:44:25.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:44:25.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_6 2024-09-11T06:44:25.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-09-11T06:44:25.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:44:25.672 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_6: (2) No such file or directory 2024-09-11T06:44:25.678 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_image_id= 2024-09-11T06:44:25.678 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:44:25.678 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test deleted = deleted 2024-09-11T06:44:25.678 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:44:25.678 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:44:25.678 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_7 2024-09-11T06:44:25.678 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_present cluster1 mirror image_7 deleted 2024-09-11T06:44:25.678 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:44:25.678 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:25.678 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_7 2024-09-11T06:44:25.678 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=deleted 2024-09-11T06:44:25.679 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:44:25.679 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:44:25.679 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:44:25.679 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_7 2024-09-11T06:44:25.679 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:44:25.679 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:44:25.679 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_7 2024-09-11T06:44:25.679 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-09-11T06:44:25.679 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:44:25.714 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_7: (2) No such file or directory 2024-09-11T06:44:25.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image_id= 2024-09-11T06:44:25.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T06:44:25.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.1 2024-09-11T06:44:25.820 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_present cluster1 mirror image_7 deleted '' 2024-09-11T06:44:25.820 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:44:25.820 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:25.820 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_7 2024-09-11T06:44:25.820 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=deleted 2024-09-11T06:44:25.820 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:44:25.820 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=deleted 2024-09-11T06:44:25.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_image_id 2024-09-11T06:44:25.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_7 2024-09-11T06:44:25.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:44:25.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:44:25.822 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_7 2024-09-11T06:44:25.822 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-09-11T06:44:25.822 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:44:25.865 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_7: (2) No such file or directory 2024-09-11T06:44:25.870 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_image_id= 2024-09-11T06:44:25.871 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:44:25.871 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test deleted = deleted 2024-09-11T06:44:25.871 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:44:25.871 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:44:25.871 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_8 2024-09-11T06:44:25.871 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_present cluster1 mirror image_8 deleted 2024-09-11T06:44:25.871 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:44:25.871 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:25.871 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_8 2024-09-11T06:44:25.871 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=deleted 2024-09-11T06:44:25.871 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:44:25.871 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:44:25.871 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:44:25.872 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_8 2024-09-11T06:44:25.872 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:44:25.872 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:44:25.872 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_8 2024-09-11T06:44:25.872 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-09-11T06:44:25.872 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:44:25.909 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_8: (2) No such file or directory 2024-09-11T06:44:25.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image_id= 2024-09-11T06:44:25.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T06:44:25.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.1 2024-09-11T06:44:26.016 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_present cluster1 mirror image_8 deleted '' 2024-09-11T06:44:26.016 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:44:26.016 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:26.016 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_8 2024-09-11T06:44:26.016 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=deleted 2024-09-11T06:44:26.016 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:44:26.016 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=deleted 2024-09-11T06:44:26.016 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_image_id 2024-09-11T06:44:26.016 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_8 2024-09-11T06:44:26.016 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:44:26.017 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:44:26.017 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_8 2024-09-11T06:44:26.017 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-09-11T06:44:26.017 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:44:26.064 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_8: (2) No such file or directory 2024-09-11T06:44:26.069 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_image_id= 2024-09-11T06:44:26.069 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:44:26.069 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test deleted = deleted 2024-09-11T06:44:26.069 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:44:26.069 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:44:26.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_9 2024-09-11T06:44:26.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_present cluster1 mirror image_9 deleted 2024-09-11T06:44:26.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:44:26.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:26.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_9 2024-09-11T06:44:26.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=deleted 2024-09-11T06:44:26.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:44:26.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:44:26.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:44:26.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_9 2024-09-11T06:44:26.070 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:44:26.071 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:44:26.071 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_9 2024-09-11T06:44:26.071 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-09-11T06:44:26.071 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:44:26.108 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_9: (2) No such file or directory 2024-09-11T06:44:26.112 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image_id= 2024-09-11T06:44:26.112 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T06:44:26.112 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.1 2024-09-11T06:44:26.213 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_present cluster1 mirror image_9 deleted '' 2024-09-11T06:44:26.214 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:44:26.214 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:26.214 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_9 2024-09-11T06:44:26.214 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=deleted 2024-09-11T06:44:26.214 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:44:26.214 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=deleted 2024-09-11T06:44:26.214 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_image_id 2024-09-11T06:44:26.215 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_9 2024-09-11T06:44:26.215 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:44:26.215 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:44:26.215 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_9 2024-09-11T06:44:26.215 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:44:26.215 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-09-11T06:44:26.260 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_9: (2) No such file or directory 2024-09-11T06:44:26.265 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_image_id= 2024-09-11T06:44:26.266 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:44:26.266 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test deleted = deleted 2024-09-11T06:44:26.266 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:44:26.266 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:44:26.266 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_10 2024-09-11T06:44:26.266 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_present cluster1 mirror image_10 deleted 2024-09-11T06:44:26.266 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:44:26.266 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:26.266 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_10 2024-09-11T06:44:26.266 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=deleted 2024-09-11T06:44:26.266 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:44:26.266 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:44:26.266 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:44:26.266 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_10 2024-09-11T06:44:26.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:44:26.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:44:26.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_10 2024-09-11T06:44:26.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-09-11T06:44:26.267 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:44:26.304 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_10: (2) No such file or directory 2024-09-11T06:44:26.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image_id= 2024-09-11T06:44:26.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T06:44:26.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.1 2024-09-11T06:44:26.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_present cluster1 mirror image_10 deleted '' 2024-09-11T06:44:26.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:44:26.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:26.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_10 2024-09-11T06:44:26.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=deleted 2024-09-11T06:44:26.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:44:26.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=deleted 2024-09-11T06:44:26.410 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_image_id 2024-09-11T06:44:26.411 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_10 2024-09-11T06:44:26.411 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:44:26.411 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:44:26.412 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_10 2024-09-11T06:44:26.412 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-09-11T06:44:26.412 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:44:26.458 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_10: (2) No such file or directory 2024-09-11T06:44:26.463 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_image_id= 2024-09-11T06:44:26.463 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:44:26.463 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test deleted = deleted 2024-09-11T06:44:26.464 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:44:26.464 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:44:26.464 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_11 2024-09-11T06:44:26.464 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_present cluster1 mirror image_11 deleted 2024-09-11T06:44:26.464 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:44:26.464 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:26.464 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_11 2024-09-11T06:44:26.464 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=deleted 2024-09-11T06:44:26.464 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:44:26.464 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:44:26.464 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:44:26.464 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_11 2024-09-11T06:44:26.465 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:44:26.465 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:44:26.465 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_11 2024-09-11T06:44:26.465 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_11 2024-09-11T06:44:26.465 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:44:26.503 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_11: (2) No such file or directory 2024-09-11T06:44:26.506 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image_id= 2024-09-11T06:44:26.507 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T06:44:26.507 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.1 2024-09-11T06:44:26.608 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_present cluster1 mirror image_11 deleted '' 2024-09-11T06:44:26.608 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:44:26.608 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:26.608 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_11 2024-09-11T06:44:26.608 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=deleted 2024-09-11T06:44:26.608 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:44:26.609 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=deleted 2024-09-11T06:44:26.609 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_image_id 2024-09-11T06:44:26.609 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_11 2024-09-11T06:44:26.610 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:44:26.610 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:44:26.610 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_11 2024-09-11T06:44:26.610 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_11 2024-09-11T06:44:26.610 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:44:26.651 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_11: (2) No such file or directory 2024-09-11T06:44:26.656 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_image_id= 2024-09-11T06:44:26.656 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:44:26.656 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test deleted = deleted 2024-09-11T06:44:26.656 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:44:26.656 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:44:26.656 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_12 2024-09-11T06:44:26.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_present cluster1 mirror image_12 deleted 2024-09-11T06:44:26.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:44:26.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:26.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_12 2024-09-11T06:44:26.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=deleted 2024-09-11T06:44:26.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:44:26.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:44:26.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:44:26.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_12 2024-09-11T06:44:26.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:44:26.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:44:26.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_12 2024-09-11T06:44:26.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_12 2024-09-11T06:44:26.658 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:44:26.695 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_12: (2) No such file or directory 2024-09-11T06:44:26.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image_id= 2024-09-11T06:44:26.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T06:44:26.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.1 2024-09-11T06:44:26.801 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_present cluster1 mirror image_12 deleted '' 2024-09-11T06:44:26.801 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:44:26.801 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:26.804 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_12 2024-09-11T06:44:26.804 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=deleted 2024-09-11T06:44:26.804 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:44:26.804 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=deleted 2024-09-11T06:44:26.804 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_image_id 2024-09-11T06:44:26.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_12 2024-09-11T06:44:26.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:44:26.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:44:26.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_12 2024-09-11T06:44:26.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_12 2024-09-11T06:44:26.805 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:44:26.849 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_12: (2) No such file or directory 2024-09-11T06:44:26.853 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_image_id= 2024-09-11T06:44:26.854 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:44:26.854 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test deleted = deleted 2024-09-11T06:44:26.854 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:44:26.854 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:44:26.854 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_13 2024-09-11T06:44:26.854 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_present cluster1 mirror image_13 deleted 2024-09-11T06:44:26.854 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:44:26.854 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:26.854 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_13 2024-09-11T06:44:26.854 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=deleted 2024-09-11T06:44:26.854 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:44:26.854 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:44:26.854 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:44:26.855 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_13 2024-09-11T06:44:26.855 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:44:26.855 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:44:26.855 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_13 2024-09-11T06:44:26.855 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_13 2024-09-11T06:44:26.855 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:44:26.891 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_13: (2) No such file or directory 2024-09-11T06:44:26.895 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image_id= 2024-09-11T06:44:26.896 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T06:44:26.896 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.1 2024-09-11T06:44:26.997 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_present cluster1 mirror image_13 deleted '' 2024-09-11T06:44:26.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:44:26.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:26.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_13 2024-09-11T06:44:26.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=deleted 2024-09-11T06:44:26.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:44:26.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=deleted 2024-09-11T06:44:26.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_image_id 2024-09-11T06:44:26.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_13 2024-09-11T06:44:26.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:44:26.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:44:26.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_13 2024-09-11T06:44:26.999 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_13 2024-09-11T06:44:26.999 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:44:27.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_13: (2) No such file or directory 2024-09-11T06:44:27.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_image_id= 2024-09-11T06:44:27.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:44:27.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test deleted = deleted 2024-09-11T06:44:27.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:44:27.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:44:27.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_14 2024-09-11T06:44:27.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_present cluster1 mirror image_14 deleted 2024-09-11T06:44:27.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:44:27.048 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:27.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_14 2024-09-11T06:44:27.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=deleted 2024-09-11T06:44:27.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:44:27.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:44:27.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:44:27.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_14 2024-09-11T06:44:27.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:44:27.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:44:27.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_14 2024-09-11T06:44:27.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_14 2024-09-11T06:44:27.049 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:44:27.086 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_14: (2) No such file or directory 2024-09-11T06:44:27.090 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image_id= 2024-09-11T06:44:27.090 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T06:44:27.090 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.1 2024-09-11T06:44:27.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_present cluster1 mirror image_14 deleted '' 2024-09-11T06:44:27.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:44:27.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:27.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_14 2024-09-11T06:44:27.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=deleted 2024-09-11T06:44:27.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:44:27.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=deleted 2024-09-11T06:44:27.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_image_id 2024-09-11T06:44:27.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_14 2024-09-11T06:44:27.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:44:27.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:44:27.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_14 2024-09-11T06:44:27.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_14 2024-09-11T06:44:27.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:44:27.239 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_14: (2) No such file or directory 2024-09-11T06:44:27.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_image_id= 2024-09-11T06:44:27.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:44:27.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test deleted = deleted 2024-09-11T06:44:27.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:44:27.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:44:27.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_15 2024-09-11T06:44:27.243 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_present cluster1 mirror image_15 deleted 2024-09-11T06:44:27.244 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:44:27.244 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:27.244 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_15 2024-09-11T06:44:27.244 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=deleted 2024-09-11T06:44:27.244 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:44:27.244 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:44:27.244 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:44:27.244 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_15 2024-09-11T06:44:27.244 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:44:27.244 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:44:27.244 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_15 2024-09-11T06:44:27.245 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_15 2024-09-11T06:44:27.245 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:44:27.282 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_15: (2) No such file or directory 2024-09-11T06:44:27.286 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image_id= 2024-09-11T06:44:27.286 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T06:44:27.286 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.1 2024-09-11T06:44:27.387 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_present cluster1 mirror image_15 deleted '' 2024-09-11T06:44:27.387 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:44:27.388 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:27.388 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_15 2024-09-11T06:44:27.388 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=deleted 2024-09-11T06:44:27.388 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:44:27.388 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=deleted 2024-09-11T06:44:27.388 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_image_id 2024-09-11T06:44:27.388 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_15 2024-09-11T06:44:27.388 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:44:27.388 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:44:27.388 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_15 2024-09-11T06:44:27.389 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_15 2024-09-11T06:44:27.389 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:44:27.434 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_15: (2) No such file or directory 2024-09-11T06:44:27.439 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_image_id= 2024-09-11T06:44:27.439 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:44:27.439 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test deleted = deleted 2024-09-11T06:44:27.439 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:44:27.439 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:44:27.439 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_16 2024-09-11T06:44:27.439 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_present cluster1 mirror image_16 deleted 2024-09-11T06:44:27.439 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:44:27.439 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:27.440 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_16 2024-09-11T06:44:27.440 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=deleted 2024-09-11T06:44:27.440 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:44:27.440 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:44:27.440 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:44:27.440 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_16 2024-09-11T06:44:27.440 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:44:27.440 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:44:27.440 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_16 2024-09-11T06:44:27.440 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_16 2024-09-11T06:44:27.440 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:44:27.477 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_16: (2) No such file or directory 2024-09-11T06:44:27.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image_id= 2024-09-11T06:44:27.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T06:44:27.482 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.1 2024-09-11T06:44:27.583 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_present cluster1 mirror image_16 deleted '' 2024-09-11T06:44:27.583 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:44:27.583 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:27.583 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_16 2024-09-11T06:44:27.583 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=deleted 2024-09-11T06:44:27.583 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:44:27.583 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=deleted 2024-09-11T06:44:27.583 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_image_id 2024-09-11T06:44:27.584 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_16 2024-09-11T06:44:27.584 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:44:27.584 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:44:27.584 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_16 2024-09-11T06:44:27.585 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_16 2024-09-11T06:44:27.585 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:44:27.633 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_16: (2) No such file or directory 2024-09-11T06:44:27.638 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_image_id= 2024-09-11T06:44:27.638 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:44:27.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test deleted = deleted 2024-09-11T06:44:27.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:44:27.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:44:27.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_17 2024-09-11T06:44:27.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_present cluster1 mirror image_17 deleted 2024-09-11T06:44:27.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:44:27.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:27.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_17 2024-09-11T06:44:27.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=deleted 2024-09-11T06:44:27.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:44:27.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:44:27.639 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:44:27.640 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_17 2024-09-11T06:44:27.640 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:44:27.640 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:44:27.640 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_17 2024-09-11T06:44:27.640 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_17 2024-09-11T06:44:27.640 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:44:27.675 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_17: (2) No such file or directory 2024-09-11T06:44:27.679 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image_id= 2024-09-11T06:44:27.679 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T06:44:27.679 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.1 2024-09-11T06:44:27.781 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_present cluster1 mirror image_17 deleted '' 2024-09-11T06:44:27.781 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:44:27.781 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:27.781 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_17 2024-09-11T06:44:27.781 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=deleted 2024-09-11T06:44:27.781 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:44:27.781 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=deleted 2024-09-11T06:44:27.781 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_image_id 2024-09-11T06:44:27.782 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_17 2024-09-11T06:44:27.782 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:44:27.782 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:44:27.782 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_17 2024-09-11T06:44:27.783 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_17 2024-09-11T06:44:27.783 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:44:27.827 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_17: (2) No such file or directory 2024-09-11T06:44:27.832 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_image_id= 2024-09-11T06:44:27.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:44:27.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test deleted = deleted 2024-09-11T06:44:27.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:44:27.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:44:27.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_18 2024-09-11T06:44:27.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_present cluster1 mirror image_18 deleted 2024-09-11T06:44:27.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:44:27.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:27.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_18 2024-09-11T06:44:27.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=deleted 2024-09-11T06:44:27.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:44:27.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:44:27.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:44:27.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_18 2024-09-11T06:44:27.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:44:27.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:44:27.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_18 2024-09-11T06:44:27.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_18 2024-09-11T06:44:27.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:44:27.871 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_18: (2) No such file or directory 2024-09-11T06:44:27.875 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image_id= 2024-09-11T06:44:27.875 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T06:44:27.876 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.1 2024-09-11T06:44:27.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_present cluster1 mirror image_18 deleted '' 2024-09-11T06:44:27.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:44:27.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:27.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_18 2024-09-11T06:44:27.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=deleted 2024-09-11T06:44:27.979 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:44:27.979 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=deleted 2024-09-11T06:44:27.979 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_image_id 2024-09-11T06:44:27.979 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_18 2024-09-11T06:44:27.979 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:44:27.979 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:44:27.979 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_18 2024-09-11T06:44:27.979 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_18 2024-09-11T06:44:27.979 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:44:28.023 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_18: (2) No such file or directory 2024-09-11T06:44:28.028 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_image_id= 2024-09-11T06:44:28.029 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:44:28.029 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test deleted = deleted 2024-09-11T06:44:28.029 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:44:28.029 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:44:28.029 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_19 2024-09-11T06:44:28.029 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_present cluster1 mirror image_19 deleted 2024-09-11T06:44:28.032 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:44:28.032 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:28.032 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_19 2024-09-11T06:44:28.032 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=deleted 2024-09-11T06:44:28.032 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:44:28.032 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:44:28.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:44:28.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_19 2024-09-11T06:44:28.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:44:28.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:44:28.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_19 2024-09-11T06:44:28.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_19 2024-09-11T06:44:28.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:44:28.067 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_19: (2) No such file or directory 2024-09-11T06:44:28.071 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image_id= 2024-09-11T06:44:28.071 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T06:44:28.071 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.1 2024-09-11T06:44:28.172 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_present cluster1 mirror image_19 deleted '' 2024-09-11T06:44:28.172 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:44:28.172 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:28.173 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_19 2024-09-11T06:44:28.173 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=deleted 2024-09-11T06:44:28.173 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:44:28.173 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=deleted 2024-09-11T06:44:28.173 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_image_id 2024-09-11T06:44:28.174 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_19 2024-09-11T06:44:28.174 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:44:28.174 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:44:28.174 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_19 2024-09-11T06:44:28.174 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_19 2024-09-11T06:44:28.174 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:44:28.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_19: (2) No such file or directory 2024-09-11T06:44:28.224 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_image_id= 2024-09-11T06:44:28.224 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:44:28.224 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test deleted = deleted 2024-09-11T06:44:28.224 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:44:28.225 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:44:28.225 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_20 2024-09-11T06:44:28.225 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_present cluster1 mirror image_20 deleted 2024-09-11T06:44:28.225 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:44:28.225 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:28.225 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_20 2024-09-11T06:44:28.225 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=deleted 2024-09-11T06:44:28.225 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:44:28.225 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:44:28.225 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:44:28.225 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_20 2024-09-11T06:44:28.225 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:44:28.225 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:44:28.226 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_20 2024-09-11T06:44:28.226 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_20 2024-09-11T06:44:28.226 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:44:28.262 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_20: (2) No such file or directory 2024-09-11T06:44:28.266 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image_id= 2024-09-11T06:44:28.266 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T06:44:28.266 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.1 2024-09-11T06:44:28.368 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_present cluster1 mirror image_20 deleted '' 2024-09-11T06:44:28.368 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:44:28.368 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:28.368 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_20 2024-09-11T06:44:28.368 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=deleted 2024-09-11T06:44:28.368 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:44:28.368 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=deleted 2024-09-11T06:44:28.369 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_image_id 2024-09-11T06:44:28.369 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_20 2024-09-11T06:44:28.369 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:44:28.369 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:44:28.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_20 2024-09-11T06:44:28.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_20 2024-09-11T06:44:28.370 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:44:28.413 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_20: (2) No such file or directory 2024-09-11T06:44:28.418 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_image_id= 2024-09-11T06:44:28.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:44:28.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test deleted = deleted 2024-09-11T06:44:28.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:44:28.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:44:28.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_21 2024-09-11T06:44:28.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_present cluster1 mirror image_21 deleted 2024-09-11T06:44:28.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:44:28.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:28.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_21 2024-09-11T06:44:28.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=deleted 2024-09-11T06:44:28.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:44:28.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:44:28.420 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:44:28.420 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_21 2024-09-11T06:44:28.420 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:44:28.420 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:44:28.420 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_21 2024-09-11T06:44:28.420 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_21 2024-09-11T06:44:28.420 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:44:28.467 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_21: (2) No such file or directory 2024-09-11T06:44:28.471 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image_id= 2024-09-11T06:44:28.471 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T06:44:28.471 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.1 2024-09-11T06:44:28.572 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_present cluster1 mirror image_21 deleted '' 2024-09-11T06:44:28.572 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:44:28.573 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:28.573 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_21 2024-09-11T06:44:28.573 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=deleted 2024-09-11T06:44:28.573 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:44:28.573 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=deleted 2024-09-11T06:44:28.573 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_image_id 2024-09-11T06:44:28.573 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_21 2024-09-11T06:44:28.573 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:44:28.573 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:44:28.573 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_21 2024-09-11T06:44:28.574 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_21 2024-09-11T06:44:28.574 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:44:28.617 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_21: (2) No such file or directory 2024-09-11T06:44:28.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_image_id= 2024-09-11T06:44:28.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:44:28.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test deleted = deleted 2024-09-11T06:44:28.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:44:28.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:44:28.622 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_22 2024-09-11T06:44:28.623 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_present cluster1 mirror image_22 deleted 2024-09-11T06:44:28.623 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:44:28.623 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:28.623 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_22 2024-09-11T06:44:28.623 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=deleted 2024-09-11T06:44:28.623 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:44:28.623 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:44:28.623 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:44:28.623 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_22 2024-09-11T06:44:28.623 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:44:28.623 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:44:28.623 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_22 2024-09-11T06:44:28.623 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_22 2024-09-11T06:44:28.623 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:44:28.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_22: (2) No such file or directory 2024-09-11T06:44:28.670 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image_id= 2024-09-11T06:44:28.670 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T06:44:28.670 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.1 2024-09-11T06:44:28.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_present cluster1 mirror image_22 deleted '' 2024-09-11T06:44:28.772 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:44:28.772 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:28.772 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_22 2024-09-11T06:44:28.772 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=deleted 2024-09-11T06:44:28.772 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:44:28.772 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=deleted 2024-09-11T06:44:28.772 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_image_id 2024-09-11T06:44:28.773 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_22 2024-09-11T06:44:28.773 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:44:28.773 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:44:28.773 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_22 2024-09-11T06:44:28.773 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_22 2024-09-11T06:44:28.773 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:44:28.819 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_22: (2) No such file or directory 2024-09-11T06:44:28.824 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_image_id= 2024-09-11T06:44:28.824 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:44:28.824 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test deleted = deleted 2024-09-11T06:44:28.824 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:44:28.824 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:44:28.824 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_23 2024-09-11T06:44:28.824 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_present cluster1 mirror image_23 deleted 2024-09-11T06:44:28.824 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:44:28.825 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:28.825 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_23 2024-09-11T06:44:28.825 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=deleted 2024-09-11T06:44:28.825 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:44:28.825 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:44:28.825 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:44:28.825 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_23 2024-09-11T06:44:28.825 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:44:28.825 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:44:28.825 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_23 2024-09-11T06:44:28.825 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_23 2024-09-11T06:44:28.825 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:44:28.864 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_23: (2) No such file or directory 2024-09-11T06:44:28.869 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image_id= 2024-09-11T06:44:28.869 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T06:44:28.869 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.1 2024-09-11T06:44:28.970 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_present cluster1 mirror image_23 deleted '' 2024-09-11T06:44:28.970 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:44:28.970 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:28.970 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_23 2024-09-11T06:44:28.970 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=deleted 2024-09-11T06:44:28.970 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:44:28.971 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=deleted 2024-09-11T06:44:28.971 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_image_id 2024-09-11T06:44:28.972 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_23 2024-09-11T06:44:28.972 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:44:28.972 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:44:28.972 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_23 2024-09-11T06:44:28.972 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_23 2024-09-11T06:44:28.972 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:44:29.018 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_23: (2) No such file or directory 2024-09-11T06:44:29.024 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_image_id= 2024-09-11T06:44:29.024 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:44:29.024 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test deleted = deleted 2024-09-11T06:44:29.024 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:44:29.024 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:44:29.024 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_24 2024-09-11T06:44:29.024 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_present cluster1 mirror image_24 deleted 2024-09-11T06:44:29.024 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:44:29.024 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:29.024 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_24 2024-09-11T06:44:29.024 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=deleted 2024-09-11T06:44:29.024 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:44:29.025 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:44:29.025 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:44:29.025 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_24 2024-09-11T06:44:29.025 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:44:29.025 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:44:29.025 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_24 2024-09-11T06:44:29.025 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_24 2024-09-11T06:44:29.025 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:44:29.063 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_24: (2) No such file or directory 2024-09-11T06:44:29.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image_id= 2024-09-11T06:44:29.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T06:44:29.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.1 2024-09-11T06:44:29.168 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_present cluster1 mirror image_24 deleted '' 2024-09-11T06:44:29.169 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:44:29.169 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:29.169 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_24 2024-09-11T06:44:29.169 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=deleted 2024-09-11T06:44:29.169 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:44:29.169 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=deleted 2024-09-11T06:44:29.169 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_image_id 2024-09-11T06:44:29.169 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_24 2024-09-11T06:44:29.169 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:44:29.169 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:44:29.169 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_24 2024-09-11T06:44:29.170 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_24 2024-09-11T06:44:29.170 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:44:29.214 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_24: (2) No such file or directory 2024-09-11T06:44:29.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_image_id= 2024-09-11T06:44:29.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:44:29.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test deleted = deleted 2024-09-11T06:44:29.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:44:29.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:44:29.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_25 2024-09-11T06:44:29.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_present cluster1 mirror image_25 deleted 2024-09-11T06:44:29.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:44:29.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:29.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_25 2024-09-11T06:44:29.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=deleted 2024-09-11T06:44:29.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:44:29.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:44:29.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:44:29.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_25 2024-09-11T06:44:29.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:44:29.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:44:29.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_25 2024-09-11T06:44:29.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_25 2024-09-11T06:44:29.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:44:29.259 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_25: (2) No such file or directory 2024-09-11T06:44:29.262 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image_id= 2024-09-11T06:44:29.262 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T06:44:29.262 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.1 2024-09-11T06:44:29.364 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_present cluster1 mirror image_25 deleted '' 2024-09-11T06:44:29.364 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:44:29.364 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:29.364 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_25 2024-09-11T06:44:29.365 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=deleted 2024-09-11T06:44:29.365 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:44:29.365 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=deleted 2024-09-11T06:44:29.365 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_image_id 2024-09-11T06:44:29.365 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_25 2024-09-11T06:44:29.365 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:44:29.366 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:44:29.366 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_25 2024-09-11T06:44:29.366 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_25 2024-09-11T06:44:29.366 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:44:29.411 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_25: (2) No such file or directory 2024-09-11T06:44:29.416 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_image_id= 2024-09-11T06:44:29.416 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:44:29.416 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test deleted = deleted 2024-09-11T06:44:29.416 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:44:29.416 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:44:29.416 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_26 2024-09-11T06:44:29.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_present cluster1 mirror image_26 deleted 2024-09-11T06:44:29.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:44:29.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:29.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_26 2024-09-11T06:44:29.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=deleted 2024-09-11T06:44:29.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:44:29.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:44:29.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:44:29.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_26 2024-09-11T06:44:29.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:44:29.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:44:29.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_26 2024-09-11T06:44:29.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_26 2024-09-11T06:44:29.417 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:44:29.454 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_26: (2) No such file or directory 2024-09-11T06:44:29.457 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image_id= 2024-09-11T06:44:29.458 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T06:44:29.458 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.1 2024-09-11T06:44:29.560 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_present cluster1 mirror image_26 deleted '' 2024-09-11T06:44:29.560 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:44:29.561 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:29.561 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_26 2024-09-11T06:44:29.561 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=deleted 2024-09-11T06:44:29.561 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:44:29.561 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=deleted 2024-09-11T06:44:29.561 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_image_id 2024-09-11T06:44:29.561 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_26 2024-09-11T06:44:29.561 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:44:29.562 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:44:29.562 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_26 2024-09-11T06:44:29.563 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_26 2024-09-11T06:44:29.563 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:44:29.605 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_26: (2) No such file or directory 2024-09-11T06:44:29.610 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_image_id= 2024-09-11T06:44:29.611 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:44:29.611 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test deleted = deleted 2024-09-11T06:44:29.611 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:44:29.611 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:44:29.611 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_27 2024-09-11T06:44:29.611 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_present cluster1 mirror image_27 deleted 2024-09-11T06:44:29.611 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:44:29.611 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:29.611 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_27 2024-09-11T06:44:29.611 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=deleted 2024-09-11T06:44:29.611 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:44:29.611 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:44:29.611 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:44:29.612 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_27 2024-09-11T06:44:29.612 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:44:29.612 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:44:29.612 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_27 2024-09-11T06:44:29.612 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_27 2024-09-11T06:44:29.612 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:44:29.648 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_27: (2) No such file or directory 2024-09-11T06:44:29.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image_id= 2024-09-11T06:44:29.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T06:44:29.653 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.1 2024-09-11T06:44:29.754 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_present cluster1 mirror image_27 deleted '' 2024-09-11T06:44:29.754 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:44:29.754 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:29.754 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_27 2024-09-11T06:44:29.754 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=deleted 2024-09-11T06:44:29.754 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:44:29.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=deleted 2024-09-11T06:44:29.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_image_id 2024-09-11T06:44:29.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_27 2024-09-11T06:44:29.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:44:29.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:44:29.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_27 2024-09-11T06:44:29.756 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_27 2024-09-11T06:44:29.756 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:44:29.801 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_27: (2) No such file or directory 2024-09-11T06:44:29.807 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_image_id= 2024-09-11T06:44:29.807 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:44:29.807 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test deleted = deleted 2024-09-11T06:44:29.807 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:44:29.808 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:44:29.808 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_28 2024-09-11T06:44:29.808 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_present cluster1 mirror image_28 deleted 2024-09-11T06:44:29.808 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:44:29.808 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:29.808 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_28 2024-09-11T06:44:29.808 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=deleted 2024-09-11T06:44:29.808 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:44:29.808 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:44:29.808 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:44:29.808 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_28 2024-09-11T06:44:29.808 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:44:29.808 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:44:29.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_28 2024-09-11T06:44:29.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_28 2024-09-11T06:44:29.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:44:29.846 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_28: (2) No such file or directory 2024-09-11T06:44:29.850 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image_id= 2024-09-11T06:44:29.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T06:44:29.851 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.1 2024-09-11T06:44:29.953 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_present cluster1 mirror image_28 deleted '' 2024-09-11T06:44:29.953 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:44:29.953 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:29.953 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_28 2024-09-11T06:44:29.953 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=deleted 2024-09-11T06:44:29.953 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:44:29.953 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=deleted 2024-09-11T06:44:29.953 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_image_id 2024-09-11T06:44:29.953 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_28 2024-09-11T06:44:29.954 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:44:29.954 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:44:29.954 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_28 2024-09-11T06:44:29.954 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_28 2024-09-11T06:44:29.955 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:44:29.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_28: (2) No such file or directory 2024-09-11T06:44:30.004 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_image_id= 2024-09-11T06:44:30.004 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:44:30.004 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test deleted = deleted 2024-09-11T06:44:30.004 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:44:30.004 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:44:30.004 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_29 2024-09-11T06:44:30.005 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_present cluster1 mirror image_29 deleted 2024-09-11T06:44:30.005 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:44:30.005 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:30.005 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_29 2024-09-11T06:44:30.005 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=deleted 2024-09-11T06:44:30.005 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:44:30.005 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:44:30.005 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:44:30.005 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_29 2024-09-11T06:44:30.005 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:44:30.005 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:44:30.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_29 2024-09-11T06:44:30.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_29 2024-09-11T06:44:30.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:44:30.041 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_29: (2) No such file or directory 2024-09-11T06:44:30.045 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image_id= 2024-09-11T06:44:30.045 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T06:44:30.045 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.1 2024-09-11T06:44:30.147 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_present cluster1 mirror image_29 deleted '' 2024-09-11T06:44:30.147 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:44:30.147 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:30.147 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_29 2024-09-11T06:44:30.147 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=deleted 2024-09-11T06:44:30.147 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:44:30.147 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=deleted 2024-09-11T06:44:30.147 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_image_id 2024-09-11T06:44:30.148 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_29 2024-09-11T06:44:30.148 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:44:30.148 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:44:30.149 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_29 2024-09-11T06:44:30.149 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_29 2024-09-11T06:44:30.149 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:44:30.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_29: (2) No such file or directory 2024-09-11T06:44:30.197 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_image_id= 2024-09-11T06:44:30.197 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:44:30.197 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test deleted = deleted 2024-09-11T06:44:30.197 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:44:30.197 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:44:30.197 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_30 2024-09-11T06:44:30.197 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_present cluster1 mirror image_30 deleted 2024-09-11T06:44:30.197 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:44:30.197 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:30.197 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_30 2024-09-11T06:44:30.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=deleted 2024-09-11T06:44:30.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:44:30.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:44:30.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:44:30.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_30 2024-09-11T06:44:30.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:44:30.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:44:30.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_30 2024-09-11T06:44:30.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_30 2024-09-11T06:44:30.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:44:30.236 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_30: (2) No such file or directory 2024-09-11T06:44:30.240 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image_id= 2024-09-11T06:44:30.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T06:44:30.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.1 2024-09-11T06:44:30.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_present cluster1 mirror image_30 deleted '' 2024-09-11T06:44:30.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:44:30.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:30.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_30 2024-09-11T06:44:30.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=deleted 2024-09-11T06:44:30.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:44:30.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=deleted 2024-09-11T06:44:30.343 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_image_id 2024-09-11T06:44:30.343 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_30 2024-09-11T06:44:30.343 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:44:30.343 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:44:30.343 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_30 2024-09-11T06:44:30.343 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_30 2024-09-11T06:44:30.343 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:44:30.388 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_30: (2) No such file or directory 2024-09-11T06:44:30.392 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_image_id= 2024-09-11T06:44:30.392 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:44:30.393 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test deleted = deleted 2024-09-11T06:44:30.393 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:44:30.393 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:44:30.393 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_31 2024-09-11T06:44:30.393 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_present cluster1 mirror image_31 deleted 2024-09-11T06:44:30.393 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:44:30.393 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:30.393 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_31 2024-09-11T06:44:30.393 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=deleted 2024-09-11T06:44:30.393 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:44:30.393 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:44:30.393 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:44:30.394 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_31 2024-09-11T06:44:30.394 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:44:30.394 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:44:30.394 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_31 2024-09-11T06:44:30.394 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_31 2024-09-11T06:44:30.394 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:44:30.431 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_31: (2) No such file or directory 2024-09-11T06:44:30.435 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image_id= 2024-09-11T06:44:30.435 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T06:44:30.435 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.1 2024-09-11T06:44:30.538 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_present cluster1 mirror image_31 deleted '' 2024-09-11T06:44:30.538 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:44:30.538 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:30.538 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_31 2024-09-11T06:44:30.538 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=deleted 2024-09-11T06:44:30.538 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:44:30.538 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=deleted 2024-09-11T06:44:30.538 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_image_id 2024-09-11T06:44:30.538 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_31 2024-09-11T06:44:30.539 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:44:30.539 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:44:30.539 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_31 2024-09-11T06:44:30.539 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_31 2024-09-11T06:44:30.539 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:44:30.585 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_31: (2) No such file or directory 2024-09-11T06:44:30.590 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_image_id= 2024-09-11T06:44:30.590 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:44:30.590 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test deleted = deleted 2024-09-11T06:44:30.590 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:44:30.590 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:44:30.590 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_32 2024-09-11T06:44:30.590 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_present cluster1 mirror image_32 deleted 2024-09-11T06:44:30.590 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:44:30.591 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:30.591 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_32 2024-09-11T06:44:30.591 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=deleted 2024-09-11T06:44:30.591 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:44:30.591 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:44:30.591 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:44:30.591 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_32 2024-09-11T06:44:30.591 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:44:30.591 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:44:30.591 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_32 2024-09-11T06:44:30.591 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_32 2024-09-11T06:44:30.591 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:44:30.628 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_32: (2) No such file or directory 2024-09-11T06:44:30.632 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image_id= 2024-09-11T06:44:30.632 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T06:44:30.632 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.1 2024-09-11T06:44:30.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_present cluster1 mirror image_32 deleted '' 2024-09-11T06:44:30.733 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:44:30.734 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:30.734 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_32 2024-09-11T06:44:30.734 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=deleted 2024-09-11T06:44:30.734 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:44:30.734 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=deleted 2024-09-11T06:44:30.734 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_image_id 2024-09-11T06:44:30.735 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_32 2024-09-11T06:44:30.735 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:44:30.735 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:44:30.735 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_32 2024-09-11T06:44:30.735 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_32 2024-09-11T06:44:30.735 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:44:30.780 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_32: (2) No such file or directory 2024-09-11T06:44:30.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_image_id= 2024-09-11T06:44:30.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:44:30.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test deleted = deleted 2024-09-11T06:44:30.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:44:30.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:44:30.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_33 2024-09-11T06:44:30.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_present cluster1 mirror image_33 deleted 2024-09-11T06:44:30.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:44:30.786 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:30.787 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_33 2024-09-11T06:44:30.787 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=deleted 2024-09-11T06:44:30.787 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:44:30.787 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:44:30.787 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:44:30.787 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_33 2024-09-11T06:44:30.787 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:44:30.787 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:44:30.787 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_33 2024-09-11T06:44:30.787 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_33 2024-09-11T06:44:30.788 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:44:30.824 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_33: (2) No such file or directory 2024-09-11T06:44:30.828 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image_id= 2024-09-11T06:44:30.828 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T06:44:30.828 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.1 2024-09-11T06:44:30.930 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_present cluster1 mirror image_33 deleted '' 2024-09-11T06:44:30.930 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:44:30.930 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:30.930 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_33 2024-09-11T06:44:30.930 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=deleted 2024-09-11T06:44:30.930 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:44:30.930 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=deleted 2024-09-11T06:44:30.930 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_image_id 2024-09-11T06:44:30.930 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_33 2024-09-11T06:44:30.931 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:44:30.931 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:44:30.931 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_33 2024-09-11T06:44:30.931 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_33 2024-09-11T06:44:30.931 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:44:30.976 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_33: (2) No such file or directory 2024-09-11T06:44:30.981 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_image_id= 2024-09-11T06:44:30.981 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:44:30.981 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test deleted = deleted 2024-09-11T06:44:30.981 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:44:30.982 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:44:30.982 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_34 2024-09-11T06:44:30.982 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_present cluster1 mirror image_34 deleted 2024-09-11T06:44:30.982 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:44:30.982 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:30.982 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_34 2024-09-11T06:44:30.982 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=deleted 2024-09-11T06:44:30.982 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:44:30.982 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:44:30.982 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:44:30.982 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_34 2024-09-11T06:44:30.982 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:44:30.982 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:44:30.983 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_34 2024-09-11T06:44:30.983 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_34 2024-09-11T06:44:30.983 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:44:31.019 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_34: (2) No such file or directory 2024-09-11T06:44:31.023 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image_id= 2024-09-11T06:44:31.023 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T06:44:31.023 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.1 2024-09-11T06:44:31.124 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_present cluster1 mirror image_34 deleted '' 2024-09-11T06:44:31.124 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:44:31.124 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:31.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_34 2024-09-11T06:44:31.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=deleted 2024-09-11T06:44:31.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:44:31.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=deleted 2024-09-11T06:44:31.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_image_id 2024-09-11T06:44:31.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_34 2024-09-11T06:44:31.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:44:31.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:44:31.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_34 2024-09-11T06:44:31.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_34 2024-09-11T06:44:31.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:44:31.171 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_34: (2) No such file or directory 2024-09-11T06:44:31.177 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_image_id= 2024-09-11T06:44:31.178 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:44:31.178 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test deleted = deleted 2024-09-11T06:44:31.178 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:44:31.178 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:44:31.178 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_35 2024-09-11T06:44:31.178 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_present cluster1 mirror image_35 deleted 2024-09-11T06:44:31.178 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:44:31.178 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:31.178 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_35 2024-09-11T06:44:31.178 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=deleted 2024-09-11T06:44:31.178 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:44:31.178 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:44:31.178 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:44:31.178 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_35 2024-09-11T06:44:31.179 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:44:31.179 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:44:31.179 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_35 2024-09-11T06:44:31.179 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_35 2024-09-11T06:44:31.179 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:44:31.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_35: (2) No such file or directory 2024-09-11T06:44:31.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image_id= 2024-09-11T06:44:31.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T06:44:31.221 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.1 2024-09-11T06:44:31.322 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_present cluster1 mirror image_35 deleted '' 2024-09-11T06:44:31.322 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:44:31.322 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:31.322 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_35 2024-09-11T06:44:31.322 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=deleted 2024-09-11T06:44:31.322 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:44:31.322 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=deleted 2024-09-11T06:44:31.323 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_image_id 2024-09-11T06:44:31.323 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_35 2024-09-11T06:44:31.323 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:44:31.323 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:44:31.323 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_35 2024-09-11T06:44:31.324 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:44:31.324 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_35 2024-09-11T06:44:31.371 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_35: (2) No such file or directory 2024-09-11T06:44:31.376 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_image_id= 2024-09-11T06:44:31.376 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:44:31.376 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test deleted = deleted 2024-09-11T06:44:31.376 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:44:31.376 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:44:31.376 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_36 2024-09-11T06:44:31.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_present cluster1 mirror image_36 deleted 2024-09-11T06:44:31.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:44:31.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:31.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_36 2024-09-11T06:44:31.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=deleted 2024-09-11T06:44:31.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:44:31.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:44:31.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:44:31.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_36 2024-09-11T06:44:31.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:44:31.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:44:31.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_36 2024-09-11T06:44:31.377 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_36 2024-09-11T06:44:31.378 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:44:31.416 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_36: (2) No such file or directory 2024-09-11T06:44:31.420 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image_id= 2024-09-11T06:44:31.420 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T06:44:31.420 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.1 2024-09-11T06:44:31.522 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_present cluster1 mirror image_36 deleted '' 2024-09-11T06:44:31.522 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:44:31.522 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:31.523 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_36 2024-09-11T06:44:31.523 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=deleted 2024-09-11T06:44:31.523 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:44:31.523 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=deleted 2024-09-11T06:44:31.523 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_image_id 2024-09-11T06:44:31.523 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_36 2024-09-11T06:44:31.523 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:44:31.523 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:44:31.523 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_36 2024-09-11T06:44:31.524 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_36 2024-09-11T06:44:31.524 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:44:31.567 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_36: (2) No such file or directory 2024-09-11T06:44:31.574 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_image_id= 2024-09-11T06:44:31.574 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:44:31.574 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test deleted = deleted 2024-09-11T06:44:31.574 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:44:31.574 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:44:31.574 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_37 2024-09-11T06:44:31.574 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_present cluster1 mirror image_37 deleted 2024-09-11T06:44:31.574 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:44:31.574 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:31.575 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_37 2024-09-11T06:44:31.575 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=deleted 2024-09-11T06:44:31.575 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:44:31.575 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:44:31.575 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:44:31.575 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_37 2024-09-11T06:44:31.575 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:44:31.575 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:44:31.575 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_37 2024-09-11T06:44:31.575 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_37 2024-09-11T06:44:31.575 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:44:31.613 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_37: (2) No such file or directory 2024-09-11T06:44:31.617 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image_id= 2024-09-11T06:44:31.617 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T06:44:31.617 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.1 2024-09-11T06:44:31.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_present cluster1 mirror image_37 deleted '' 2024-09-11T06:44:31.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:44:31.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:31.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_37 2024-09-11T06:44:31.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=deleted 2024-09-11T06:44:31.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:44:31.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=deleted 2024-09-11T06:44:31.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_image_id 2024-09-11T06:44:31.719 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_37 2024-09-11T06:44:31.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:44:31.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:44:31.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_37 2024-09-11T06:44:31.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_37 2024-09-11T06:44:31.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:44:31.764 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_37: (2) No such file or directory 2024-09-11T06:44:31.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_image_id= 2024-09-11T06:44:31.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:44:31.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test deleted = deleted 2024-09-11T06:44:31.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:44:31.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:44:31.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_38 2024-09-11T06:44:31.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_present cluster1 mirror image_38 deleted 2024-09-11T06:44:31.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:44:31.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:31.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_38 2024-09-11T06:44:31.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=deleted 2024-09-11T06:44:31.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:44:31.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:44:31.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:44:31.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_38 2024-09-11T06:44:31.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:44:31.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:44:31.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_38 2024-09-11T06:44:31.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_38 2024-09-11T06:44:31.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:44:31.808 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_38: (2) No such file or directory 2024-09-11T06:44:31.812 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image_id= 2024-09-11T06:44:31.812 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T06:44:31.812 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.1 2024-09-11T06:44:31.913 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_present cluster1 mirror image_38 deleted '' 2024-09-11T06:44:31.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:44:31.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:31.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_38 2024-09-11T06:44:31.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=deleted 2024-09-11T06:44:31.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:44:31.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=deleted 2024-09-11T06:44:31.914 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_image_id 2024-09-11T06:44:31.915 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_38 2024-09-11T06:44:31.915 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:44:31.915 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:44:31.915 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_38 2024-09-11T06:44:31.915 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_38 2024-09-11T06:44:31.915 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:44:31.958 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_38: (2) No such file or directory 2024-09-11T06:44:31.964 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_image_id= 2024-09-11T06:44:31.964 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:44:31.964 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test deleted = deleted 2024-09-11T06:44:31.964 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:44:31.964 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:44:31.964 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_39 2024-09-11T06:44:31.964 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_present cluster1 mirror image_39 deleted 2024-09-11T06:44:31.964 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:44:31.964 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:31.964 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_39 2024-09-11T06:44:31.964 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=deleted 2024-09-11T06:44:31.965 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:44:31.965 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:44:31.965 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:44:31.965 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_39 2024-09-11T06:44:31.965 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:44:31.965 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:44:31.965 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_39 2024-09-11T06:44:31.965 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_39 2024-09-11T06:44:31.965 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:44:32.003 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_39: (2) No such file or directory 2024-09-11T06:44:32.007 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image_id= 2024-09-11T06:44:32.008 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T06:44:32.008 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.1 2024-09-11T06:44:32.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_present cluster1 mirror image_39 deleted '' 2024-09-11T06:44:32.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:44:32.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:32.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_39 2024-09-11T06:44:32.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=deleted 2024-09-11T06:44:32.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:44:32.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=deleted 2024-09-11T06:44:32.109 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_image_id 2024-09-11T06:44:32.110 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_39 2024-09-11T06:44:32.110 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:44:32.110 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:44:32.110 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_39 2024-09-11T06:44:32.110 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_39 2024-09-11T06:44:32.110 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:44:32.157 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_39: (2) No such file or directory 2024-09-11T06:44:32.162 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_image_id= 2024-09-11T06:44:32.162 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:44:32.162 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test deleted = deleted 2024-09-11T06:44:32.162 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:44:32.162 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:44:32.162 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_40 2024-09-11T06:44:32.162 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_present cluster1 mirror image_40 deleted 2024-09-11T06:44:32.162 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:44:32.163 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:32.163 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_40 2024-09-11T06:44:32.163 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=deleted 2024-09-11T06:44:32.163 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:44:32.163 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:44:32.163 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:44:32.163 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_40 2024-09-11T06:44:32.163 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:44:32.163 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:44:32.163 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_40 2024-09-11T06:44:32.163 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_40 2024-09-11T06:44:32.163 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:44:32.201 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_40: (2) No such file or directory 2024-09-11T06:44:32.205 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image_id= 2024-09-11T06:44:32.205 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T06:44:32.205 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.1 2024-09-11T06:44:32.307 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_present cluster1 mirror image_40 deleted '' 2024-09-11T06:44:32.307 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:44:32.307 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:32.307 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_40 2024-09-11T06:44:32.307 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=deleted 2024-09-11T06:44:32.307 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:44:32.307 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=deleted 2024-09-11T06:44:32.307 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_image_id 2024-09-11T06:44:32.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_40 2024-09-11T06:44:32.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:44:32.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:44:32.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_40 2024-09-11T06:44:32.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_40 2024-09-11T06:44:32.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:44:32.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_40: (2) No such file or directory 2024-09-11T06:44:32.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_image_id= 2024-09-11T06:44:32.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:44:32.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test deleted = deleted 2024-09-11T06:44:32.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:44:32.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:44:32.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_41 2024-09-11T06:44:32.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_present cluster1 mirror image_41 deleted 2024-09-11T06:44:32.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:44:32.360 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:32.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_41 2024-09-11T06:44:32.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=deleted 2024-09-11T06:44:32.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:44:32.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:44:32.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:44:32.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_41 2024-09-11T06:44:32.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:44:32.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:44:32.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_41 2024-09-11T06:44:32.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_41 2024-09-11T06:44:32.361 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:44:32.400 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_41: (2) No such file or directory 2024-09-11T06:44:32.404 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image_id= 2024-09-11T06:44:32.405 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T06:44:32.405 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.1 2024-09-11T06:44:32.507 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_present cluster1 mirror image_41 deleted '' 2024-09-11T06:44:32.508 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:44:32.508 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:32.508 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_41 2024-09-11T06:44:32.508 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=deleted 2024-09-11T06:44:32.508 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:44:32.508 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=deleted 2024-09-11T06:44:32.509 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_image_id 2024-09-11T06:44:32.509 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_41 2024-09-11T06:44:32.509 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:44:32.509 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:44:32.509 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_41 2024-09-11T06:44:32.509 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_41 2024-09-11T06:44:32.509 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:44:32.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_41: (2) No such file or directory 2024-09-11T06:44:32.558 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_image_id= 2024-09-11T06:44:32.558 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:44:32.558 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test deleted = deleted 2024-09-11T06:44:32.558 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:44:32.558 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:44:32.558 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_42 2024-09-11T06:44:32.559 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_present cluster1 mirror image_42 deleted 2024-09-11T06:44:32.559 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:44:32.559 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:32.559 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_42 2024-09-11T06:44:32.559 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=deleted 2024-09-11T06:44:32.559 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:44:32.559 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:44:32.559 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:44:32.559 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_42 2024-09-11T06:44:32.559 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:44:32.559 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:44:32.559 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_42 2024-09-11T06:44:32.559 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_42 2024-09-11T06:44:32.560 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:44:32.597 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_42: (2) No such file or directory 2024-09-11T06:44:32.602 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image_id= 2024-09-11T06:44:32.602 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T06:44:32.602 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.1 2024-09-11T06:44:32.703 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_present cluster1 mirror image_42 deleted '' 2024-09-11T06:44:32.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:44:32.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:32.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_42 2024-09-11T06:44:32.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=deleted 2024-09-11T06:44:32.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:44:32.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=deleted 2024-09-11T06:44:32.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_image_id 2024-09-11T06:44:32.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_42 2024-09-11T06:44:32.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:44:32.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:44:32.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_42 2024-09-11T06:44:32.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_42 2024-09-11T06:44:32.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:44:32.755 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_42: (2) No such file or directory 2024-09-11T06:44:32.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_image_id= 2024-09-11T06:44:32.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:44:32.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test deleted = deleted 2024-09-11T06:44:32.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:44:32.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:44:32.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_43 2024-09-11T06:44:32.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_present cluster1 mirror image_43 deleted 2024-09-11T06:44:32.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:44:32.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:32.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_43 2024-09-11T06:44:32.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=deleted 2024-09-11T06:44:32.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:44:32.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:44:32.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:44:32.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_43 2024-09-11T06:44:32.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:44:32.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:44:32.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_43 2024-09-11T06:44:32.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_43 2024-09-11T06:44:32.762 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:44:32.799 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_43: (2) No such file or directory 2024-09-11T06:44:32.803 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image_id= 2024-09-11T06:44:32.803 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T06:44:32.803 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.1 2024-09-11T06:44:32.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_present cluster1 mirror image_43 deleted '' 2024-09-11T06:44:32.905 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:44:32.905 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:32.905 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_43 2024-09-11T06:44:32.905 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=deleted 2024-09-11T06:44:32.905 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:44:32.905 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=deleted 2024-09-11T06:44:32.905 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_image_id 2024-09-11T06:44:32.906 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_43 2024-09-11T06:44:32.906 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:44:32.906 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:44:32.906 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_43 2024-09-11T06:44:32.906 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_43 2024-09-11T06:44:32.906 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:44:32.952 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_43: (2) No such file or directory 2024-09-11T06:44:32.957 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_image_id= 2024-09-11T06:44:32.957 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:44:32.957 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test deleted = deleted 2024-09-11T06:44:32.957 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:44:32.958 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:44:32.958 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_44 2024-09-11T06:44:32.958 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_present cluster1 mirror image_44 deleted 2024-09-11T06:44:32.958 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:44:32.958 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:32.958 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_44 2024-09-11T06:44:32.958 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=deleted 2024-09-11T06:44:32.958 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:44:32.958 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:44:32.958 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:44:32.958 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_44 2024-09-11T06:44:32.958 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:44:32.958 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:44:32.959 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_44 2024-09-11T06:44:32.959 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_44 2024-09-11T06:44:32.959 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:44:32.995 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_44: (2) No such file or directory 2024-09-11T06:44:33.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image_id= 2024-09-11T06:44:33.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T06:44:33.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.1 2024-09-11T06:44:33.101 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_present cluster1 mirror image_44 deleted '' 2024-09-11T06:44:33.101 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:44:33.101 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:33.101 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_44 2024-09-11T06:44:33.101 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=deleted 2024-09-11T06:44:33.102 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:44:33.102 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=deleted 2024-09-11T06:44:33.102 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_image_id 2024-09-11T06:44:33.102 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_44 2024-09-11T06:44:33.103 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:44:33.103 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:44:33.103 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_44 2024-09-11T06:44:33.103 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_44 2024-09-11T06:44:33.103 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:44:33.145 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_44: (2) No such file or directory 2024-09-11T06:44:33.150 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_image_id= 2024-09-11T06:44:33.150 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:44:33.151 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test deleted = deleted 2024-09-11T06:44:33.151 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:44:33.151 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:44:33.151 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_45 2024-09-11T06:44:33.151 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_present cluster1 mirror image_45 deleted 2024-09-11T06:44:33.151 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:44:33.151 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:33.151 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_45 2024-09-11T06:44:33.151 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=deleted 2024-09-11T06:44:33.151 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:44:33.151 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:44:33.151 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:44:33.151 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_45 2024-09-11T06:44:33.152 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:44:33.152 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:44:33.152 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_45 2024-09-11T06:44:33.152 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_45 2024-09-11T06:44:33.152 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:44:33.190 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_45: (2) No such file or directory 2024-09-11T06:44:33.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image_id= 2024-09-11T06:44:33.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T06:44:33.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.1 2024-09-11T06:44:33.295 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_present cluster1 mirror image_45 deleted '' 2024-09-11T06:44:33.295 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:44:33.295 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:33.295 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_45 2024-09-11T06:44:33.295 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=deleted 2024-09-11T06:44:33.295 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:44:33.295 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=deleted 2024-09-11T06:44:33.295 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_image_id 2024-09-11T06:44:33.296 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_45 2024-09-11T06:44:33.296 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:44:33.296 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:44:33.297 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_45 2024-09-11T06:44:33.297 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_45 2024-09-11T06:44:33.297 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:44:33.337 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_45: (2) No such file or directory 2024-09-11T06:44:33.343 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_image_id= 2024-09-11T06:44:33.343 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:44:33.343 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test deleted = deleted 2024-09-11T06:44:33.343 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:44:33.343 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:44:33.344 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_46 2024-09-11T06:44:33.344 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_present cluster1 mirror image_46 deleted 2024-09-11T06:44:33.344 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:44:33.344 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:33.344 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_46 2024-09-11T06:44:33.344 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=deleted 2024-09-11T06:44:33.344 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:44:33.344 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:44:33.344 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:44:33.344 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_46 2024-09-11T06:44:33.345 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:44:33.345 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:44:33.345 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_46 2024-09-11T06:44:33.345 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_46 2024-09-11T06:44:33.345 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:44:33.382 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_46: (2) No such file or directory 2024-09-11T06:44:33.386 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image_id= 2024-09-11T06:44:33.386 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T06:44:33.386 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.1 2024-09-11T06:44:33.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_present cluster1 mirror image_46 deleted '' 2024-09-11T06:44:33.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:44:33.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:33.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_46 2024-09-11T06:44:33.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=deleted 2024-09-11T06:44:33.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:44:33.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=deleted 2024-09-11T06:44:33.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_image_id 2024-09-11T06:44:33.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_46 2024-09-11T06:44:33.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:44:33.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:44:33.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_46 2024-09-11T06:44:33.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_46 2024-09-11T06:44:33.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:44:33.535 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_46: (2) No such file or directory 2024-09-11T06:44:33.541 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_image_id= 2024-09-11T06:44:33.541 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:44:33.541 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test deleted = deleted 2024-09-11T06:44:33.541 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:44:33.541 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:44:33.541 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_47 2024-09-11T06:44:33.541 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_present cluster1 mirror image_47 deleted 2024-09-11T06:44:33.541 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:44:33.541 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:33.541 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_47 2024-09-11T06:44:33.541 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=deleted 2024-09-11T06:44:33.541 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:44:33.541 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:44:33.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:44:33.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_47 2024-09-11T06:44:33.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:44:33.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:44:33.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_47 2024-09-11T06:44:33.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_47 2024-09-11T06:44:33.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:44:33.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_47: (2) No such file or directory 2024-09-11T06:44:33.583 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image_id= 2024-09-11T06:44:33.583 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T06:44:33.583 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.1 2024-09-11T06:44:33.684 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_present cluster1 mirror image_47 deleted '' 2024-09-11T06:44:33.685 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:44:33.685 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:33.685 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_47 2024-09-11T06:44:33.685 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=deleted 2024-09-11T06:44:33.685 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:44:33.685 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=deleted 2024-09-11T06:44:33.685 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_image_id 2024-09-11T06:44:33.686 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_47 2024-09-11T06:44:33.686 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:44:33.686 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:44:33.686 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_47 2024-09-11T06:44:33.686 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_47 2024-09-11T06:44:33.686 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:44:33.731 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_47: (2) No such file or directory 2024-09-11T06:44:33.736 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_image_id= 2024-09-11T06:44:33.737 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:44:33.737 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test deleted = deleted 2024-09-11T06:44:33.737 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:44:33.737 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:44:33.737 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_48 2024-09-11T06:44:33.737 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_present cluster1 mirror image_48 deleted 2024-09-11T06:44:33.737 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:44:33.737 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:33.737 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_48 2024-09-11T06:44:33.737 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=deleted 2024-09-11T06:44:33.737 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:44:33.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:44:33.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:44:33.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_48 2024-09-11T06:44:33.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:44:33.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:44:33.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_48 2024-09-11T06:44:33.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_48 2024-09-11T06:44:33.738 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:44:33.776 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_48: (2) No such file or directory 2024-09-11T06:44:33.780 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image_id= 2024-09-11T06:44:33.780 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T06:44:33.780 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.1 2024-09-11T06:44:33.882 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_present cluster1 mirror image_48 deleted '' 2024-09-11T06:44:33.882 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:44:33.882 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:33.882 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_48 2024-09-11T06:44:33.882 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=deleted 2024-09-11T06:44:33.882 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:44:33.882 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=deleted 2024-09-11T06:44:33.882 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_image_id 2024-09-11T06:44:33.883 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_48 2024-09-11T06:44:33.883 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:44:33.883 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:44:33.883 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_48 2024-09-11T06:44:33.883 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_48 2024-09-11T06:44:33.883 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:44:33.930 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_48: (2) No such file or directory 2024-09-11T06:44:33.936 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_image_id= 2024-09-11T06:44:33.936 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:44:33.936 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test deleted = deleted 2024-09-11T06:44:33.936 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:44:33.936 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:44:33.936 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_49 2024-09-11T06:44:33.936 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_present cluster1 mirror image_49 deleted 2024-09-11T06:44:33.936 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:44:33.936 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:33.936 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_49 2024-09-11T06:44:33.936 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=deleted 2024-09-11T06:44:33.936 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:44:33.936 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:44:33.937 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:44:33.937 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_49 2024-09-11T06:44:33.937 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:44:33.937 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:44:33.937 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_49 2024-09-11T06:44:33.937 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_49 2024-09-11T06:44:33.937 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:44:33.973 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_49: (2) No such file or directory 2024-09-11T06:44:33.977 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image_id= 2024-09-11T06:44:33.977 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T06:44:33.977 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.1 2024-09-11T06:44:34.079 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_present cluster1 mirror image_49 deleted '' 2024-09-11T06:44:34.079 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:44:34.079 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:34.079 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_49 2024-09-11T06:44:34.079 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=deleted 2024-09-11T06:44:34.079 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:44:34.079 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=deleted 2024-09-11T06:44:34.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_image_id 2024-09-11T06:44:34.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_49 2024-09-11T06:44:34.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:44:34.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:44:34.081 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_49 2024-09-11T06:44:34.081 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_49 2024-09-11T06:44:34.081 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:44:34.124 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_49: (2) No such file or directory 2024-09-11T06:44:34.129 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_image_id= 2024-09-11T06:44:34.130 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:44:34.130 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test deleted = deleted 2024-09-11T06:44:34.130 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:44:34.130 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T06:44:34.130 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_50 2024-09-11T06:44:34.130 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_present cluster1 mirror image_50 deleted 2024-09-11T06:44:34.130 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:44:34.130 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:34.130 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_50 2024-09-11T06:44:34.130 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=deleted 2024-09-11T06:44:34.130 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:44:34.130 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:44:34.130 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:44:34.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_50 2024-09-11T06:44:34.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:44:34.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:44:34.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_50 2024-09-11T06:44:34.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_50 2024-09-11T06:44:34.131 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:44:34.167 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_50: (2) No such file or directory 2024-09-11T06:44:34.171 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image_id= 2024-09-11T06:44:34.172 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T06:44:34.172 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.1 2024-09-11T06:44:34.273 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_present cluster1 mirror image_50 deleted '' 2024-09-11T06:44:34.273 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:44:34.273 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:34.273 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_50 2024-09-11T06:44:34.273 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=deleted 2024-09-11T06:44:34.273 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:44:34.274 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=deleted 2024-09-11T06:44:34.274 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_image_id 2024-09-11T06:44:34.274 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_50 2024-09-11T06:44:34.274 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:44:34.274 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:44:34.274 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_50 2024-09-11T06:44:34.275 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_50 2024-09-11T06:44:34.275 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:44:34.319 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_50: (2) No such file or directory 2024-09-11T06:44:34.324 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_image_id= 2024-09-11T06:44:34.324 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:44:34.324 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test deleted = deleted 2024-09-11T06:44:34.324 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:44:34.324 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ testlog 'TEST: delete images during bootstrap' 2024-09-11T06:44:34.325 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ tee -a /tmp/tmp.Dxwd1ESqKa/rbd-mirror.test.log 2024-09-11T06:44:34.325 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ date '+%F %T' 2024-09-11T06:44:34.326 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ echo 2024-09-11 06:44:34 TEST: delete images during bootstrap 2024-09-11T06:44:34.326 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:2024-09-11 06:44:34 TEST: delete images during bootstrap 2024-09-11T06:44:34.326 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_pool_mirror_mode cluster1 mirror image 2024-09-11T06:44:34.326 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:44:34.326 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:34.326 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local mode=image 2024-09-11T06:44:34.326 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster1 mirror pool enable mirror image 2024-09-11T06:44:34.364 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:note: changing mirroring mode from pool to image 2024-09-11T06:44:34.380 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_pool_mirror_mode cluster2 mirror image 2024-09-11T06:44:34.380 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:44:34.380 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:34.380 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local mode=image 2024-09-11T06:44:34.380 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 mirror pool enable mirror image 2024-09-11T06:44:34.416 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:note: changing mirroring mode from pool to image 2024-09-11T06:44:34.425 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ start_mirror cluster1 2024-09-11T06:44:34.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:44:34.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance 2024-09-11T06:44:34.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set_cluster_instance cluster1 cluster instance 2024-09-11T06:44:34.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local val=cluster1 2024-09-11T06:44:34.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster_var_name=cluster 2024-09-11T06:44:34.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local instance_var_name=instance 2024-09-11T06:44:34.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cluster=cluster1 2024-09-11T06:44:34.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=cluster1 2024-09-11T06:44:34.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' cluster1 = cluster1 ']' 2024-09-11T06:44:34.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ instance=0 2024-09-11T06:44:34.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval cluster=cluster1 2024-09-11T06:44:34.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:44:34.426 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ eval instance=0 2024-09-11T06:44:34.427 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:44:34.427 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:44:34.427 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd-mirror --cluster cluster1 --id mirror.0 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2024-09-11T06:44:34.597 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:2024-09-11T06:44:34.599+0000 7f1151c1f880 -1 open: failed to lock pidfile /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid because another process locked it': (11) Resource temporarily unavailable 2024-09-11T06:44:34.597 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=test 2024-09-11T06:44:34.598 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 1 10 2024-09-11T06:44:34.599 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 10` 2024-09-11T06:44:34.599 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_1 2024-09-11T06:44:34.599 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_image cluster2 mirror image_1 512M 2024-09-11T06:44:34.599 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:44:34.599 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:44:34.599 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:34.599 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:44:34.599 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_1 2024-09-11T06:44:34.600 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:44:34.600 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local size=128 2024-09-11T06:44:34.600 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n 512M ']' 2024-09-11T06:44:34.600 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ size=512M 2024-09-11T06:44:34.600 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:44:34.600 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,journaling mirror/image_1 2024-09-11T06:44:34.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ enable_mirror cluster2 mirror image_1 2024-09-11T06:44:34.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:44:34.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:34.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_1 2024-09-11T06:44:34.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local mode=journal 2024-09-11T06:44:34.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_1 journal 2024-09-11T06:44:34.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:Mirroring enabled 2024-09-11T06:44:34.725 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 info mirror/image_1 2024-09-11T06:44:34.765 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:rbd image 'image_1': 2024-09-11T06:44:34.765 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: size 512 MiB in 128 objects 2024-09-11T06:44:34.765 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: order 22 (4 MiB objects) 2024-09-11T06:44:34.765 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: snapshot_count: 0 2024-09-11T06:44:34.765 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: id: 2e317e9adad4 2024-09-11T06:44:34.765 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: block_name_prefix: rbd_data.2e317e9adad4 2024-09-11T06:44:34.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: format: 2 2024-09-11T06:44:34.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: features: layering, exclusive-lock, journaling 2024-09-11T06:44:34.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: op_features: 2024-09-11T06:44:34.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: flags: 2024-09-11T06:44:34.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: create_timestamp: Wed Sep 11 06:44:34 2024 2024-09-11T06:44:34.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: access_timestamp: Wed Sep 11 06:44:34 2024 2024-09-11T06:44:34.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: modify_timestamp: Wed Sep 11 06:44:34 2024 2024-09-11T06:44:34.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: journal: 2e317e9adad4 2024-09-11T06:44:34.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring state: enabled 2024-09-11T06:44:34.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring mode: journal 2024-09-11T06:44:34.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring global id: f30e3a2b-2f1d-4667-96e0-7be2ac859482 2024-09-11T06:44:34.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring primary: true 2024-09-11T06:44:34.769 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ stress_write_image cluster2 mirror image_1 2024-09-11T06:44:34.769 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:44:34.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:34.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_1 2024-09-11T06:44:34.770 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T06:44:34.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local duration=8 2024-09-11T06:44:34.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set +e 2024-09-11T06:44:34.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_1 --debug-rbd=20 --debug-journaler=20 2024-09-11T06:44:38.968 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:100 IOs, 51339264 bytes 2024-09-11T06:44:40.449 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:200 IOs, 70917120 bytes 2024-09-11T06:44:40.870 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:300 IOs, 74584064 bytes 2024-09-11T06:44:41.519 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:400 IOs, 81506304 bytes 2024-09-11T06:44:42.779 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ error_code=124 2024-09-11T06:44:42.780 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set -e 2024-09-11T06:44:42.780 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T06:44:42.780 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:44:42.780 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_present cluster1 mirror image_1 present 2024-09-11T06:44:42.780 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:44:42.780 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:42.780 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_1 2024-09-11T06:44:42.780 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=present 2024-09-11T06:44:42.780 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:44:42.780 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:44:42.780 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:44:42.781 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_1 2024-09-11T06:44:42.781 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:44:42.781 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:44:42.781 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_1 2024-09-11T06:44:42.781 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-09-11T06:44:42.781 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:44:42.835 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image_id=10a6815d2063 2024-09-11T06:44:42.835 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T06:44:42.835 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.1 2024-09-11T06:44:42.937 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_present cluster1 mirror image_1 present 10a6815d2063 2024-09-11T06:44:42.937 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:44:42.937 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:42.937 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_1 2024-09-11T06:44:42.937 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=present 2024-09-11T06:44:42.937 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id=10a6815d2063 2024-09-11T06:44:42.937 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=deleted 2024-09-11T06:44:42.937 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_image_id 2024-09-11T06:44:42.938 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_1 2024-09-11T06:44:42.938 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:44:42.938 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:44:42.939 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_1 2024-09-11T06:44:42.939 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-09-11T06:44:42.939 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:44:43.002 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_image_id=10a6815d2063 2024-09-11T06:44:43.002 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 10a6815d2063 2024-09-11T06:44:43.002 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -z 10a6815d2063 -o 10a6815d2063 = 10a6815d2063 2024-09-11T06:44:43.003 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_state=present 2024-09-11T06:44:43.003 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test present = present 2024-09-11T06:44:43.003 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:44:43.003 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ disable_mirror cluster2 mirror image_1 2024-09-11T06:44:43.003 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:44:43.003 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:44:43.003 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_1 2024-09-11T06:44:43.003 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_1 2024-09-11T06:45:11.813 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:Mirroring disabled 2024-09-11T06:45:11.827 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_present cluster1 mirror image_1 deleted 2024-09-11T06:45:11.827 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:45:11.828 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:45:11.828 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_1 2024-09-11T06:45:11.828 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=deleted 2024-09-11T06:45:11.828 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:45:11.828 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:45:11.828 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:45:11.828 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_1 2024-09-11T06:45:11.828 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:45:11.828 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:45:11.828 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_1 2024-09-11T06:45:11.829 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-09-11T06:45:11.829 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:45:11.890 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image_id=10a6815d2063 2024-09-11T06:45:11.891 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T06:45:11.891 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.1 2024-09-11T06:45:11.992 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_present cluster1 mirror image_1 deleted 10a6815d2063 2024-09-11T06:45:11.992 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:45:11.992 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:45:11.992 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_1 2024-09-11T06:45:11.992 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=deleted 2024-09-11T06:45:11.992 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id=10a6815d2063 2024-09-11T06:45:11.992 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=deleted 2024-09-11T06:45:11.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_image_id 2024-09-11T06:45:11.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_1 2024-09-11T06:45:11.993 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:45:11.994 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:45:11.994 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_1 2024-09-11T06:45:11.994 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-09-11T06:45:11.994 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:45:12.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_image_id=10a6815d2063 2024-09-11T06:45:12.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 10a6815d2063 2024-09-11T06:45:12.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -z 10a6815d2063 -o 10a6815d2063 = 10a6815d2063 2024-09-11T06:45:12.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_state=present 2024-09-11T06:45:12.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test deleted = present 2024-09-11T06:45:12.062 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T06:45:12.063 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:45:13.063 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_present cluster1 mirror image_1 deleted 10a6815d2063 2024-09-11T06:45:13.063 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:45:13.063 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:45:13.063 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_1 2024-09-11T06:45:13.064 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=deleted 2024-09-11T06:45:13.064 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id=10a6815d2063 2024-09-11T06:45:13.064 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=deleted 2024-09-11T06:45:13.064 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_image_id 2024-09-11T06:45:13.065 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_1 2024-09-11T06:45:13.065 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:45:13.065 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:45:13.065 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_1 2024-09-11T06:45:13.065 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-09-11T06:45:13.065 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:45:13.113 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_1: (2) No such file or directory 2024-09-11T06:45:13.117 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_image_id= 2024-09-11T06:45:13.118 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:45:13.118 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test deleted = deleted 2024-09-11T06:45:13.118 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:45:13.118 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ purge_snapshots cluster2 mirror image_1 2024-09-11T06:45:13.118 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:45:13.118 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:45:13.118 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_1 2024-09-11T06:45:13.118 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 snap purge mirror/image_1 2024-09-11T06:45:13.735 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing all snapshots: 100% complete... Removing all snapshots: 100% complete...done. 2024-09-11T06:45:13.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image_retry cluster2 mirror image_1 2024-09-11T06:45:13.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:45:13.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:45:13.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_1 2024-09-11T06:45:13.767 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T06:45:13.768 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:45:13.769 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image cluster2 mirror image_1 2024-09-11T06:45:13.769 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:45:13.769 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:45:13.769 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_1 2024-09-11T06:45:13.769 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_1 2024-09-11T06:45:13.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 rm mirror/image_1 2024-09-11T06:45:14.060 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-11T06:45:14.065 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:45:14.065 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 10` 2024-09-11T06:45:14.065 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_2 2024-09-11T06:45:14.065 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_image cluster2 mirror image_2 512M 2024-09-11T06:45:14.065 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:45:14.065 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:45:14.065 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:45:14.065 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:45:14.065 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_2 2024-09-11T06:45:14.065 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:45:14.065 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local size=128 2024-09-11T06:45:14.065 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n 512M ']' 2024-09-11T06:45:14.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ size=512M 2024-09-11T06:45:14.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:45:14.066 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,journaling mirror/image_2 2024-09-11T06:45:14.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ enable_mirror cluster2 mirror image_2 2024-09-11T06:45:14.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:45:14.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:45:14.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_2 2024-09-11T06:45:14.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local mode=journal 2024-09-11T06:45:14.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_2 journal 2024-09-11T06:45:14.173 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:Mirroring enabled 2024-09-11T06:45:14.179 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 info mirror/image_2 2024-09-11T06:45:14.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:rbd image 'image_2': 2024-09-11T06:45:14.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: size 512 MiB in 128 objects 2024-09-11T06:45:14.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: order 22 (4 MiB objects) 2024-09-11T06:45:14.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: snapshot_count: 0 2024-09-11T06:45:14.219 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: id: 2e4960018d09 2024-09-11T06:45:14.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: block_name_prefix: rbd_data.2e4960018d09 2024-09-11T06:45:14.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: format: 2 2024-09-11T06:45:14.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: features: layering, exclusive-lock, journaling 2024-09-11T06:45:14.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: op_features: 2024-09-11T06:45:14.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: flags: 2024-09-11T06:45:14.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: create_timestamp: Wed Sep 11 06:45:14 2024 2024-09-11T06:45:14.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: access_timestamp: Wed Sep 11 06:45:14 2024 2024-09-11T06:45:14.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: modify_timestamp: Wed Sep 11 06:45:14 2024 2024-09-11T06:45:14.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: journal: 2e4960018d09 2024-09-11T06:45:14.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring state: enabled 2024-09-11T06:45:14.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring mode: journal 2024-09-11T06:45:14.220 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring global id: c1259713-569c-45f1-b6f7-8a5be2cd8906 2024-09-11T06:45:14.221 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring primary: true 2024-09-11T06:45:14.223 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ stress_write_image cluster2 mirror image_2 2024-09-11T06:45:14.223 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:45:14.223 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:45:14.223 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_2 2024-09-11T06:45:14.223 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T06:45:14.224 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local duration=11 2024-09-11T06:45:14.225 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set +e 2024-09-11T06:45:14.225 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_2 --debug-rbd=20 --debug-journaler=20 2024-09-11T06:45:17.614 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:100 IOs, 46189568 bytes 2024-09-11T06:45:21.610 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:200 IOs, 95598592 bytes 2024-09-11T06:45:23.449 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:300 IOs, 115876864 bytes 2024-09-11T06:45:24.853 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:45:24.854+0000 7efd718c5640 -1 reset not still connected to 0x560b15a708f0 2024-09-11T06:45:24.853 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:45:24.854+0000 7efd718c5640 -1 reset not still connected to 0x560b15a70dd0 2024-09-11T06:45:24.853 INFO:tasks.ceph.cluster2.osd.2.smithi112.stderr:2024-09-11T06:45:24.854+0000 7efd718c5640 -1 reset not still connected to 0x560b15a71520 2024-09-11T06:45:25.233 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ error_code=124 2024-09-11T06:45:25.233 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set -e 2024-09-11T06:45:25.234 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T06:45:25.234 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:45:25.234 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_present cluster1 mirror image_2 present 2024-09-11T06:45:25.234 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:45:25.234 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:45:25.234 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_2 2024-09-11T06:45:25.234 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=present 2024-09-11T06:45:25.234 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:45:25.234 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:45:25.234 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:45:25.234 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_2 2024-09-11T06:45:25.234 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:45:25.234 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:45:25.234 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_2 2024-09-11T06:45:25.235 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-09-11T06:45:25.235 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:45:25.421 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image_id=10a670312e24 2024-09-11T06:45:25.421 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T06:45:25.421 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.1 2024-09-11T06:45:25.523 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_present cluster1 mirror image_2 present 10a670312e24 2024-09-11T06:45:25.523 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:45:25.523 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:45:25.523 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_2 2024-09-11T06:45:25.523 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=present 2024-09-11T06:45:25.523 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id=10a670312e24 2024-09-11T06:45:25.523 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=deleted 2024-09-11T06:45:25.523 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_image_id 2024-09-11T06:45:25.523 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_2 2024-09-11T06:45:25.524 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:45:25.524 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:45:25.524 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_2 2024-09-11T06:45:25.524 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-09-11T06:45:25.524 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:45:25.591 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_image_id=10a670312e24 2024-09-11T06:45:25.591 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 10a670312e24 2024-09-11T06:45:25.591 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -z 10a670312e24 -o 10a670312e24 = 10a670312e24 2024-09-11T06:45:25.591 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_state=present 2024-09-11T06:45:25.591 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test present = present 2024-09-11T06:45:25.591 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:45:25.592 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ disable_mirror cluster2 mirror image_2 2024-09-11T06:45:25.592 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:45:25.592 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:45:25.592 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_2 2024-09-11T06:45:25.592 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_2 2024-09-11T06:45:55.878 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:Mirroring disabled 2024-09-11T06:45:55.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_present cluster1 mirror image_2 deleted 2024-09-11T06:45:55.886 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:45:55.887 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:45:55.887 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_2 2024-09-11T06:45:55.887 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=deleted 2024-09-11T06:45:55.887 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:45:55.887 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:45:55.887 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:45:55.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_2 2024-09-11T06:45:55.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:45:55.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:45:55.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_2 2024-09-11T06:45:55.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-09-11T06:45:55.888 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:45:55.925 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_2: (2) No such file or directory 2024-09-11T06:45:55.929 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image_id= 2024-09-11T06:45:55.929 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T06:45:55.929 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.1 2024-09-11T06:45:56.031 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_present cluster1 mirror image_2 deleted '' 2024-09-11T06:45:56.031 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:45:56.031 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:45:56.031 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_2 2024-09-11T06:45:56.031 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=deleted 2024-09-11T06:45:56.031 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:45:56.031 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=deleted 2024-09-11T06:45:56.031 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_image_id 2024-09-11T06:45:56.032 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_2 2024-09-11T06:45:56.032 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:45:56.032 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:45:56.032 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_2 2024-09-11T06:45:56.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-09-11T06:45:56.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:45:56.076 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_2: (2) No such file or directory 2024-09-11T06:45:56.080 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_image_id= 2024-09-11T06:45:56.081 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:45:56.081 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test deleted = deleted 2024-09-11T06:45:56.081 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:45:56.081 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ purge_snapshots cluster2 mirror image_2 2024-09-11T06:45:56.081 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:45:56.081 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:45:56.081 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_2 2024-09-11T06:45:56.081 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 snap purge mirror/image_2 2024-09-11T06:45:56.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image_retry cluster2 mirror image_2 2024-09-11T06:45:56.343 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:45:56.343 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:45:56.343 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_2 2024-09-11T06:45:56.343 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T06:45:56.343 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:45:56.343 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image cluster2 mirror image_2 2024-09-11T06:45:56.344 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:45:56.344 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:45:56.344 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_2 2024-09-11T06:45:56.344 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_2 2024-09-11T06:45:56.398 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 rm mirror/image_2 2024-09-11T06:45:56.655 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-11T06:45:56.659 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:45:56.659 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 10` 2024-09-11T06:45:56.659 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_3 2024-09-11T06:45:56.659 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_image cluster2 mirror image_3 512M 2024-09-11T06:45:56.660 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:45:56.660 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:45:56.660 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:45:56.660 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:45:56.660 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_3 2024-09-11T06:45:56.660 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:45:56.660 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local size=128 2024-09-11T06:45:56.660 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n 512M ']' 2024-09-11T06:45:56.660 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ size=512M 2024-09-11T06:45:56.660 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:45:56.660 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,journaling mirror/image_3 2024-09-11T06:45:56.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ enable_mirror cluster2 mirror image_3 2024-09-11T06:45:56.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:45:56.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:45:56.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_3 2024-09-11T06:45:56.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local mode=journal 2024-09-11T06:45:56.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_3 journal 2024-09-11T06:45:56.768 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:Mirroring enabled 2024-09-11T06:45:56.774 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 info mirror/image_3 2024-09-11T06:45:56.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:rbd image 'image_3': 2024-09-11T06:45:56.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: size 512 MiB in 128 objects 2024-09-11T06:45:56.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: order 22 (4 MiB objects) 2024-09-11T06:45:56.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: snapshot_count: 0 2024-09-11T06:45:56.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: id: 2e61b1d0515c 2024-09-11T06:45:56.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: block_name_prefix: rbd_data.2e61b1d0515c 2024-09-11T06:45:56.814 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: format: 2 2024-09-11T06:45:56.815 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: features: layering, exclusive-lock, journaling 2024-09-11T06:45:56.815 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: op_features: 2024-09-11T06:45:56.815 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: flags: 2024-09-11T06:45:56.815 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: create_timestamp: Wed Sep 11 06:45:56 2024 2024-09-11T06:45:56.815 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: access_timestamp: Wed Sep 11 06:45:56 2024 2024-09-11T06:45:56.815 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: modify_timestamp: Wed Sep 11 06:45:56 2024 2024-09-11T06:45:56.815 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: journal: 2e61b1d0515c 2024-09-11T06:45:56.815 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring state: enabled 2024-09-11T06:45:56.815 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring mode: journal 2024-09-11T06:45:56.815 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring global id: c183b8d5-59ee-415d-a8ba-6af8f4620d77 2024-09-11T06:45:56.815 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring primary: true 2024-09-11T06:45:56.817 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ stress_write_image cluster2 mirror image_3 2024-09-11T06:45:56.817 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:45:56.817 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:45:56.818 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_3 2024-09-11T06:45:56.818 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T06:45:56.819 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local duration=6 2024-09-11T06:45:56.819 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set +e 2024-09-11T06:45:56.819 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_3 --debug-rbd=20 --debug-journaler=20 2024-09-11T06:46:01.848 INFO:tasks.ceph.cluster2.osd.1.smithi112.stderr:2024-09-11T06:46:01.849+0000 7f2cecaac640 -1 reset not still connected to 0x55da8e8c29c0 2024-09-11T06:46:01.849 INFO:tasks.ceph.cluster2.osd.1.smithi112.stderr:2024-09-11T06:46:01.849+0000 7f2cecaac640 -1 reset not still connected to 0x55da8ee0bd40 2024-09-11T06:46:01.849 INFO:tasks.ceph.cluster2.osd.1.smithi112.stderr:2024-09-11T06:46:01.849+0000 7f2cecaac640 -1 reset not still connected to 0x55daa2cd3e10 2024-09-11T06:46:01.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:100 IOs, 79831040 bytes 2024-09-11T06:46:02.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ error_code=124 2024-09-11T06:46:02.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set -e 2024-09-11T06:46:02.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T06:46:02.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:46:02.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_present cluster1 mirror image_3 present 2024-09-11T06:46:02.833 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:46:02.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:46:02.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_3 2024-09-11T06:46:02.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=present 2024-09-11T06:46:02.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:46:02.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:46:02.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:46:02.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_3 2024-09-11T06:46:02.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:46:02.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:46:02.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_3 2024-09-11T06:46:02.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-09-11T06:46:02.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:46:02.905 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image_id=10a6231b2fef 2024-09-11T06:46:02.905 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T06:46:02.905 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.1 2024-09-11T06:46:03.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_present cluster1 mirror image_3 present 10a6231b2fef 2024-09-11T06:46:03.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:46:03.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:46:03.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_3 2024-09-11T06:46:03.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=present 2024-09-11T06:46:03.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id=10a6231b2fef 2024-09-11T06:46:03.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=deleted 2024-09-11T06:46:03.006 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_image_id 2024-09-11T06:46:03.007 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_3 2024-09-11T06:46:03.007 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:46:03.007 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:46:03.007 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_3 2024-09-11T06:46:03.007 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:46:03.007 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-09-11T06:46:03.078 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_image_id=10a6231b2fef 2024-09-11T06:46:03.078 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 10a6231b2fef 2024-09-11T06:46:03.078 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -z 10a6231b2fef -o 10a6231b2fef = 10a6231b2fef 2024-09-11T06:46:03.078 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_state=present 2024-09-11T06:46:03.078 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test present = present 2024-09-11T06:46:03.079 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:46:03.079 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ disable_mirror cluster2 mirror image_3 2024-09-11T06:46:03.079 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:46:03.079 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:46:03.079 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_3 2024-09-11T06:46:03.079 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_3 2024-09-11T06:46:32.858 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:Mirroring disabled 2024-09-11T06:46:32.869 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_present cluster1 mirror image_3 deleted 2024-09-11T06:46:32.869 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:46:32.869 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:46:32.869 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_3 2024-09-11T06:46:32.869 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=deleted 2024-09-11T06:46:32.870 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:46:32.870 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:46:32.870 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:46:32.870 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_3 2024-09-11T06:46:32.870 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:46:32.870 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:46:32.870 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_3 2024-09-11T06:46:32.870 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-09-11T06:46:32.870 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:46:32.935 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image_id=10a6231b2fef 2024-09-11T06:46:32.935 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T06:46:32.935 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.1 2024-09-11T06:46:33.036 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_present cluster1 mirror image_3 deleted 10a6231b2fef 2024-09-11T06:46:33.036 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:46:33.036 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:46:33.036 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_3 2024-09-11T06:46:33.036 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=deleted 2024-09-11T06:46:33.036 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id=10a6231b2fef 2024-09-11T06:46:33.036 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=deleted 2024-09-11T06:46:33.037 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_image_id 2024-09-11T06:46:33.037 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_3 2024-09-11T06:46:33.037 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:46:33.037 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:46:33.038 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_3 2024-09-11T06:46:33.038 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-09-11T06:46:33.038 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:46:33.086 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_3: (2) No such file or directory 2024-09-11T06:46:33.091 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_image_id= 2024-09-11T06:46:33.091 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:46:33.091 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test deleted = deleted 2024-09-11T06:46:33.091 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:46:33.091 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ purge_snapshots cluster2 mirror image_3 2024-09-11T06:46:33.091 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:46:33.091 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:46:33.091 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_3 2024-09-11T06:46:33.091 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 snap purge mirror/image_3 2024-09-11T06:46:33.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image_retry cluster2 mirror image_3 2024-09-11T06:46:33.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:46:33.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:46:33.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_3 2024-09-11T06:46:33.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T06:46:33.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:46:33.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image cluster2 mirror image_3 2024-09-11T06:46:33.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:46:33.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:46:33.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_3 2024-09-11T06:46:33.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_3 2024-09-11T06:46:33.188 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 rm mirror/image_3 2024-09-11T06:46:33.436 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-11T06:46:33.440 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:46:33.440 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 10` 2024-09-11T06:46:33.440 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_4 2024-09-11T06:46:33.441 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_image cluster2 mirror image_4 512M 2024-09-11T06:46:33.441 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:46:33.441 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:46:33.441 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:46:33.441 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:46:33.441 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_4 2024-09-11T06:46:33.441 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:46:33.441 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local size=128 2024-09-11T06:46:33.441 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n 512M ']' 2024-09-11T06:46:33.441 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ size=512M 2024-09-11T06:46:33.441 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:46:33.441 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,journaling mirror/image_4 2024-09-11T06:46:33.501 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ enable_mirror cluster2 mirror image_4 2024-09-11T06:46:33.501 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:46:33.501 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:46:33.501 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_4 2024-09-11T06:46:33.501 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local mode=journal 2024-09-11T06:46:33.501 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_4 journal 2024-09-11T06:46:33.553 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:Mirroring enabled 2024-09-11T06:46:33.560 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 info mirror/image_4 2024-09-11T06:46:33.599 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:rbd image 'image_4': 2024-09-11T06:46:33.599 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: size 512 MiB in 128 objects 2024-09-11T06:46:33.599 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: order 22 (4 MiB objects) 2024-09-11T06:46:33.599 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: snapshot_count: 0 2024-09-11T06:46:33.599 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: id: 2e791c348a74 2024-09-11T06:46:33.599 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: block_name_prefix: rbd_data.2e791c348a74 2024-09-11T06:46:33.599 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: format: 2 2024-09-11T06:46:33.599 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: features: layering, exclusive-lock, journaling 2024-09-11T06:46:33.599 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: op_features: 2024-09-11T06:46:33.599 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: flags: 2024-09-11T06:46:33.599 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: create_timestamp: Wed Sep 11 06:46:33 2024 2024-09-11T06:46:33.600 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: access_timestamp: Wed Sep 11 06:46:33 2024 2024-09-11T06:46:33.600 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: modify_timestamp: Wed Sep 11 06:46:33 2024 2024-09-11T06:46:33.600 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: journal: 2e791c348a74 2024-09-11T06:46:33.600 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring state: enabled 2024-09-11T06:46:33.600 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring mode: journal 2024-09-11T06:46:33.600 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring global id: 4869d8cb-3ed2-45e3-aeb4-10cd989987b6 2024-09-11T06:46:33.600 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring primary: true 2024-09-11T06:46:33.602 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ stress_write_image cluster2 mirror image_4 2024-09-11T06:46:33.602 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:46:33.602 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:46:33.603 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_4 2024-09-11T06:46:33.603 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T06:46:33.604 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local duration=13 2024-09-11T06:46:33.605 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set +e 2024-09-11T06:46:33.605 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_4 --debug-rbd=20 --debug-journaler=20 2024-09-11T06:46:38.454 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:100 IOs, 62282752 bytes 2024-09-11T06:46:42.227 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:200 IOs, 112009216 bytes 2024-09-11T06:46:42.448 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:300 IOs, 112624640 bytes 2024-09-11T06:46:44.030 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:400 IOs, 134291456 bytes 2024-09-11T06:46:44.247 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:500 IOs, 134894592 bytes 2024-09-11T06:46:46.611 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ error_code=124 2024-09-11T06:46:46.611 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set -e 2024-09-11T06:46:46.612 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T06:46:46.612 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:46:46.612 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_present cluster1 mirror image_4 present 2024-09-11T06:46:46.612 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:46:46.612 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:46:46.612 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_4 2024-09-11T06:46:46.612 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=present 2024-09-11T06:46:46.612 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:46:46.612 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:46:46.612 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:46:46.612 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_4 2024-09-11T06:46:46.612 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:46:46.613 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:46:46.613 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_4 2024-09-11T06:46:46.613 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-09-11T06:46:46.613 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:46:46.669 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image_id=10a91477fcdd 2024-09-11T06:46:46.670 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T06:46:46.670 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.1 2024-09-11T06:46:46.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_present cluster1 mirror image_4 present 10a91477fcdd 2024-09-11T06:46:46.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:46:46.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:46:46.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_4 2024-09-11T06:46:46.771 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=present 2024-09-11T06:46:46.772 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id=10a91477fcdd 2024-09-11T06:46:46.772 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=deleted 2024-09-11T06:46:46.772 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_image_id 2024-09-11T06:46:46.773 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_4 2024-09-11T06:46:46.773 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:46:46.773 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:46:46.773 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_4 2024-09-11T06:46:46.773 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-09-11T06:46:46.773 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:46:46.837 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_image_id=10a91477fcdd 2024-09-11T06:46:46.838 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 10a91477fcdd 2024-09-11T06:46:46.838 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -z 10a91477fcdd -o 10a91477fcdd = 10a91477fcdd 2024-09-11T06:46:46.838 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_state=present 2024-09-11T06:46:46.838 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test present = present 2024-09-11T06:46:46.838 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:46:46.838 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ disable_mirror cluster2 mirror image_4 2024-09-11T06:46:46.838 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:46:46.838 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:46:46.838 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_4 2024-09-11T06:46:46.838 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_4 2024-09-11T06:47:16.142 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:Mirroring disabled 2024-09-11T06:47:16.152 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_present cluster1 mirror image_4 deleted 2024-09-11T06:47:16.152 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:47:16.152 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:47:16.152 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_4 2024-09-11T06:47:16.152 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=deleted 2024-09-11T06:47:16.152 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:47:16.152 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:47:16.152 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:47:16.153 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_4 2024-09-11T06:47:16.153 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:47:16.153 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:47:16.153 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_4 2024-09-11T06:47:16.153 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-09-11T06:47:16.153 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:47:16.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_4: (2) No such file or directory 2024-09-11T06:47:16.194 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image_id= 2024-09-11T06:47:16.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T06:47:16.195 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.1 2024-09-11T06:47:16.296 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_present cluster1 mirror image_4 deleted '' 2024-09-11T06:47:16.296 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:47:16.296 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:47:16.296 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_4 2024-09-11T06:47:16.296 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=deleted 2024-09-11T06:47:16.296 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:47:16.296 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=deleted 2024-09-11T06:47:16.296 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_image_id 2024-09-11T06:47:16.297 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_4 2024-09-11T06:47:16.297 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:47:16.297 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:47:16.297 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_4 2024-09-11T06:47:16.298 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-09-11T06:47:16.298 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:47:16.342 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_4: (2) No such file or directory 2024-09-11T06:47:16.347 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_image_id= 2024-09-11T06:47:16.347 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:47:16.347 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test deleted = deleted 2024-09-11T06:47:16.347 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:47:16.347 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ purge_snapshots cluster2 mirror image_4 2024-09-11T06:47:16.347 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:47:16.347 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:47:16.347 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_4 2024-09-11T06:47:16.347 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 snap purge mirror/image_4 2024-09-11T06:47:16.396 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image_retry cluster2 mirror image_4 2024-09-11T06:47:16.396 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:47:16.396 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:47:16.396 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_4 2024-09-11T06:47:16.396 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T06:47:16.396 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:47:16.397 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image cluster2 mirror image_4 2024-09-11T06:47:16.397 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:47:16.397 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:47:16.397 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_4 2024-09-11T06:47:16.397 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_4 2024-09-11T06:47:16.443 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 rm mirror/image_4 2024-09-11T06:47:16.702 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-11T06:47:16.707 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:47:16.707 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 10` 2024-09-11T06:47:16.707 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_5 2024-09-11T06:47:16.707 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_image cluster2 mirror image_5 512M 2024-09-11T06:47:16.707 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:47:16.707 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:47:16.707 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:47:16.707 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:47:16.707 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_5 2024-09-11T06:47:16.707 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:47:16.707 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local size=128 2024-09-11T06:47:16.707 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n 512M ']' 2024-09-11T06:47:16.708 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ size=512M 2024-09-11T06:47:16.708 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:47:16.708 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,journaling mirror/image_5 2024-09-11T06:47:16.765 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ enable_mirror cluster2 mirror image_5 2024-09-11T06:47:16.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:47:16.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:47:16.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_5 2024-09-11T06:47:16.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local mode=journal 2024-09-11T06:47:16.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_5 journal 2024-09-11T06:47:16.813 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:Mirroring enabled 2024-09-11T06:47:16.819 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 info mirror/image_5 2024-09-11T06:47:16.858 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:rbd image 'image_5': 2024-09-11T06:47:16.858 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: size 512 MiB in 128 objects 2024-09-11T06:47:16.858 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: order 22 (4 MiB objects) 2024-09-11T06:47:16.858 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: snapshot_count: 0 2024-09-11T06:47:16.858 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: id: 2e91a922304c 2024-09-11T06:47:16.858 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: block_name_prefix: rbd_data.2e91a922304c 2024-09-11T06:47:16.858 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: format: 2 2024-09-11T06:47:16.858 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: features: layering, exclusive-lock, journaling 2024-09-11T06:47:16.858 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: op_features: 2024-09-11T06:47:16.858 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: flags: 2024-09-11T06:47:16.858 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: create_timestamp: Wed Sep 11 06:47:16 2024 2024-09-11T06:47:16.859 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: access_timestamp: Wed Sep 11 06:47:16 2024 2024-09-11T06:47:16.859 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: modify_timestamp: Wed Sep 11 06:47:16 2024 2024-09-11T06:47:16.859 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: journal: 2e91a922304c 2024-09-11T06:47:16.859 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring state: enabled 2024-09-11T06:47:16.859 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring mode: journal 2024-09-11T06:47:16.859 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring global id: d89b2b58-f128-436a-a655-af44fc8bd508 2024-09-11T06:47:16.859 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring primary: true 2024-09-11T06:47:16.862 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ stress_write_image cluster2 mirror image_5 2024-09-11T06:47:16.862 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:47:16.862 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:47:16.862 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_5 2024-09-11T06:47:16.862 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T06:47:16.863 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local duration=9 2024-09-11T06:47:16.863 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set +e 2024-09-11T06:47:16.864 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_5 --debug-rbd=20 --debug-journaler=20 2024-09-11T06:47:18.090 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:100 IOs, 10676224 bytes 2024-09-11T06:47:23.253 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:200 IOs, 73582592 bytes 2024-09-11T06:47:25.380 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:300 IOs, 100030464 bytes 2024-09-11T06:47:25.872 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ error_code=124 2024-09-11T06:47:25.873 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set -e 2024-09-11T06:47:25.873 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T06:47:25.873 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:47:25.873 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_present cluster1 mirror image_5 present 2024-09-11T06:47:25.873 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:47:25.873 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:47:25.873 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_5 2024-09-11T06:47:25.873 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=present 2024-09-11T06:47:25.873 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:47:25.873 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:47:25.873 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:47:25.873 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_5 2024-09-11T06:47:25.874 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:47:25.874 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:47:25.874 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_5 2024-09-11T06:47:25.874 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-09-11T06:47:25.882 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:47:25.943 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image_id=10a64345e96e 2024-09-11T06:47:25.944 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T06:47:25.944 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.1 2024-09-11T06:47:26.045 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_present cluster1 mirror image_5 present 10a64345e96e 2024-09-11T06:47:26.045 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:47:26.045 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:47:26.045 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_5 2024-09-11T06:47:26.045 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=present 2024-09-11T06:47:26.045 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id=10a64345e96e 2024-09-11T06:47:26.045 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=deleted 2024-09-11T06:47:26.045 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_image_id 2024-09-11T06:47:26.046 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_5 2024-09-11T06:47:26.046 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:47:26.046 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:47:26.046 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_5 2024-09-11T06:47:26.047 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-09-11T06:47:26.047 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:47:26.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_image_id=10a64345e96e 2024-09-11T06:47:26.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 10a64345e96e 2024-09-11T06:47:26.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -z 10a64345e96e -o 10a64345e96e = 10a64345e96e 2024-09-11T06:47:26.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_state=present 2024-09-11T06:47:26.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test present = present 2024-09-11T06:47:26.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:47:26.111 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ disable_mirror cluster2 mirror image_5 2024-09-11T06:47:26.112 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:47:26.112 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:47:26.112 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_5 2024-09-11T06:47:26.112 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_5 2024-09-11T06:47:55.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:Mirroring disabled 2024-09-11T06:47:55.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_present cluster1 mirror image_5 deleted 2024-09-11T06:47:55.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:47:55.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:47:55.204 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_5 2024-09-11T06:47:55.205 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=deleted 2024-09-11T06:47:55.205 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:47:55.205 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:47:55.205 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:47:55.205 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_5 2024-09-11T06:47:55.205 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:47:55.205 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:47:55.205 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_5 2024-09-11T06:47:55.205 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-09-11T06:47:55.206 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:47:55.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_5: (2) No such file or directory 2024-09-11T06:47:55.245 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image_id= 2024-09-11T06:47:55.245 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T06:47:55.245 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.1 2024-09-11T06:47:55.347 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_present cluster1 mirror image_5 deleted '' 2024-09-11T06:47:55.347 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:47:55.347 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:47:55.347 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_5 2024-09-11T06:47:55.347 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=deleted 2024-09-11T06:47:55.347 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:47:55.347 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=deleted 2024-09-11T06:47:55.348 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_image_id 2024-09-11T06:47:55.348 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_5 2024-09-11T06:47:55.348 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:47:55.348 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:47:55.348 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_5 2024-09-11T06:47:55.348 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-09-11T06:47:55.348 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:47:55.392 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_5: (2) No such file or directory 2024-09-11T06:47:55.396 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_image_id= 2024-09-11T06:47:55.397 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:47:55.397 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test deleted = deleted 2024-09-11T06:47:55.397 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:47:55.397 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ purge_snapshots cluster2 mirror image_5 2024-09-11T06:47:55.397 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:47:55.397 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:47:55.397 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_5 2024-09-11T06:47:55.397 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 snap purge mirror/image_5 2024-09-11T06:47:55.448 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image_retry cluster2 mirror image_5 2024-09-11T06:47:55.448 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:47:55.448 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:47:55.448 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_5 2024-09-11T06:47:55.448 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T06:47:55.448 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:47:55.449 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image cluster2 mirror image_5 2024-09-11T06:47:55.449 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:47:55.449 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:47:55.449 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_5 2024-09-11T06:47:55.450 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_5 2024-09-11T06:47:55.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 rm mirror/image_5 2024-09-11T06:47:55.746 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-11T06:47:55.751 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:47:55.751 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 10` 2024-09-11T06:47:55.751 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_6 2024-09-11T06:47:55.751 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_image cluster2 mirror image_6 512M 2024-09-11T06:47:55.751 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:47:55.751 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:47:55.751 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:47:55.751 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:47:55.751 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_6 2024-09-11T06:47:55.751 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:47:55.751 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local size=128 2024-09-11T06:47:55.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n 512M ']' 2024-09-11T06:47:55.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ size=512M 2024-09-11T06:47:55.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:47:55.752 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,journaling mirror/image_6 2024-09-11T06:47:55.808 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ enable_mirror cluster2 mirror image_6 2024-09-11T06:47:55.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:47:55.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:47:55.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_6 2024-09-11T06:47:55.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local mode=journal 2024-09-11T06:47:55.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_6 journal 2024-09-11T06:47:55.857 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:Mirroring enabled 2024-09-11T06:47:55.863 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 info mirror/image_6 2024-09-11T06:47:55.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:rbd image 'image_6': 2024-09-11T06:47:55.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: size 512 MiB in 128 objects 2024-09-11T06:47:55.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: order 22 (4 MiB objects) 2024-09-11T06:47:55.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: snapshot_count: 0 2024-09-11T06:47:55.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: id: 2ea9b1c91c0 2024-09-11T06:47:55.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: block_name_prefix: rbd_data.2ea9b1c91c0 2024-09-11T06:47:55.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: format: 2 2024-09-11T06:47:55.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: features: layering, exclusive-lock, journaling 2024-09-11T06:47:55.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: op_features: 2024-09-11T06:47:55.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: flags: 2024-09-11T06:47:55.903 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: create_timestamp: Wed Sep 11 06:47:55 2024 2024-09-11T06:47:55.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: access_timestamp: Wed Sep 11 06:47:55 2024 2024-09-11T06:47:55.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: modify_timestamp: Wed Sep 11 06:47:55 2024 2024-09-11T06:47:55.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: journal: 2ea9b1c91c0 2024-09-11T06:47:55.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring state: enabled 2024-09-11T06:47:55.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring mode: journal 2024-09-11T06:47:55.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring global id: 4c468a57-f3dc-4a10-960e-69d35d2a7828 2024-09-11T06:47:55.904 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring primary: true 2024-09-11T06:47:55.906 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ stress_write_image cluster2 mirror image_6 2024-09-11T06:47:55.906 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:47:55.907 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:47:55.907 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_6 2024-09-11T06:47:55.907 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T06:47:55.908 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local duration=9 2024-09-11T06:47:55.908 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set +e 2024-09-11T06:47:55.909 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_6 --debug-rbd=20 --debug-journaler=20 2024-09-11T06:48:01.624 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:100 IOs, 76345344 bytes 2024-09-11T06:48:04.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ error_code=124 2024-09-11T06:48:04.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set -e 2024-09-11T06:48:04.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T06:48:04.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:48:04.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_present cluster1 mirror image_6 present 2024-09-11T06:48:04.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:48:04.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:48:04.920 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_6 2024-09-11T06:48:04.921 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=present 2024-09-11T06:48:04.921 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:48:04.921 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:48:04.921 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:48:04.921 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_6 2024-09-11T06:48:04.921 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:48:04.921 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:48:04.921 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_6 2024-09-11T06:48:04.921 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-09-11T06:48:04.921 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:48:05.190 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image_id=10a6228b99a6 2024-09-11T06:48:05.190 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T06:48:05.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.1 2024-09-11T06:48:05.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_present cluster1 mirror image_6 present 10a6228b99a6 2024-09-11T06:48:05.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:48:05.292 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:48:05.293 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_6 2024-09-11T06:48:05.293 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=present 2024-09-11T06:48:05.293 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id=10a6228b99a6 2024-09-11T06:48:05.293 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=deleted 2024-09-11T06:48:05.293 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_image_id 2024-09-11T06:48:05.293 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_6 2024-09-11T06:48:05.293 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:48:05.293 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:48:05.293 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_6 2024-09-11T06:48:05.294 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-09-11T06:48:05.294 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:48:05.362 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_image_id=10a6228b99a6 2024-09-11T06:48:05.363 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 10a6228b99a6 2024-09-11T06:48:05.363 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -z 10a6228b99a6 -o 10a6228b99a6 = 10a6228b99a6 2024-09-11T06:48:05.363 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_state=present 2024-09-11T06:48:05.363 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test present = present 2024-09-11T06:48:05.363 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:48:05.363 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ disable_mirror cluster2 mirror image_6 2024-09-11T06:48:05.363 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:48:05.363 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:48:05.363 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_6 2024-09-11T06:48:05.363 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_6 2024-09-11T06:48:42.380 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:Mirroring disabled 2024-09-11T06:48:42.388 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_present cluster1 mirror image_6 deleted 2024-09-11T06:48:42.388 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:48:42.388 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:48:42.388 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_6 2024-09-11T06:48:42.389 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=deleted 2024-09-11T06:48:42.389 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:48:42.389 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:48:42.389 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:48:42.389 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_6 2024-09-11T06:48:42.389 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:48:42.389 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:48:42.389 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_6 2024-09-11T06:48:42.389 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-09-11T06:48:42.389 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:48:42.452 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image_id=10a6228b99a6 2024-09-11T06:48:42.452 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T06:48:42.452 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.1 2024-09-11T06:48:42.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_present cluster1 mirror image_6 deleted 10a6228b99a6 2024-09-11T06:48:42.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:48:42.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:48:42.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_6 2024-09-11T06:48:42.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=deleted 2024-09-11T06:48:42.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id=10a6228b99a6 2024-09-11T06:48:42.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=deleted 2024-09-11T06:48:42.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_image_id 2024-09-11T06:48:42.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_6 2024-09-11T06:48:42.555 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:48:42.555 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:48:42.555 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_6 2024-09-11T06:48:42.555 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-09-11T06:48:42.555 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:48:42.602 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_6: (2) No such file or directory 2024-09-11T06:48:42.607 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_image_id= 2024-09-11T06:48:42.607 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:48:42.607 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test deleted = deleted 2024-09-11T06:48:42.607 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:48:42.607 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ purge_snapshots cluster2 mirror image_6 2024-09-11T06:48:42.607 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:48:42.607 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:48:42.607 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_6 2024-09-11T06:48:42.607 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 snap purge mirror/image_6 2024-09-11T06:48:42.656 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image_retry cluster2 mirror image_6 2024-09-11T06:48:42.656 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:48:42.656 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:48:42.656 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_6 2024-09-11T06:48:42.656 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T06:48:42.656 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:48:42.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image cluster2 mirror image_6 2024-09-11T06:48:42.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:48:42.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:48:42.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_6 2024-09-11T06:48:42.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_6 2024-09-11T06:48:42.704 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 rm mirror/image_6 2024-09-11T06:48:42.966 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-11T06:48:42.970 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:48:42.970 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 10` 2024-09-11T06:48:42.970 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_7 2024-09-11T06:48:42.970 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_image cluster2 mirror image_7 512M 2024-09-11T06:48:42.970 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:48:42.970 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:48:42.970 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:48:42.970 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:48:42.971 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_7 2024-09-11T06:48:42.971 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:48:42.971 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local size=128 2024-09-11T06:48:42.971 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n 512M ']' 2024-09-11T06:48:42.971 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ size=512M 2024-09-11T06:48:42.971 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:48:42.971 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,journaling mirror/image_7 2024-09-11T06:48:43.029 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ enable_mirror cluster2 mirror image_7 2024-09-11T06:48:43.029 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:48:43.029 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:48:43.029 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_7 2024-09-11T06:48:43.030 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local mode=journal 2024-09-11T06:48:43.030 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_7 journal 2024-09-11T06:48:43.076 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:Mirroring enabled 2024-09-11T06:48:43.082 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 info mirror/image_7 2024-09-11T06:48:43.121 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:rbd image 'image_7': 2024-09-11T06:48:43.122 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: size 512 MiB in 128 objects 2024-09-11T06:48:43.122 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: order 22 (4 MiB objects) 2024-09-11T06:48:43.122 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: snapshot_count: 0 2024-09-11T06:48:43.122 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: id: 2ec13e6eaff4 2024-09-11T06:48:43.122 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: block_name_prefix: rbd_data.2ec13e6eaff4 2024-09-11T06:48:43.122 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: format: 2 2024-09-11T06:48:43.122 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: features: layering, exclusive-lock, journaling 2024-09-11T06:48:43.122 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: op_features: 2024-09-11T06:48:43.122 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: flags: 2024-09-11T06:48:43.122 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: create_timestamp: Wed Sep 11 06:48:43 2024 2024-09-11T06:48:43.122 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: access_timestamp: Wed Sep 11 06:48:43 2024 2024-09-11T06:48:43.122 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: modify_timestamp: Wed Sep 11 06:48:43 2024 2024-09-11T06:48:43.122 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: journal: 2ec13e6eaff4 2024-09-11T06:48:43.122 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring state: enabled 2024-09-11T06:48:43.123 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring mode: journal 2024-09-11T06:48:43.123 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring global id: 9448fa78-976e-4d75-bbec-842467c23af5 2024-09-11T06:48:43.123 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring primary: true 2024-09-11T06:48:43.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ stress_write_image cluster2 mirror image_7 2024-09-11T06:48:43.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:48:43.125 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:48:43.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_7 2024-09-11T06:48:43.126 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T06:48:43.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local duration=7 2024-09-11T06:48:43.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set +e 2024-09-11T06:48:43.127 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_7 --debug-rbd=20 --debug-journaler=20 2024-09-11T06:48:44.573 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:48:44.566+0000 7fb2ca654640 -1 reset not still connected to 0x55ed0e47ba00 2024-09-11T06:48:44.573 INFO:tasks.ceph.cluster1.osd.2.smithi006.stderr:2024-09-11T06:48:44.566+0000 7fb2ca654640 -1 reset not still connected to 0x55ed11f509c0 2024-09-11T06:48:44.834 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:100 IOs, 40622080 bytes 2024-09-11T06:48:50.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ error_code=124 2024-09-11T06:48:50.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set -e 2024-09-11T06:48:50.136 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T06:48:50.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:48:50.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_present cluster1 mirror image_7 present 2024-09-11T06:48:50.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:48:50.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:48:50.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_7 2024-09-11T06:48:50.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=present 2024-09-11T06:48:50.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:48:50.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:48:50.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:48:50.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_7 2024-09-11T06:48:50.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:48:50.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:48:50.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_7 2024-09-11T06:48:50.137 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-09-11T06:48:50.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:48:50.198 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image_id=10a9d49cbf8f 2024-09-11T06:48:50.199 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T06:48:50.199 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.1 2024-09-11T06:48:50.300 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_present cluster1 mirror image_7 present 10a9d49cbf8f 2024-09-11T06:48:50.300 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:48:50.300 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:48:50.300 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_7 2024-09-11T06:48:50.300 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=present 2024-09-11T06:48:50.300 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id=10a9d49cbf8f 2024-09-11T06:48:50.300 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=deleted 2024-09-11T06:48:50.301 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_image_id 2024-09-11T06:48:50.301 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_7 2024-09-11T06:48:50.301 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:48:50.301 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:48:50.301 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_7 2024-09-11T06:48:50.301 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-09-11T06:48:50.301 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:48:50.363 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_image_id=10a9d49cbf8f 2024-09-11T06:48:50.363 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 10a9d49cbf8f 2024-09-11T06:48:50.363 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -z 10a9d49cbf8f -o 10a9d49cbf8f = 10a9d49cbf8f 2024-09-11T06:48:50.364 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_state=present 2024-09-11T06:48:50.364 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test present = present 2024-09-11T06:48:50.364 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:48:50.364 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ disable_mirror cluster2 mirror image_7 2024-09-11T06:48:50.364 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:48:50.364 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:48:50.364 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_7 2024-09-11T06:48:50.364 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_7 2024-09-11T06:49:31.735 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:Mirroring disabled 2024-09-11T06:49:31.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_present cluster1 mirror image_7 deleted 2024-09-11T06:49:31.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:49:31.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:49:31.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_7 2024-09-11T06:49:31.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=deleted 2024-09-11T06:49:31.742 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:49:31.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:49:31.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:49:31.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_7 2024-09-11T06:49:31.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:49:31.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:49:31.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_7 2024-09-11T06:49:31.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-09-11T06:49:31.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:49:31.804 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image_id=10a9d49cbf8f 2024-09-11T06:49:31.804 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T06:49:31.804 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.1 2024-09-11T06:49:31.905 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_present cluster1 mirror image_7 deleted 10a9d49cbf8f 2024-09-11T06:49:31.905 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:49:31.905 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:49:31.905 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_7 2024-09-11T06:49:31.906 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=deleted 2024-09-11T06:49:31.906 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id=10a9d49cbf8f 2024-09-11T06:49:31.906 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=deleted 2024-09-11T06:49:31.906 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_image_id 2024-09-11T06:49:31.906 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_7 2024-09-11T06:49:31.907 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:49:31.907 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:49:31.907 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_7 2024-09-11T06:49:31.907 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-09-11T06:49:31.907 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:49:31.956 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_7: (2) No such file or directory 2024-09-11T06:49:31.960 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_image_id= 2024-09-11T06:49:31.960 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:49:31.960 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test deleted = deleted 2024-09-11T06:49:31.960 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:49:31.960 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ purge_snapshots cluster2 mirror image_7 2024-09-11T06:49:31.960 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:49:31.960 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:49:31.961 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_7 2024-09-11T06:49:31.961 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 snap purge mirror/image_7 2024-09-11T06:49:32.008 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image_retry cluster2 mirror image_7 2024-09-11T06:49:32.008 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:49:32.008 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:49:32.008 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_7 2024-09-11T06:49:32.008 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T06:49:32.008 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:49:32.009 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image cluster2 mirror image_7 2024-09-11T06:49:32.009 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:49:32.009 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:49:32.009 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_7 2024-09-11T06:49:32.009 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_7 2024-09-11T06:49:32.056 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 rm mirror/image_7 2024-09-11T06:49:32.327 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-11T06:49:32.331 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:49:32.332 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 10` 2024-09-11T06:49:32.332 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_8 2024-09-11T06:49:32.332 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_image cluster2 mirror image_8 512M 2024-09-11T06:49:32.332 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:49:32.332 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:49:32.332 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:49:32.332 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:49:32.332 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_8 2024-09-11T06:49:32.332 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:49:32.332 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local size=128 2024-09-11T06:49:32.332 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n 512M ']' 2024-09-11T06:49:32.332 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ size=512M 2024-09-11T06:49:32.333 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:49:32.333 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,journaling mirror/image_8 2024-09-11T06:49:32.391 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ enable_mirror cluster2 mirror image_8 2024-09-11T06:49:32.391 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:49:32.391 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:49:32.391 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_8 2024-09-11T06:49:32.391 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local mode=journal 2024-09-11T06:49:32.391 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_8 journal 2024-09-11T06:49:32.439 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:Mirroring enabled 2024-09-11T06:49:32.445 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 info mirror/image_8 2024-09-11T06:49:32.485 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:rbd image 'image_8': 2024-09-11T06:49:32.485 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: size 512 MiB in 128 objects 2024-09-11T06:49:32.485 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: order 22 (4 MiB objects) 2024-09-11T06:49:32.485 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: snapshot_count: 0 2024-09-11T06:49:32.485 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: id: 2ed945627b18 2024-09-11T06:49:32.485 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: block_name_prefix: rbd_data.2ed945627b18 2024-09-11T06:49:32.485 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: format: 2 2024-09-11T06:49:32.485 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: features: layering, exclusive-lock, journaling 2024-09-11T06:49:32.485 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: op_features: 2024-09-11T06:49:32.485 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: flags: 2024-09-11T06:49:32.485 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: create_timestamp: Wed Sep 11 06:49:32 2024 2024-09-11T06:49:32.485 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: access_timestamp: Wed Sep 11 06:49:32 2024 2024-09-11T06:49:32.486 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: modify_timestamp: Wed Sep 11 06:49:32 2024 2024-09-11T06:49:32.486 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: journal: 2ed945627b18 2024-09-11T06:49:32.486 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring state: enabled 2024-09-11T06:49:32.486 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring mode: journal 2024-09-11T06:49:32.486 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring global id: a577b8ba-e117-426f-a54a-bd40a9ace736 2024-09-11T06:49:32.486 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring primary: true 2024-09-11T06:49:32.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ stress_write_image cluster2 mirror image_8 2024-09-11T06:49:32.488 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:49:32.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:49:32.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_8 2024-09-11T06:49:32.489 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T06:49:32.490 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local duration=14 2024-09-11T06:49:32.490 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set +e 2024-09-11T06:49:32.491 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_8 --debug-rbd=20 --debug-journaler=20 2024-09-11T06:49:33.408 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:100 IOs, 11236352 bytes 2024-09-11T06:49:35.458 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:200 IOs, 40079360 bytes 2024-09-11T06:49:39.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:300 IOs, 99301376 bytes 2024-09-11T06:49:40.925 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:400 IOs, 112936960 bytes 2024-09-11T06:49:45.780 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:500 IOs, 197018624 bytes 2024-09-11T06:49:46.499 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ error_code=124 2024-09-11T06:49:46.499 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set -e 2024-09-11T06:49:46.499 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T06:49:46.499 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:49:46.499 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_present cluster1 mirror image_8 present 2024-09-11T06:49:46.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:49:46.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:49:46.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_8 2024-09-11T06:49:46.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=present 2024-09-11T06:49:46.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:49:46.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:49:46.500 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:49:46.501 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_8 2024-09-11T06:49:46.501 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:49:46.501 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:49:46.501 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_8 2024-09-11T06:49:46.501 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-09-11T06:49:46.501 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:49:46.564 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image_id=10a65d7bf112 2024-09-11T06:49:46.564 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T06:49:46.564 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.1 2024-09-11T06:49:46.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_present cluster1 mirror image_8 present 10a65d7bf112 2024-09-11T06:49:46.666 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:49:46.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:49:46.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_8 2024-09-11T06:49:46.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=present 2024-09-11T06:49:46.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id=10a65d7bf112 2024-09-11T06:49:46.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=deleted 2024-09-11T06:49:46.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_image_id 2024-09-11T06:49:46.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_8 2024-09-11T06:49:46.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:49:46.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:49:46.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_8 2024-09-11T06:49:46.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-09-11T06:49:46.667 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:49:46.723 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_image_id=10a65d7bf112 2024-09-11T06:49:46.723 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 10a65d7bf112 2024-09-11T06:49:46.723 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -z 10a65d7bf112 -o 10a65d7bf112 = 10a65d7bf112 2024-09-11T06:49:46.723 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_state=present 2024-09-11T06:49:46.723 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test present = present 2024-09-11T06:49:46.723 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:49:46.723 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ disable_mirror cluster2 mirror image_8 2024-09-11T06:49:46.723 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:49:46.724 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:49:46.724 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_8 2024-09-11T06:49:46.724 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_8 2024-09-11T06:50:17.031 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:Mirroring disabled 2024-09-11T06:50:17.041 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_present cluster1 mirror image_8 deleted 2024-09-11T06:50:17.041 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:50:17.041 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:50:17.041 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_8 2024-09-11T06:50:17.041 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=deleted 2024-09-11T06:50:17.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:50:17.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:50:17.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:50:17.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_8 2024-09-11T06:50:17.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:50:17.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:50:17.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_8 2024-09-11T06:50:17.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-09-11T06:50:17.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:50:17.078 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_8: (2) No such file or directory 2024-09-11T06:50:17.082 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image_id= 2024-09-11T06:50:17.082 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T06:50:17.082 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.1 2024-09-11T06:50:17.184 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_present cluster1 mirror image_8 deleted '' 2024-09-11T06:50:17.184 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:50:17.184 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:50:17.184 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_8 2024-09-11T06:50:17.184 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=deleted 2024-09-11T06:50:17.184 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:50:17.184 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=deleted 2024-09-11T06:50:17.184 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_image_id 2024-09-11T06:50:17.185 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_8 2024-09-11T06:50:17.185 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:50:17.185 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:50:17.185 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_8 2024-09-11T06:50:17.185 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-09-11T06:50:17.185 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:50:17.229 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_8: (2) No such file or directory 2024-09-11T06:50:17.234 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_image_id= 2024-09-11T06:50:17.234 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:50:17.234 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test deleted = deleted 2024-09-11T06:50:17.234 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:50:17.234 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ purge_snapshots cluster2 mirror image_8 2024-09-11T06:50:17.234 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:50:17.234 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:50:17.234 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_8 2024-09-11T06:50:17.234 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 snap purge mirror/image_8 2024-09-11T06:50:17.282 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image_retry cluster2 mirror image_8 2024-09-11T06:50:17.282 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:50:17.282 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:50:17.282 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_8 2024-09-11T06:50:17.282 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T06:50:17.282 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:50:17.283 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image cluster2 mirror image_8 2024-09-11T06:50:17.283 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:50:17.283 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:50:17.283 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_8 2024-09-11T06:50:17.283 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_8 2024-09-11T06:50:17.330 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 rm mirror/image_8 2024-09-11T06:50:17.584 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-11T06:50:17.588 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:50:17.589 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 10` 2024-09-11T06:50:17.589 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_9 2024-09-11T06:50:17.589 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_image cluster2 mirror image_9 512M 2024-09-11T06:50:17.589 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:50:17.589 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:50:17.589 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:50:17.589 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:50:17.589 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_9 2024-09-11T06:50:17.589 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:50:17.589 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local size=128 2024-09-11T06:50:17.589 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n 512M ']' 2024-09-11T06:50:17.589 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ size=512M 2024-09-11T06:50:17.589 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:50:17.590 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,journaling mirror/image_9 2024-09-11T06:50:17.650 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ enable_mirror cluster2 mirror image_9 2024-09-11T06:50:17.651 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:50:17.651 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:50:17.651 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_9 2024-09-11T06:50:17.651 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local mode=journal 2024-09-11T06:50:17.651 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_9 journal 2024-09-11T06:50:17.699 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:Mirroring enabled 2024-09-11T06:50:17.705 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 info mirror/image_9 2024-09-11T06:50:17.743 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:rbd image 'image_9': 2024-09-11T06:50:17.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: size 512 MiB in 128 objects 2024-09-11T06:50:17.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: order 22 (4 MiB objects) 2024-09-11T06:50:17.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: snapshot_count: 0 2024-09-11T06:50:17.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: id: 2ef19b2defec 2024-09-11T06:50:17.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: block_name_prefix: rbd_data.2ef19b2defec 2024-09-11T06:50:17.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: format: 2 2024-09-11T06:50:17.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: features: layering, exclusive-lock, journaling 2024-09-11T06:50:17.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: op_features: 2024-09-11T06:50:17.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: flags: 2024-09-11T06:50:17.744 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: create_timestamp: Wed Sep 11 06:50:17 2024 2024-09-11T06:50:17.745 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: access_timestamp: Wed Sep 11 06:50:17 2024 2024-09-11T06:50:17.745 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: modify_timestamp: Wed Sep 11 06:50:17 2024 2024-09-11T06:50:17.745 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: journal: 2ef19b2defec 2024-09-11T06:50:17.745 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring state: enabled 2024-09-11T06:50:17.745 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring mode: journal 2024-09-11T06:50:17.745 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring global id: beb57aeb-16e9-480d-a691-6b39c0dff576 2024-09-11T06:50:17.745 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring primary: true 2024-09-11T06:50:17.748 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ stress_write_image cluster2 mirror image_9 2024-09-11T06:50:17.748 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:50:17.748 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:50:17.748 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_9 2024-09-11T06:50:17.748 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T06:50:17.749 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local duration=9 2024-09-11T06:50:17.749 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set +e 2024-09-11T06:50:17.749 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_9 --debug-rbd=20 --debug-journaler=20 2024-09-11T06:50:18.265 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:100 IOs, 4636672 bytes 2024-09-11T06:50:19.994 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:200 IOs, 29856768 bytes 2024-09-11T06:50:22.466 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:300 IOs, 68502528 bytes 2024-09-11T06:50:23.451 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:400 IOs, 69410816 bytes 2024-09-11T06:50:25.493 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:500 IOs, 104856576 bytes 2024-09-11T06:50:26.759 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ error_code=124 2024-09-11T06:50:26.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set -e 2024-09-11T06:50:26.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T06:50:26.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:50:26.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_present cluster1 mirror image_9 present 2024-09-11T06:50:26.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:50:26.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:50:26.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_9 2024-09-11T06:50:26.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=present 2024-09-11T06:50:26.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:50:26.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:50:26.760 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:50:26.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_9 2024-09-11T06:50:26.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:50:26.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:50:26.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_9 2024-09-11T06:50:26.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:50:26.761 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-09-11T06:50:26.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image_id=10a6ee05450 2024-09-11T06:50:26.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T06:50:26.821 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.1 2024-09-11T06:50:26.923 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_present cluster1 mirror image_9 present 10a6ee05450 2024-09-11T06:50:26.923 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:50:26.923 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:50:26.923 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_9 2024-09-11T06:50:26.923 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=present 2024-09-11T06:50:26.923 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id=10a6ee05450 2024-09-11T06:50:26.923 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=deleted 2024-09-11T06:50:26.923 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_image_id 2024-09-11T06:50:26.923 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_9 2024-09-11T06:50:26.923 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:50:26.924 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:50:26.924 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_9 2024-09-11T06:50:26.924 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-09-11T06:50:26.924 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:50:26.989 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_image_id=10a6ee05450 2024-09-11T06:50:26.989 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 10a6ee05450 2024-09-11T06:50:26.989 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -z 10a6ee05450 -o 10a6ee05450 = 10a6ee05450 2024-09-11T06:50:26.989 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_state=present 2024-09-11T06:50:26.989 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test present = present 2024-09-11T06:50:26.990 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:50:26.990 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ disable_mirror cluster2 mirror image_9 2024-09-11T06:50:26.990 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:50:26.990 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:50:26.990 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_9 2024-09-11T06:50:26.990 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_9 2024-09-11T06:50:56.990 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:Mirroring disabled 2024-09-11T06:50:57.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_present cluster1 mirror image_9 deleted 2024-09-11T06:50:57.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:50:57.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:50:57.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_9 2024-09-11T06:50:57.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=deleted 2024-09-11T06:50:57.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:50:57.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:50:57.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:50:57.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_9 2024-09-11T06:50:57.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:50:57.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:50:57.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_9 2024-09-11T06:50:57.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-09-11T06:50:57.001 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:50:57.038 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_9: (2) No such file or directory 2024-09-11T06:50:57.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image_id= 2024-09-11T06:50:57.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T06:50:57.042 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.1 2024-09-11T06:50:57.143 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_present cluster1 mirror image_9 deleted '' 2024-09-11T06:50:57.143 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:50:57.143 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:50:57.144 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_9 2024-09-11T06:50:57.144 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=deleted 2024-09-11T06:50:57.144 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:50:57.144 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=deleted 2024-09-11T06:50:57.144 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_image_id 2024-09-11T06:50:57.145 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_9 2024-09-11T06:50:57.145 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:50:57.145 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:50:57.145 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_9 2024-09-11T06:50:57.145 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-09-11T06:50:57.145 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:50:57.188 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_9: (2) No such file or directory 2024-09-11T06:50:57.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_image_id= 2024-09-11T06:50:57.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:50:57.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test deleted = deleted 2024-09-11T06:50:57.192 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:50:57.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ purge_snapshots cluster2 mirror image_9 2024-09-11T06:50:57.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:50:57.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:50:57.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_9 2024-09-11T06:50:57.193 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 snap purge mirror/image_9 2024-09-11T06:50:57.240 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image_retry cluster2 mirror image_9 2024-09-11T06:50:57.240 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:50:57.240 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:50:57.240 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_9 2024-09-11T06:50:57.240 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T06:50:57.240 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:50:57.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image cluster2 mirror image_9 2024-09-11T06:50:57.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:50:57.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:50:57.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_9 2024-09-11T06:50:57.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_9 2024-09-11T06:50:57.288 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 rm mirror/image_9 2024-09-11T06:50:57.538 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-11T06:50:57.543 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:50:57.543 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for i in `seq 1 10` 2024-09-11T06:50:57.543 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image=image_10 2024-09-11T06:50:57.543 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ create_image cluster2 mirror image_10 512M 2024-09-11T06:50:57.543 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:50:57.543 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:50:57.543 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:50:57.543 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:50:57.544 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_10 2024-09-11T06:50:57.544 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:50:57.544 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local size=128 2024-09-11T06:50:57.544 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n 512M ']' 2024-09-11T06:50:57.544 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ size=512M 2024-09-11T06:50:57.544 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ shift 2024-09-11T06:50:57.544 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,journaling mirror/image_10 2024-09-11T06:50:57.602 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ enable_mirror cluster2 mirror image_10 2024-09-11T06:50:57.602 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:50:57.602 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:50:57.602 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_10 2024-09-11T06:50:57.602 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local mode=journal 2024-09-11T06:50:57.602 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_10 journal 2024-09-11T06:50:57.651 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:Mirroring enabled 2024-09-11T06:50:57.657 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 info mirror/image_10 2024-09-11T06:50:57.696 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:rbd image 'image_10': 2024-09-11T06:50:57.697 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: size 512 MiB in 128 objects 2024-09-11T06:50:57.697 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: order 22 (4 MiB objects) 2024-09-11T06:50:57.697 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: snapshot_count: 0 2024-09-11T06:50:57.697 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: id: 2f081f97be85 2024-09-11T06:50:57.697 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: block_name_prefix: rbd_data.2f081f97be85 2024-09-11T06:50:57.697 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: format: 2 2024-09-11T06:50:57.697 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: features: layering, exclusive-lock, journaling 2024-09-11T06:50:57.697 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: op_features: 2024-09-11T06:50:57.697 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: flags: 2024-09-11T06:50:57.697 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: create_timestamp: Wed Sep 11 06:50:57 2024 2024-09-11T06:50:57.697 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: access_timestamp: Wed Sep 11 06:50:57 2024 2024-09-11T06:50:57.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: modify_timestamp: Wed Sep 11 06:50:57 2024 2024-09-11T06:50:57.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: journal: 2f081f97be85 2024-09-11T06:50:57.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring state: enabled 2024-09-11T06:50:57.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring mode: journal 2024-09-11T06:50:57.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring global id: 6de1721d-1233-47de-946a-3259bc1bad68 2024-09-11T06:50:57.698 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout: mirroring primary: true 2024-09-11T06:50:57.700 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ stress_write_image cluster2 mirror image_10 2024-09-11T06:50:57.700 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:50:57.700 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:50:57.700 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_10 2024-09-11T06:50:57.701 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T06:50:57.702 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local duration=8 2024-09-11T06:50:57.702 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set +e 2024-09-11T06:50:57.702 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_10 --debug-rbd=20 --debug-journaler=20 2024-09-11T06:51:01.861 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:100 IOs, 50661376 bytes 2024-09-11T06:51:05.715 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ error_code=124 2024-09-11T06:51:05.716 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set -e 2024-09-11T06:51:05.716 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T06:51:05.716 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:51:05.716 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_present cluster1 mirror image_10 present 2024-09-11T06:51:05.716 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:51:05.716 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:51:05.716 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_10 2024-09-11T06:51:05.716 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=present 2024-09-11T06:51:05.716 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:51:05.716 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:51:05.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:51:05.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_10 2024-09-11T06:51:05.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:51:05.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:51:05.717 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_10 2024-09-11T06:51:05.718 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-09-11T06:51:05.721 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:51:05.799 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image_id=10a62c30dcd5 2024-09-11T06:51:05.799 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T06:51:05.799 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.1 2024-09-11T06:51:05.900 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_present cluster1 mirror image_10 present 10a62c30dcd5 2024-09-11T06:51:05.900 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:51:05.900 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:51:05.900 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_10 2024-09-11T06:51:05.900 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=present 2024-09-11T06:51:05.900 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id=10a62c30dcd5 2024-09-11T06:51:05.900 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=deleted 2024-09-11T06:51:05.900 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_image_id 2024-09-11T06:51:05.901 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_10 2024-09-11T06:51:05.901 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:51:05.901 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:51:05.901 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_10 2024-09-11T06:51:05.901 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-09-11T06:51:05.902 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:51:05.966 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_image_id=10a62c30dcd5 2024-09-11T06:51:05.966 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n 10a62c30dcd5 2024-09-11T06:51:05.967 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -z 10a62c30dcd5 -o 10a62c30dcd5 = 10a62c30dcd5 2024-09-11T06:51:05.967 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_state=present 2024-09-11T06:51:05.967 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test present = present 2024-09-11T06:51:05.967 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:51:05.967 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ disable_mirror cluster2 mirror image_10 2024-09-11T06:51:05.968 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:51:05.968 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:51:05.968 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_10 2024-09-11T06:51:05.968 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_10 2024-09-11T06:51:35.984 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:Mirroring disabled 2024-09-11T06:51:35.994 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_present cluster1 mirror image_10 deleted 2024-09-11T06:51:35.994 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:51:35.994 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:51:35.994 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_10 2024-09-11T06:51:35.994 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local state=deleted 2024-09-11T06:51:35.994 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:51:35.994 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local s 2024-09-11T06:51:35.994 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:51:35.995 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_10 2024-09-11T06:51:35.995 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:51:35.995 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:51:35.995 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_10 2024-09-11T06:51:35.995 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-09-11T06:51:35.995 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:51:36.034 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_10: (2) No such file or directory 2024-09-11T06:51:36.037 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ image_id= 2024-09-11T06:51:36.037 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T06:51:36.038 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0.1 2024-09-11T06:51:36.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test_image_present cluster1 mirror image_10 deleted '' 2024-09-11T06:51:36.139 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:51:36.139 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:51:36.139 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_10 2024-09-11T06:51:36.139 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local test_state=deleted 2024-09-11T06:51:36.139 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image_id= 2024-09-11T06:51:36.139 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_state=deleted 2024-09-11T06:51:36.139 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local current_image_id 2024-09-11T06:51:36.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ get_image_id cluster1 mirror image_10 2024-09-11T06:51:36.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:51:36.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:51:36.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local image=image_10 2024-09-11T06:51:36.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-09-11T06:51:36.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T06:51:36.185 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:rbd: error opening image image_10: (2) No such file or directory 2024-09-11T06:51:36.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ current_image_id= 2024-09-11T06:51:36.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:51:36.190 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test deleted = deleted 2024-09-11T06:51:36.190 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:51:36.190 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ purge_snapshots cluster2 mirror image_10 2024-09-11T06:51:36.190 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:51:36.190 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:51:36.190 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_10 2024-09-11T06:51:36.190 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster cluster2 snap purge mirror/image_10 2024-09-11T06:51:36.239 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image_retry cluster2 mirror image_10 2024-09-11T06:51:36.239 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:51:36.239 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:51:36.239 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_10 2024-09-11T06:51:36.239 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T06:51:36.239 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:51:36.240 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ remove_image cluster2 mirror image_10 2024-09-11T06:51:36.240 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:51:36.240 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:51:36.240 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local image=image_10 2024-09-11T06:51:36.241 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_10 2024-09-11T06:51:36.288 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rbd --cluster=cluster2 rm mirror/image_10 2024-09-11T06:51:36.545 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-11T06:51:36.549 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:51:36.550 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ testlog 'TEST: check if removed images'\'' OMAP are removed' 2024-09-11T06:51:36.550 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ tee -a /tmp/tmp.Dxwd1ESqKa/rbd-mirror.test.log 2024-09-11T06:51:36.550 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ date '+%F %T' 2024-09-11T06:51:36.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ echo 2024-09-11 06:51:36 TEST: check if removed 'images'\''' OMAP are removed 2024-09-11T06:51:36.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:2024-09-11 06:51:36 TEST: check if removed images' OMAP are removed 2024-09-11T06:51:36.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_in_omap cluster1 mirror 2024-09-11T06:51:36.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:51:36.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:51:36.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirroring status_global 2024-09-11T06:51:36.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:51:36.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:51:36.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local obj_name=rbd_mirroring 2024-09-11T06:51:36.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local filter=status_global 2024-09-11T06:51:36.553 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-11T06:51:36.553 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:51:36.553 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set +e 2024-09-11T06:51:36.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirroring status_global 2024-09-11T06:51:36.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:51:36.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:51:36.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local obj_name=rbd_mirroring 2024-09-11T06:51:36.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local filter=status_global 2024-09-11T06:51:36.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ list_omap_keys cluster1 mirror rbd_mirroring 2024-09-11T06:51:36.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:51:36.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:51:36.555 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local obj_name=rbd_mirroring 2024-09-11T06:51:36.555 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirroring 2024-09-11T06:51:36.555 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -c status_global 2024-09-11T06:51:36.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 0 = 0 2024-09-11T06:51:36.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ error_code=0 2024-09-11T06:51:36.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set -e 2024-09-11T06:51:36.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 -eq 0 ']' 2024-09-11T06:51:36.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:51:36.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirroring image_ 2024-09-11T06:51:36.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:51:36.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:51:36.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local obj_name=rbd_mirroring 2024-09-11T06:51:36.720 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local filter=image_ 2024-09-11T06:51:36.721 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-11T06:51:36.721 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:51:36.722 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set +e 2024-09-11T06:51:36.722 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirroring image_ 2024-09-11T06:51:36.722 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:51:36.722 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:51:36.722 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local obj_name=rbd_mirroring 2024-09-11T06:51:36.722 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local filter=image_ 2024-09-11T06:51:36.722 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ list_omap_keys cluster1 mirror rbd_mirroring 2024-09-11T06:51:36.722 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:51:36.722 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -c image_ 2024-09-11T06:51:36.723 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:51:36.723 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local obj_name=rbd_mirroring 2024-09-11T06:51:36.723 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirroring 2024-09-11T06:51:36.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 0 = 0 2024-09-11T06:51:36.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ error_code=0 2024-09-11T06:51:36.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set -e 2024-09-11T06:51:36.763 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 -eq 0 ']' 2024-09-11T06:51:36.764 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:51:36.764 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirror_leader image_map 2024-09-11T06:51:36.764 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:51:36.764 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:51:36.764 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local obj_name=rbd_mirror_leader 2024-09-11T06:51:36.764 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local filter=image_map 2024-09-11T06:51:36.764 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-11T06:51:36.764 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:51:36.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set +e 2024-09-11T06:51:36.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirror_leader image_map 2024-09-11T06:51:36.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:51:36.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:51:36.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local obj_name=rbd_mirror_leader 2024-09-11T06:51:36.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local filter=image_map 2024-09-11T06:51:36.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ list_omap_keys cluster1 mirror rbd_mirror_leader 2024-09-11T06:51:36.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:51:36.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:51:36.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local obj_name=rbd_mirror_leader 2024-09-11T06:51:36.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirror_leader 2024-09-11T06:51:36.766 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -c image_map 2024-09-11T06:51:36.807 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 0 = 0 2024-09-11T06:51:36.807 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ error_code=0 2024-09-11T06:51:36.807 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set -e 2024-09-11T06:51:36.807 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 -eq 0 ']' 2024-09-11T06:51:36.807 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:51:36.807 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_image_in_omap cluster2 mirror 2024-09-11T06:51:36.807 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:51:36.807 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:51:36.808 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirroring status_global 2024-09-11T06:51:36.808 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:51:36.808 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:51:36.808 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local obj_name=rbd_mirroring 2024-09-11T06:51:36.808 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local filter=status_global 2024-09-11T06:51:36.808 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-11T06:51:36.808 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:51:36.808 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set +e 2024-09-11T06:51:36.808 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ count_omap_keys_with_filter cluster2 mirror rbd_mirroring status_global 2024-09-11T06:51:36.808 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:51:36.808 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:51:36.808 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local obj_name=rbd_mirroring 2024-09-11T06:51:36.808 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local filter=status_global 2024-09-11T06:51:36.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ list_omap_keys cluster2 mirror rbd_mirroring 2024-09-11T06:51:36.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:51:36.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:51:36.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -c status_global 2024-09-11T06:51:36.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local obj_name=rbd_mirroring 2024-09-11T06:51:36.809 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rados --cluster cluster2 -p mirror listomapkeys rbd_mirroring 2024-09-11T06:51:36.843 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 0 = 0 2024-09-11T06:51:36.843 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ error_code=0 2024-09-11T06:51:36.843 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set -e 2024-09-11T06:51:36.843 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 -eq 0 ']' 2024-09-11T06:51:36.843 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:51:36.843 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirroring image_ 2024-09-11T06:51:36.843 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:51:36.843 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:51:36.844 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local obj_name=rbd_mirroring 2024-09-11T06:51:36.844 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local filter=image_ 2024-09-11T06:51:36.844 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-11T06:51:36.844 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:51:36.844 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set +e 2024-09-11T06:51:36.845 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ count_omap_keys_with_filter cluster2 mirror rbd_mirroring image_ 2024-09-11T06:51:36.845 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:51:36.845 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:51:36.845 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local obj_name=rbd_mirroring 2024-09-11T06:51:36.845 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local filter=image_ 2024-09-11T06:51:36.845 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ list_omap_keys cluster2 mirror rbd_mirroring 2024-09-11T06:51:36.846 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:51:36.846 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:51:36.846 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local obj_name=rbd_mirroring 2024-09-11T06:51:36.846 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rados --cluster cluster2 -p mirror listomapkeys rbd_mirroring 2024-09-11T06:51:36.846 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -c image_ 2024-09-11T06:51:36.879 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 0 = 0 2024-09-11T06:51:36.879 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ error_code=0 2024-09-11T06:51:36.880 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set -e 2024-09-11T06:51:36.880 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 -eq 0 ']' 2024-09-11T06:51:36.880 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:51:36.880 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirror_leader image_map 2024-09-11T06:51:36.880 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:51:36.880 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pool=mirror 2024-09-11T06:51:36.880 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local obj_name=rbd_mirror_leader 2024-09-11T06:51:36.880 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local filter=image_map 2024-09-11T06:51:36.880 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-11T06:51:36.880 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 0 2024-09-11T06:51:36.880 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set +e 2024-09-11T06:51:36.881 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ count_omap_keys_with_filter cluster2 mirror rbd_mirror_leader image_map 2024-09-11T06:51:36.882 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:51:36.882 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:51:36.882 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local obj_name=rbd_mirror_leader 2024-09-11T06:51:36.882 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local filter=image_map 2024-09-11T06:51:36.882 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ list_omap_keys cluster2 mirror rbd_mirror_leader 2024-09-11T06:51:36.882 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:51:36.882 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local pool=mirror 2024-09-11T06:51:36.882 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local obj_name=rbd_mirror_leader 2024-09-11T06:51:36.882 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ grep -c image_map 2024-09-11T06:51:36.882 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ rados --cluster cluster2 -p mirror listomapkeys rbd_mirror_leader 2024-09-11T06:51:36.915 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test 0 = 0 2024-09-11T06:51:36.915 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ error_code=0 2024-09-11T06:51:36.915 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set -e 2024-09-11T06:51:36.915 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 -eq 0 ']' 2024-09-11T06:51:36.915 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ return 0 2024-09-11T06:51:36.915 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ cleanup 0 2024-09-11T06:51:36.915 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local error_code=0 2024-09-11T06:51:36.915 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ set +e 2024-09-11T06:51:36.915 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 -ne 0 ']' 2024-09-11T06:51:36.916 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -z '' ']' 2024-09-11T06:51:36.916 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster instance 2024-09-11T06:51:36.916 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ CEPH_ARGS= 2024-09-11T06:51:36.916 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --cluster cluster1 osd pool rm mirror mirror --yes-i-really-really-mean-it 2024-09-11T06:51:37.511 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:pool 'mirror' does not exist 2024-09-11T06:51:37.527 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ CEPH_ARGS= 2024-09-11T06:51:37.527 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --cluster cluster2 osd pool rm mirror mirror --yes-i-really-really-mean-it 2024-09-11T06:51:38.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:pool 'mirror' does not exist 2024-09-11T06:51:39.046 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ CEPH_ARGS= 2024-09-11T06:51:39.047 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --cluster cluster1 osd pool rm mirror_parent mirror_parent --yes-i-really-really-mean-it 2024-09-11T06:51:40.526 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:pool 'mirror_parent' does not exist 2024-09-11T06:51:40.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ CEPH_ARGS= 2024-09-11T06:51:40.542 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ceph --cluster cluster2 osd pool rm mirror_parent mirror_parent --yes-i-really-really-mean-it 2024-09-11T06:51:40.978 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:pool 'mirror_parent' does not exist 2024-09-11T06:51:40.995 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for cluster in "${CLUSTER1}" "${CLUSTER2}" 2024-09-11T06:51:40.995 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ stop_mirrors cluster1 2024-09-11T06:51:40.995 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1 2024-09-11T06:51:40.996 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local sig= 2024-09-11T06:51:40.996 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:51:40.997 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:51:40.997 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ stop_mirror cluster1:0 '' 2024-09-11T06:51:40.997 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:0 2024-09-11T06:51:40.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local sig= 2024-09-11T06:51:40.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:51:40.998 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pid 2024-09-11T06:51:40.999 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ daemon_pid_file cluster1:0 2024-09-11T06:51:40.999 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster=cluster1:0 2024-09-11T06:51:40.999 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance 2024-09-11T06:51:40.999 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:51:40.999 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local val=cluster1:0 2024-09-11T06:51:41.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster_var_name=cluster 2024-09-11T06:51:41.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance_var_name=instance 2024-09-11T06:51:41.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:51:41.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:51:41.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T06:51:41.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval cluster=cluster1 2024-09-11T06:51:41.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ cluster=cluster1 2024-09-11T06:51:41.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval instance=0 2024-09-11T06:51:41.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ instance=0 2024-09-11T06:51:41.000 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'pid file' 2024-09-11T06:51:41.017 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2024-09-11T06:51:41.017 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cat /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2024-09-11T06:51:41.031 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ pid=29729 2024-09-11T06:51:41.031 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n 29729 ']' 2024-09-11T06:51:41.031 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ kill 29729 2024-09-11T06:51:41.031 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 16 32 2024-09-11T06:51:41.031 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:51:42.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ps auxww 2024-09-11T06:51:42.033 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ awk -v pid=29729 '$2 == pid {print; exit 1}' 2024-09-11T06:51:42.096 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:51:42.097 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ps auxww 2024-09-11T06:51:42.097 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ awk -v pid=29729 '$2 == pid {print; exit 1}' 2024-09-11T06:51:42.113 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-11T06:51:42.113 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:51:42.113 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:51:42.113 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:51:42.114 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:51:42.114 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:51:42.114 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:51:42.114 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:51:42.114 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:51:42.114 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:51:42.114 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:51:42.114 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:51:42.114 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:51:42.114 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:51:42.114 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:51:42.115 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:51:42.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:51:42.138 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:51:42.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:0 cluster2 2024-09-11T06:51:42.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:0 2024-09-11T06:51:42.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:51:42.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:51:42.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T06:51:42.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:51:42.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:51:42.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:51:42.140 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:51:42.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:51:42.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:51:42.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:51:42.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:51:42.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:51:42.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:51:42.141 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T06:51:42.163 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:51:42.163 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T06:51:42.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_pid_file cluster1:0 2024-09-11T06:51:42.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1:0 2024-09-11T06:51:42.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:51:42.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T06:51:42.165 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:0 2024-09-11T06:51:42.166 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=cluster 2024-09-11T06:51:42.166 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:51:42.166 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:51:42.166 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:51:42.166 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T06:51:42.166 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval cluster=cluster1 2024-09-11T06:51:42.166 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:51:42.166 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:51:42.166 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:51:42.166 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'pid file' 2024-09-11T06:51:42.187 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2024-09-11T06:51:42.187 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2024-09-11T06:51:42.188 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:51:42.188 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ stop_mirror cluster1:1 '' 2024-09-11T06:51:42.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster1:1 2024-09-11T06:51:42.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local sig= 2024-09-11T06:51:42.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:51:42.189 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pid 2024-09-11T06:51:42.190 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ daemon_pid_file cluster1:1 2024-09-11T06:51:42.190 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster=cluster1:1 2024-09-11T06:51:42.190 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance 2024-09-11T06:51:42.190 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ set_cluster_instance cluster1:1 cluster instance 2024-09-11T06:51:42.190 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local val=cluster1:1 2024-09-11T06:51:42.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster_var_name=cluster 2024-09-11T06:51:42.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance_var_name=instance 2024-09-11T06:51:42.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:51:42.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=1 2024-09-11T06:51:42.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-11T06:51:42.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval cluster=cluster1 2024-09-11T06:51:42.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ cluster=cluster1 2024-09-11T06:51:42.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval instance=1 2024-09-11T06:51:42.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ instance=1 2024-09-11T06:51:42.191 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'pid file' 2024-09-11T06:51:42.214 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2024-09-11T06:51:42.214 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cat /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2024-09-11T06:51:42.215 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ pid=29757 2024-09-11T06:51:42.216 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n 29757 ']' 2024-09-11T06:51:42.216 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ kill 29757 2024-09-11T06:51:42.216 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 16 32 2024-09-11T06:51:42.216 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:51:43.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ps auxww 2024-09-11T06:51:43.217 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ awk -v pid=29757 '$2 == pid {print; exit 1}' 2024-09-11T06:51:43.233 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:51:43.234 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ps auxww 2024-09-11T06:51:43.234 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ awk -v pid=29757 '$2 == pid {print; exit 1}' 2024-09-11T06:51:43.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:1 cluster1 2024-09-11T06:51:43.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:1 2024-09-11T06:51:43.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:51:43.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:51:43.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T06:51:43.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:1 2024-09-11T06:51:43.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:51:43.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:51:43.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:51:43.251 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=1 2024-09-11T06:51:43.252 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T06:51:43.252 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:51:43.252 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:51:43.252 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=1 2024-09-11T06:51:43.252 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=1 2024-09-11T06:51:43.252 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T06:51:43.279 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:51:43.279 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:51:43.281 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster1:1 cluster2 2024-09-11T06:51:43.281 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster1:1 2024-09-11T06:51:43.282 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:51:43.282 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:51:43.282 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T06:51:43.282 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:1 2024-09-11T06:51:43.282 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:51:43.282 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:51:43.282 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:51:43.282 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=1 2024-09-11T06:51:43.282 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T06:51:43.282 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster1 2024-09-11T06:51:43.282 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster1 2024-09-11T06:51:43.282 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=1 2024-09-11T06:51:43.282 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=1 2024-09-11T06:51:43.283 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T06:51:43.305 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:51:43.305 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T06:51:43.307 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_pid_file cluster1:1 2024-09-11T06:51:43.307 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1:1 2024-09-11T06:51:43.307 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:51:43.307 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-11T06:51:43.307 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster1:1 2024-09-11T06:51:43.307 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=cluster 2024-09-11T06:51:43.307 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:51:43.307 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster1 2024-09-11T06:51:43.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=1 2024-09-11T06:51:43.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T06:51:43.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval cluster=cluster1 2024-09-11T06:51:43.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster1 2024-09-11T06:51:43.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=1 2024-09-11T06:51:43.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=1 2024-09-11T06:51:43.308 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'pid file' 2024-09-11T06:51:43.328 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2024-09-11T06:51:43.328 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2024-09-11T06:51:43.329 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for cluster in "${CLUSTER1}" "${CLUSTER2}" 2024-09-11T06:51:43.329 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ stop_mirrors cluster2 2024-09-11T06:51:43.329 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2 2024-09-11T06:51:43.329 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local sig= 2024-09-11T06:51:43.330 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ seq 0 1 2024-09-11T06:51:43.331 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:51:43.331 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ stop_mirror cluster2:0 '' 2024-09-11T06:51:43.331 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2:0 2024-09-11T06:51:43.331 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local sig= 2024-09-11T06:51:43.331 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:51:43.331 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pid 2024-09-11T06:51:43.333 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ daemon_pid_file cluster2:0 2024-09-11T06:51:43.333 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster=cluster2:0 2024-09-11T06:51:43.333 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance 2024-09-11T06:51:43.333 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ set_cluster_instance cluster2:0 cluster instance 2024-09-11T06:51:43.333 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local val=cluster2:0 2024-09-11T06:51:43.333 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster_var_name=cluster 2024-09-11T06:51:43.334 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance_var_name=instance 2024-09-11T06:51:43.334 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster2 2024-09-11T06:51:43.334 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:51:43.334 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ '[' 0 = cluster2:0 ']' 2024-09-11T06:51:43.334 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval cluster=cluster2 2024-09-11T06:51:43.334 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ cluster=cluster2 2024-09-11T06:51:43.334 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval instance=0 2024-09-11T06:51:43.334 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ instance=0 2024-09-11T06:51:43.334 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'pid file' 2024-09-11T06:51:43.354 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.pid 2024-09-11T06:51:43.354 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cat /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.pid 2024-09-11T06:51:43.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ pid=29802 2024-09-11T06:51:43.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n 29802 ']' 2024-09-11T06:51:43.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ kill 29802 2024-09-11T06:51:43.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 16 32 2024-09-11T06:51:43.355 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:51:44.357 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ps auxww 2024-09-11T06:51:44.357 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ awk -v pid=29802 '$2 == pid {print; exit 1}' 2024-09-11T06:51:44.373 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:51:44.374 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ps auxww 2024-09-11T06:51:44.374 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ awk -v pid=29802 '$2 == pid {print; exit 1}' 2024-09-11T06:51:44.391 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster2:0 cluster1 2024-09-11T06:51:44.391 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster2:0 2024-09-11T06:51:44.391 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:51:44.391 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:51:44.391 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-11T06:51:44.391 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster2:0 2024-09-11T06:51:44.391 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:51:44.391 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:51:44.391 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster2 2024-09-11T06:51:44.391 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:51:44.391 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-11T06:51:44.391 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster2 2024-09-11T06:51:44.392 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster2 2024-09-11T06:51:44.392 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:51:44.392 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:51:44.392 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-11T06:51:44.416 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-11T06:51:44.416 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-11T06:51:44.418 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-09-11T06:51:44.418 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster2:0 2024-09-11T06:51:44.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:51:44.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:51:44.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-11T06:51:44.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster2:0 2024-09-11T06:51:44.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:51:44.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:51:44.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster2 2024-09-11T06:51:44.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:51:44.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-11T06:51:44.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster2 2024-09-11T06:51:44.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster2 2024-09-11T06:51:44.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:51:44.419 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:51:44.420 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-11T06:51:44.442 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-11T06:51:44.442 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-11T06:51:44.446 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_pid_file cluster2:0 2024-09-11T06:51:44.446 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2:0 2024-09-11T06:51:44.446 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:51:44.446 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-09-11T06:51:44.447 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster2:0 2024-09-11T06:51:44.447 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=cluster 2024-09-11T06:51:44.447 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:51:44.447 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster2 2024-09-11T06:51:44.447 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=0 2024-09-11T06:51:44.447 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-11T06:51:44.448 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval cluster=cluster2 2024-09-11T06:51:44.448 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster2 2024-09-11T06:51:44.448 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=0 2024-09-11T06:51:44.448 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=0 2024-09-11T06:51:44.448 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'pid file' 2024-09-11T06:51:44.467 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.pid 2024-09-11T06:51:44.467 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.pid 2024-09-11T06:51:44.469 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T06:51:44.469 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ stop_mirror cluster2:1 '' 2024-09-11T06:51:44.469 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local cluster=cluster2:1 2024-09-11T06:51:44.469 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local sig= 2024-09-11T06:51:44.469 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test -n '' 2024-09-11T06:51:44.469 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ local pid 2024-09-11T06:51:44.470 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ daemon_pid_file cluster2:1 2024-09-11T06:51:44.470 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster=cluster2:1 2024-09-11T06:51:44.470 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance 2024-09-11T06:51:44.471 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ set_cluster_instance cluster2:1 cluster instance 2024-09-11T06:51:44.471 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local val=cluster2:1 2024-09-11T06:51:44.471 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local cluster_var_name=cluster 2024-09-11T06:51:44.471 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local instance_var_name=instance 2024-09-11T06:51:44.471 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster2 2024-09-11T06:51:44.471 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=1 2024-09-11T06:51:44.471 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ '[' 1 = cluster2:1 ']' 2024-09-11T06:51:44.471 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval cluster=cluster2 2024-09-11T06:51:44.471 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ cluster=cluster2 2024-09-11T06:51:44.471 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ eval instance=1 2024-09-11T06:51:44.471 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ instance=1 2024-09-11T06:51:44.471 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.1 'pid file' 2024-09-11T06:51:44.494 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.pid 2024-09-11T06:51:44.495 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cat /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.pid 2024-09-11T06:51:44.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ pid=29862 2024-09-11T06:51:44.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -n 29862 ']' 2024-09-11T06:51:44.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ kill 29862 2024-09-11T06:51:44.496 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ for s in 1 2 4 8 16 32 2024-09-11T06:51:44.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ sleep 1 2024-09-11T06:51:45.497 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ps auxww 2024-09-11T06:51:45.498 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ awk -v pid=29862 '$2 == pid {print; exit 1}' 2024-09-11T06:51:45.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ break 2024-09-11T06:51:45.512 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ ps auxww 2024-09-11T06:51:45.513 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ awk -v pid=29862 '$2 == pid {print; exit 1}' 2024-09-11T06:51:45.527 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster2:1 cluster1 2024-09-11T06:51:45.527 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster2:1 2024-09-11T06:51:45.527 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster1 2024-09-11T06:51:45.528 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:51:45.528 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster2:1 local_cluster instance 2024-09-11T06:51:45.528 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster2:1 2024-09-11T06:51:45.528 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:51:45.528 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:51:45.528 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster2 2024-09-11T06:51:45.528 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=1 2024-09-11T06:51:45.528 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 1 = cluster2:1 ']' 2024-09-11T06:51:45.528 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster2 2024-09-11T06:51:45.528 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster2 2024-09-11T06:51:45.528 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=1 2024-09-11T06:51:45.528 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=1 2024-09-11T06:51:45.528 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2024-09-11T06:51:45.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-11T06:51:45.552 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-11T06:51:45.554 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_asok_file cluster2:1 cluster2 2024-09-11T06:51:45.555 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local local_cluster=cluster2:1 2024-09-11T06:51:45.555 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2 2024-09-11T06:51:45.555 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:51:45.555 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster2:1 local_cluster instance 2024-09-11T06:51:45.555 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster2:1 2024-09-11T06:51:45.555 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=local_cluster 2024-09-11T06:51:45.555 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:51:45.555 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster2 2024-09-11T06:51:45.555 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=1 2024-09-11T06:51:45.555 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 1 = cluster2:1 ']' 2024-09-11T06:51:45.555 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval local_cluster=cluster2 2024-09-11T06:51:45.555 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ local_cluster=cluster2 2024-09-11T06:51:45.556 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=1 2024-09-11T06:51:45.556 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=1 2024-09-11T06:51:45.556 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'admin socket' 2024-09-11T06:51:45.576 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-11T06:51:45.576 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2024-09-11T06:51:45.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ daemon_pid_file cluster2:1 2024-09-11T06:51:45.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster=cluster2:1 2024-09-11T06:51:45.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance 2024-09-11T06:51:45.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ set_cluster_instance cluster2:1 cluster instance 2024-09-11T06:51:45.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local val=cluster2:1 2024-09-11T06:51:45.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local cluster_var_name=cluster 2024-09-11T06:51:45.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ local instance_var_name=instance 2024-09-11T06:51:45.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ cluster=cluster2 2024-09-11T06:51:45.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ instance=1 2024-09-11T06:51:45.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ '[' 1 = cluster2:1 ']' 2024-09-11T06:51:45.579 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval cluster=cluster2 2024-09-11T06:51:45.580 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ cluster=cluster2 2024-09-11T06:51:45.580 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ eval instance=1 2024-09-11T06:51:45.580 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ instance=1 2024-09-11T06:51:45.580 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'pid file' 2024-09-11T06:51:45.603 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.pid 2024-09-11T06:51:45.603 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.pid 2024-09-11T06:51:45.604 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' -z 1 ']' 2024-09-11T06:51:45.604 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ test '' = /tmp/tmp.Dxwd1ESqKa 2024-09-11T06:51:45.605 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ rm -Rf /tmp/tmp.Dxwd1ESqKa 2024-09-11T06:51:45.619 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stdout:OK 2024-09-11T06:51:45.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ '[' 0 -eq 0 ']' 2024-09-11T06:51:45.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ echo OK 2024-09-11T06:51:45.620 INFO:tasks.workunit.cluster1.client.mirror.smithi112.stderr:+ exit 0 2024-09-11T06:51:45.621 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-11T06:51:45.621 DEBUG:teuthology.orchestra.run.smithi112:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp 2024-09-11T06:51:45.793 INFO:tasks.workunit:Stopping ['rbd/rbd_mirror_stress.sh'] on cluster1.client.mirror... 2024-09-11T06:51:45.793 DEBUG:teuthology.orchestra.run.smithi112:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.cluster1.client.mirror /home/ubuntu/cephtest/clone.cluster1.client.mirror 2024-09-11T06:51:46.091 DEBUG:teuthology.parallel:result is None 2024-09-11T06:51:46.092 DEBUG:teuthology.orchestra.run.smithi112:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2024-09-11T06:51:46.102 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2024-09-11T06:51:46.102 DEBUG:teuthology.orchestra.run.smithi112:> rmdir -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-09-11T06:51:46.150 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2024-09-11T06:51:46.150 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-11T06:51:46.164 INFO:tasks.ceph.ceph_manager.cluster2:waiting for clean 2024-09-11T06:51:46.164 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-11T06:51:46.490 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-11T06:51:46.490 INFO:teuthology.orchestra.run.smithi112.stderr:dumped all 2024-09-11T06:51:46.507 INFO:teuthology.orchestra.run.smithi112.stdout:{"pg_ready":true,"pg_map":{"version":1484,"stamp":"2024-09-11T06:51:45.724895+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":65,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3650628,"kb_used_data":13572,"kb_used_omap":292,"kb_used_meta":3636699,"kb_avail":279464892,"statfs":{"total":289910292480,"available":286172049408,"internally_reserved":0,"allocated":13897728,"data_stored":8051173,"data_compressed":7472,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":299013,"internal_metadata":3723980795},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-526723126,"num_objects":-148,"num_object_clones":-1,"num_object_copies":-296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-148,"num_whiteouts":-1,"num_read":-806573,"num_read_kb":-44327369,"num_write":-1084127,"num_write_kb":-19736325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-5,"num_bytes_recovered":-10615808,"num_keys_recovered":0,"num_objects_omap":-15,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001943"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":780,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-11T06:51:41.044034+0000","last_change":"2024-09-11T06:15:16.158701+0000","last_active":"2024-09-11T06:51:41.044034+0000","last_peered":"2024-09-11T06:51:41.044034+0000","last_clean":"2024-09-11T06:51:41.044034+0000","last_became_active":"2024-09-11T06:15:14.143307+0000","last_became_peered":"2024-09-11T06:15:14.143307+0000","last_unstale":"2024-09-11T06:51:41.044034+0000","last_undegraded":"2024-09-11T06:51:41.044034+0000","last_fullsized":"2024-09-11T06:51:41.044034+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:15:13.030949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:15:13.030949+0000","last_clean_scrub_stamp":"2024-09-11T06:15:13.030949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:41:30.583877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019449249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":780,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-11T06:51:41.046354+0000","last_change":"2024-09-11T06:15:16.157659+0000","last_active":"2024-09-11T06:51:41.046354+0000","last_peered":"2024-09-11T06:51:41.046354+0000","last_clean":"2024-09-11T06:51:41.046354+0000","last_became_active":"2024-09-11T06:15:14.142832+0000","last_became_peered":"2024-09-11T06:15:14.142832+0000","last_unstale":"2024-09-11T06:51:41.046354+0000","last_undegraded":"2024-09-11T06:51:41.046354+0000","last_fullsized":"2024-09-11T06:51:41.046354+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:15:13.030949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:15:13.030949+0000","last_clean_scrub_stamp":"2024-09-11T06:15:13.030949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T17:10:46.757129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001083002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":781,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-11T06:51:41.044324+0000","last_change":"2024-09-11T06:15:16.158167+0000","last_active":"2024-09-11T06:51:41.044324+0000","last_peered":"2024-09-11T06:51:41.044324+0000","last_clean":"2024-09-11T06:51:41.044324+0000","last_became_active":"2024-09-11T06:15:14.143100+0000","last_became_peered":"2024-09-11T06:15:14.143100+0000","last_unstale":"2024-09-11T06:51:41.044324+0000","last_undegraded":"2024-09-11T06:51:41.044324+0000","last_fullsized":"2024-09-11T06:51:41.044324+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:15:13.030949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:15:13.030949+0000","last_clean_scrub_stamp":"2024-09-11T06:15:13.030949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:44.555439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001527928,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":781,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-11T06:51:41.041333+0000","last_change":"2024-09-11T06:15:16.158239+0000","last_active":"2024-09-11T06:51:41.041333+0000","last_peered":"2024-09-11T06:51:41.041333+0000","last_clean":"2024-09-11T06:51:41.041333+0000","last_became_active":"2024-09-11T06:15:14.142926+0000","last_became_peered":"2024-09-11T06:15:14.142926+0000","last_unstale":"2024-09-11T06:51:41.041333+0000","last_undegraded":"2024-09-11T06:51:41.041333+0000","last_fullsized":"2024-09-11T06:51:41.041333+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:15:13.030949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:15:13.030949+0000","last_clean_scrub_stamp":"2024-09-11T06:15:13.030949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:52:31.503430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015085280000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":782,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-11T06:51:41.086704+0000","last_change":"2024-09-11T06:15:16.164947+0000","last_active":"2024-09-11T06:51:41.086704+0000","last_peered":"2024-09-11T06:51:41.086704+0000","last_clean":"2024-09-11T06:51:41.086704+0000","last_became_active":"2024-09-11T06:15:14.142347+0000","last_became_peered":"2024-09-11T06:15:14.142347+0000","last_unstale":"2024-09-11T06:51:41.086704+0000","last_undegraded":"2024-09-11T06:51:41.086704+0000","last_fullsized":"2024-09-11T06:51:41.086704+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:15:13.030949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:15:13.030949+0000","last_clean_scrub_stamp":"2024-09-11T06:15:13.030949+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:26:00.950222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022347420000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":780,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-11T06:51:40.889467+0000","last_change":"2024-09-11T06:15:16.457217+0000","last_active":"2024-09-11T06:51:40.889467+0000","last_peered":"2024-09-11T06:51:40.889467+0000","last_clean":"2024-09-11T06:51:40.889467+0000","last_became_active":"2024-09-11T06:15:14.143512+0000","last_became_peered":"2024-09-11T06:15:14.143512+0000","last_unstale":"2024-09-11T06:51:40.889467+0000","last_undegraded":"2024-09-11T06:51:40.889467+0000","last_fullsized":"2024-09-11T06:51:40.889467+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:15:13.030949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:15:13.030949+0000","last_clean_scrub_stamp":"2024-09-11T06:15:13.030949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:31:31.971513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063951500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":781,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-11T06:51:40.890363+0000","last_change":"2024-09-11T06:15:16.457653+0000","last_active":"2024-09-11T06:51:40.890363+0000","last_peered":"2024-09-11T06:51:40.890363+0000","last_clean":"2024-09-11T06:51:40.890363+0000","last_became_active":"2024-09-11T06:15:14.142323+0000","last_became_peered":"2024-09-11T06:15:14.142323+0000","last_unstale":"2024-09-11T06:51:40.890363+0000","last_undegraded":"2024-09-11T06:51:40.890363+0000","last_fullsized":"2024-09-11T06:51:40.890363+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:15:13.030949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:15:13.030949+0000","last_clean_scrub_stamp":"2024-09-11T06:15:13.030949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:28:11.865252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091634599999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":782,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-11T06:51:41.045124+0000","last_change":"2024-09-11T06:15:16.157504+0000","last_active":"2024-09-11T06:51:41.045124+0000","last_peered":"2024-09-11T06:51:41.045124+0000","last_clean":"2024-09-11T06:51:41.045124+0000","last_became_active":"2024-09-11T06:15:14.142073+0000","last_became_peered":"2024-09-11T06:15:14.142073+0000","last_unstale":"2024-09-11T06:51:41.045124+0000","last_undegraded":"2024-09-11T06:51:41.045124+0000","last_fullsized":"2024-09-11T06:51:41.045124+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:15:13.030949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:15:13.030949+0000","last_clean_scrub_stamp":"2024-09-11T06:15:13.030949+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:16:44.214033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010516950000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":908,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-11T06:51:41.045561+0000","last_change":"2024-09-11T06:15:12.513296+0000","last_active":"2024-09-11T06:51:41.045561+0000","last_peered":"2024-09-11T06:51:41.045561+0000","last_clean":"2024-09-11T06:51:41.045561+0000","last_became_active":"2024-09-11T06:15:12.511458+0000","last_became_peered":"2024-09-11T06:15:12.511458+0000","last_unstale":"2024-09-11T06:51:41.045561+0000","last_undegraded":"2024-09-11T06:51:41.045561+0000","last_fullsized":"2024-09-11T06:51:41.045561+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:15:11.022532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:15:11.022532+0000","last_clean_scrub_stamp":"2024-09-11T06:15:11.022532+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:14:38.108521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7472,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738814,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":805464,"kb_used_data":4484,"kb_used_omap":94,"kb_used_meta":800865,"kb_avail":93566376,"statfs":{"total":96636764160,"available":95811969024,"internally_reserved":0,"allocated":4591616,"data_stored":2290133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":96428,"internal_metadata":820086612},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738815,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1289880,"kb_used_data":4544,"kb_used_omap":92,"kb_used_meta":1285219,"kb_avail":93081960,"statfs":{"total":96636764160,"available":95315927040,"internally_reserved":0,"allocated":4653056,"data_stored":2880520,"data_compressed":3736,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":94588,"internal_metadata":1316064900},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738814,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1555284,"kb_used_data":4544,"kb_used_omap":105,"kb_used_meta":1550614,"kb_avail":92816556,"statfs":{"total":96636764160,"available":95044153344,"internally_reserved":0,"allocated":4653056,"data_stored":2880520,"data_compressed":3736,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":107997,"internal_metadata":1587829283},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3736,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3736,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T06:51:46.508 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-11T06:51:46.799 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-11T06:51:46.799 INFO:teuthology.orchestra.run.smithi112.stderr:dumped all 2024-09-11T06:51:46.817 INFO:teuthology.orchestra.run.smithi112.stdout:{"pg_ready":true,"pg_map":{"version":1484,"stamp":"2024-09-11T06:51:45.724895+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":65,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3650628,"kb_used_data":13572,"kb_used_omap":292,"kb_used_meta":3636699,"kb_avail":279464892,"statfs":{"total":289910292480,"available":286172049408,"internally_reserved":0,"allocated":13897728,"data_stored":8051173,"data_compressed":7472,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":299013,"internal_metadata":3723980795},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-526723126,"num_objects":-148,"num_object_clones":-1,"num_object_copies":-296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-148,"num_whiteouts":-1,"num_read":-806573,"num_read_kb":-44327369,"num_write":-1084127,"num_write_kb":-19736325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-5,"num_bytes_recovered":-10615808,"num_keys_recovered":0,"num_objects_omap":-15,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001943"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":780,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-11T06:51:41.044034+0000","last_change":"2024-09-11T06:15:16.158701+0000","last_active":"2024-09-11T06:51:41.044034+0000","last_peered":"2024-09-11T06:51:41.044034+0000","last_clean":"2024-09-11T06:51:41.044034+0000","last_became_active":"2024-09-11T06:15:14.143307+0000","last_became_peered":"2024-09-11T06:15:14.143307+0000","last_unstale":"2024-09-11T06:51:41.044034+0000","last_undegraded":"2024-09-11T06:51:41.044034+0000","last_fullsized":"2024-09-11T06:51:41.044034+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:15:13.030949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:15:13.030949+0000","last_clean_scrub_stamp":"2024-09-11T06:15:13.030949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:41:30.583877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019449249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":780,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-11T06:51:41.046354+0000","last_change":"2024-09-11T06:15:16.157659+0000","last_active":"2024-09-11T06:51:41.046354+0000","last_peered":"2024-09-11T06:51:41.046354+0000","last_clean":"2024-09-11T06:51:41.046354+0000","last_became_active":"2024-09-11T06:15:14.142832+0000","last_became_peered":"2024-09-11T06:15:14.142832+0000","last_unstale":"2024-09-11T06:51:41.046354+0000","last_undegraded":"2024-09-11T06:51:41.046354+0000","last_fullsized":"2024-09-11T06:51:41.046354+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:15:13.030949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:15:13.030949+0000","last_clean_scrub_stamp":"2024-09-11T06:15:13.030949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T17:10:46.757129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001083002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":781,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-11T06:51:41.044324+0000","last_change":"2024-09-11T06:15:16.158167+0000","last_active":"2024-09-11T06:51:41.044324+0000","last_peered":"2024-09-11T06:51:41.044324+0000","last_clean":"2024-09-11T06:51:41.044324+0000","last_became_active":"2024-09-11T06:15:14.143100+0000","last_became_peered":"2024-09-11T06:15:14.143100+0000","last_unstale":"2024-09-11T06:51:41.044324+0000","last_undegraded":"2024-09-11T06:51:41.044324+0000","last_fullsized":"2024-09-11T06:51:41.044324+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:15:13.030949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:15:13.030949+0000","last_clean_scrub_stamp":"2024-09-11T06:15:13.030949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:44.555439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001527928,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":781,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-11T06:51:41.041333+0000","last_change":"2024-09-11T06:15:16.158239+0000","last_active":"2024-09-11T06:51:41.041333+0000","last_peered":"2024-09-11T06:51:41.041333+0000","last_clean":"2024-09-11T06:51:41.041333+0000","last_became_active":"2024-09-11T06:15:14.142926+0000","last_became_peered":"2024-09-11T06:15:14.142926+0000","last_unstale":"2024-09-11T06:51:41.041333+0000","last_undegraded":"2024-09-11T06:51:41.041333+0000","last_fullsized":"2024-09-11T06:51:41.041333+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:15:13.030949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:15:13.030949+0000","last_clean_scrub_stamp":"2024-09-11T06:15:13.030949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:52:31.503430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015085280000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":782,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-11T06:51:41.086704+0000","last_change":"2024-09-11T06:15:16.164947+0000","last_active":"2024-09-11T06:51:41.086704+0000","last_peered":"2024-09-11T06:51:41.086704+0000","last_clean":"2024-09-11T06:51:41.086704+0000","last_became_active":"2024-09-11T06:15:14.142347+0000","last_became_peered":"2024-09-11T06:15:14.142347+0000","last_unstale":"2024-09-11T06:51:41.086704+0000","last_undegraded":"2024-09-11T06:51:41.086704+0000","last_fullsized":"2024-09-11T06:51:41.086704+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:15:13.030949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:15:13.030949+0000","last_clean_scrub_stamp":"2024-09-11T06:15:13.030949+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:26:00.950222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022347420000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":780,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-11T06:51:40.889467+0000","last_change":"2024-09-11T06:15:16.457217+0000","last_active":"2024-09-11T06:51:40.889467+0000","last_peered":"2024-09-11T06:51:40.889467+0000","last_clean":"2024-09-11T06:51:40.889467+0000","last_became_active":"2024-09-11T06:15:14.143512+0000","last_became_peered":"2024-09-11T06:15:14.143512+0000","last_unstale":"2024-09-11T06:51:40.889467+0000","last_undegraded":"2024-09-11T06:51:40.889467+0000","last_fullsized":"2024-09-11T06:51:40.889467+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:15:13.030949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:15:13.030949+0000","last_clean_scrub_stamp":"2024-09-11T06:15:13.030949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:31:31.971513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063951500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":781,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-11T06:51:40.890363+0000","last_change":"2024-09-11T06:15:16.457653+0000","last_active":"2024-09-11T06:51:40.890363+0000","last_peered":"2024-09-11T06:51:40.890363+0000","last_clean":"2024-09-11T06:51:40.890363+0000","last_became_active":"2024-09-11T06:15:14.142323+0000","last_became_peered":"2024-09-11T06:15:14.142323+0000","last_unstale":"2024-09-11T06:51:40.890363+0000","last_undegraded":"2024-09-11T06:51:40.890363+0000","last_fullsized":"2024-09-11T06:51:40.890363+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:15:13.030949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:15:13.030949+0000","last_clean_scrub_stamp":"2024-09-11T06:15:13.030949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:28:11.865252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091634599999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":782,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-11T06:51:41.045124+0000","last_change":"2024-09-11T06:15:16.157504+0000","last_active":"2024-09-11T06:51:41.045124+0000","last_peered":"2024-09-11T06:51:41.045124+0000","last_clean":"2024-09-11T06:51:41.045124+0000","last_became_active":"2024-09-11T06:15:14.142073+0000","last_became_peered":"2024-09-11T06:15:14.142073+0000","last_unstale":"2024-09-11T06:51:41.045124+0000","last_undegraded":"2024-09-11T06:51:41.045124+0000","last_fullsized":"2024-09-11T06:51:41.045124+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:15:13.030949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:15:13.030949+0000","last_clean_scrub_stamp":"2024-09-11T06:15:13.030949+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:16:44.214033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010516950000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":908,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-11T06:51:41.045561+0000","last_change":"2024-09-11T06:15:12.513296+0000","last_active":"2024-09-11T06:51:41.045561+0000","last_peered":"2024-09-11T06:51:41.045561+0000","last_clean":"2024-09-11T06:51:41.045561+0000","last_became_active":"2024-09-11T06:15:12.511458+0000","last_became_peered":"2024-09-11T06:15:12.511458+0000","last_unstale":"2024-09-11T06:51:41.045561+0000","last_undegraded":"2024-09-11T06:51:41.045561+0000","last_fullsized":"2024-09-11T06:51:41.045561+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:15:11.022532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:15:11.022532+0000","last_clean_scrub_stamp":"2024-09-11T06:15:11.022532+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:14:38.108521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7472,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738814,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":805464,"kb_used_data":4484,"kb_used_omap":94,"kb_used_meta":800865,"kb_avail":93566376,"statfs":{"total":96636764160,"available":95811969024,"internally_reserved":0,"allocated":4591616,"data_stored":2290133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":96428,"internal_metadata":820086612},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738815,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1289880,"kb_used_data":4544,"kb_used_omap":92,"kb_used_meta":1285219,"kb_avail":93081960,"statfs":{"total":96636764160,"available":95315927040,"internally_reserved":0,"allocated":4653056,"data_stored":2880520,"data_compressed":3736,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":94588,"internal_metadata":1316064900},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738814,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1555284,"kb_used_data":4544,"kb_used_omap":105,"kb_used_meta":1550614,"kb_avail":92816556,"statfs":{"total":96636764160,"available":95044153344,"internally_reserved":0,"allocated":4653056,"data_stored":2880520,"data_compressed":3736,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":107997,"internal_metadata":1587829283},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3736,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3736,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T06:51:46.818 INFO:tasks.ceph.ceph_manager.cluster2:clean! 2024-09-11T06:51:46.818 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-11T06:51:47.089 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-11T06:51:47.089 INFO:teuthology.orchestra.run.smithi112.stderr:dumped all 2024-09-11T06:51:47.105 INFO:teuthology.orchestra.run.smithi112.stdout:{"pg_ready":true,"pg_map":{"version":1484,"stamp":"2024-09-11T06:51:45.724895+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":65,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3650628,"kb_used_data":13572,"kb_used_omap":292,"kb_used_meta":3636699,"kb_avail":279464892,"statfs":{"total":289910292480,"available":286172049408,"internally_reserved":0,"allocated":13897728,"data_stored":8051173,"data_compressed":7472,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":299013,"internal_metadata":3723980795},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-526723126,"num_objects":-148,"num_object_clones":-1,"num_object_copies":-296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-148,"num_whiteouts":-1,"num_read":-806573,"num_read_kb":-44327369,"num_write":-1084127,"num_write_kb":-19736325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-5,"num_bytes_recovered":-10615808,"num_keys_recovered":0,"num_objects_omap":-15,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001943"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":780,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-11T06:51:41.044034+0000","last_change":"2024-09-11T06:15:16.158701+0000","last_active":"2024-09-11T06:51:41.044034+0000","last_peered":"2024-09-11T06:51:41.044034+0000","last_clean":"2024-09-11T06:51:41.044034+0000","last_became_active":"2024-09-11T06:15:14.143307+0000","last_became_peered":"2024-09-11T06:15:14.143307+0000","last_unstale":"2024-09-11T06:51:41.044034+0000","last_undegraded":"2024-09-11T06:51:41.044034+0000","last_fullsized":"2024-09-11T06:51:41.044034+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:15:13.030949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:15:13.030949+0000","last_clean_scrub_stamp":"2024-09-11T06:15:13.030949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:41:30.583877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019449249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":780,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-11T06:51:41.046354+0000","last_change":"2024-09-11T06:15:16.157659+0000","last_active":"2024-09-11T06:51:41.046354+0000","last_peered":"2024-09-11T06:51:41.046354+0000","last_clean":"2024-09-11T06:51:41.046354+0000","last_became_active":"2024-09-11T06:15:14.142832+0000","last_became_peered":"2024-09-11T06:15:14.142832+0000","last_unstale":"2024-09-11T06:51:41.046354+0000","last_undegraded":"2024-09-11T06:51:41.046354+0000","last_fullsized":"2024-09-11T06:51:41.046354+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:15:13.030949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:15:13.030949+0000","last_clean_scrub_stamp":"2024-09-11T06:15:13.030949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T17:10:46.757129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001083002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":781,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-11T06:51:41.044324+0000","last_change":"2024-09-11T06:15:16.158167+0000","last_active":"2024-09-11T06:51:41.044324+0000","last_peered":"2024-09-11T06:51:41.044324+0000","last_clean":"2024-09-11T06:51:41.044324+0000","last_became_active":"2024-09-11T06:15:14.143100+0000","last_became_peered":"2024-09-11T06:15:14.143100+0000","last_unstale":"2024-09-11T06:51:41.044324+0000","last_undegraded":"2024-09-11T06:51:41.044324+0000","last_fullsized":"2024-09-11T06:51:41.044324+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:15:13.030949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:15:13.030949+0000","last_clean_scrub_stamp":"2024-09-11T06:15:13.030949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:44.555439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001527928,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":781,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-11T06:51:41.041333+0000","last_change":"2024-09-11T06:15:16.158239+0000","last_active":"2024-09-11T06:51:41.041333+0000","last_peered":"2024-09-11T06:51:41.041333+0000","last_clean":"2024-09-11T06:51:41.041333+0000","last_became_active":"2024-09-11T06:15:14.142926+0000","last_became_peered":"2024-09-11T06:15:14.142926+0000","last_unstale":"2024-09-11T06:51:41.041333+0000","last_undegraded":"2024-09-11T06:51:41.041333+0000","last_fullsized":"2024-09-11T06:51:41.041333+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:15:13.030949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:15:13.030949+0000","last_clean_scrub_stamp":"2024-09-11T06:15:13.030949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:52:31.503430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015085280000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":782,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-11T06:51:41.086704+0000","last_change":"2024-09-11T06:15:16.164947+0000","last_active":"2024-09-11T06:51:41.086704+0000","last_peered":"2024-09-11T06:51:41.086704+0000","last_clean":"2024-09-11T06:51:41.086704+0000","last_became_active":"2024-09-11T06:15:14.142347+0000","last_became_peered":"2024-09-11T06:15:14.142347+0000","last_unstale":"2024-09-11T06:51:41.086704+0000","last_undegraded":"2024-09-11T06:51:41.086704+0000","last_fullsized":"2024-09-11T06:51:41.086704+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:15:13.030949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:15:13.030949+0000","last_clean_scrub_stamp":"2024-09-11T06:15:13.030949+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:26:00.950222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022347420000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":780,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-11T06:51:40.889467+0000","last_change":"2024-09-11T06:15:16.457217+0000","last_active":"2024-09-11T06:51:40.889467+0000","last_peered":"2024-09-11T06:51:40.889467+0000","last_clean":"2024-09-11T06:51:40.889467+0000","last_became_active":"2024-09-11T06:15:14.143512+0000","last_became_peered":"2024-09-11T06:15:14.143512+0000","last_unstale":"2024-09-11T06:51:40.889467+0000","last_undegraded":"2024-09-11T06:51:40.889467+0000","last_fullsized":"2024-09-11T06:51:40.889467+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:15:13.030949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:15:13.030949+0000","last_clean_scrub_stamp":"2024-09-11T06:15:13.030949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:31:31.971513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063951500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":781,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-11T06:51:40.890363+0000","last_change":"2024-09-11T06:15:16.457653+0000","last_active":"2024-09-11T06:51:40.890363+0000","last_peered":"2024-09-11T06:51:40.890363+0000","last_clean":"2024-09-11T06:51:40.890363+0000","last_became_active":"2024-09-11T06:15:14.142323+0000","last_became_peered":"2024-09-11T06:15:14.142323+0000","last_unstale":"2024-09-11T06:51:40.890363+0000","last_undegraded":"2024-09-11T06:51:40.890363+0000","last_fullsized":"2024-09-11T06:51:40.890363+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:15:13.030949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:15:13.030949+0000","last_clean_scrub_stamp":"2024-09-11T06:15:13.030949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:28:11.865252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091634599999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":782,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-11T06:51:41.045124+0000","last_change":"2024-09-11T06:15:16.157504+0000","last_active":"2024-09-11T06:51:41.045124+0000","last_peered":"2024-09-11T06:51:41.045124+0000","last_clean":"2024-09-11T06:51:41.045124+0000","last_became_active":"2024-09-11T06:15:14.142073+0000","last_became_peered":"2024-09-11T06:15:14.142073+0000","last_unstale":"2024-09-11T06:51:41.045124+0000","last_undegraded":"2024-09-11T06:51:41.045124+0000","last_fullsized":"2024-09-11T06:51:41.045124+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:15:13.030949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:15:13.030949+0000","last_clean_scrub_stamp":"2024-09-11T06:15:13.030949+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:16:44.214033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010516950000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":908,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-11T06:51:41.045561+0000","last_change":"2024-09-11T06:15:12.513296+0000","last_active":"2024-09-11T06:51:41.045561+0000","last_peered":"2024-09-11T06:51:41.045561+0000","last_clean":"2024-09-11T06:51:41.045561+0000","last_became_active":"2024-09-11T06:15:12.511458+0000","last_became_peered":"2024-09-11T06:15:12.511458+0000","last_unstale":"2024-09-11T06:51:41.045561+0000","last_undegraded":"2024-09-11T06:51:41.045561+0000","last_fullsized":"2024-09-11T06:51:41.045561+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:15:11.022532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:15:11.022532+0000","last_clean_scrub_stamp":"2024-09-11T06:15:11.022532+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:14:38.108521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7472,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738814,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":805464,"kb_used_data":4484,"kb_used_omap":94,"kb_used_meta":800865,"kb_avail":93566376,"statfs":{"total":96636764160,"available":95811969024,"internally_reserved":0,"allocated":4591616,"data_stored":2290133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":96428,"internal_metadata":820086612},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738815,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1289880,"kb_used_data":4544,"kb_used_omap":92,"kb_used_meta":1285219,"kb_avail":93081960,"statfs":{"total":96636764160,"available":95315927040,"internally_reserved":0,"allocated":4653056,"data_stored":2880520,"data_compressed":3736,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":94588,"internal_metadata":1316064900},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738814,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1555284,"kb_used_data":4544,"kb_used_omap":105,"kb_used_meta":1550614,"kb_avail":92816556,"statfs":{"total":96636764160,"available":95044153344,"internally_reserved":0,"allocated":4653056,"data_stored":2880520,"data_compressed":3736,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":107997,"internal_metadata":1587829283},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3736,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3736,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T06:51:47.106 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-09-11T06:51:47.385 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-11T06:51:47.385 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":388,"fsid":"f5a0f59c-ac24-45be-8681-35322ac680b8","created":"2024-09-11T06:14:57.348013+0000","modified":"2024-09-11T06:51:40.880276+0000","last_up_change":"2024-09-11T06:15:09.003637+0000","last_in_change":"2024-09-11T06:14:58.282695+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":3,"require_min_compat_client":"octopus","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-11T06:15:11.008778+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-11T06:15:12.586728+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":15,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f199f8ec-e9e7-4467-aed6-9a4445ee94b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6816","nonce":2881424358},{"type":"v1","addr":"172.21.15.112:6817","nonce":2881424358}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6818","nonce":2881424358},{"type":"v1","addr":"172.21.15.112:6819","nonce":2881424358}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6822","nonce":2881424358},{"type":"v1","addr":"172.21.15.112:6823","nonce":2881424358}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6820","nonce":2881424358},{"type":"v1","addr":"172.21.15.112:6821","nonce":2881424358}]},"public_addr":"172.21.15.112:6817/2881424358","cluster_addr":"172.21.15.112:6819/2881424358","heartbeat_back_addr":"172.21.15.112:6823/2881424358","heartbeat_front_addr":"172.21.15.112:6821/2881424358","state":["exists","up"]},{"osd":1,"uuid":"51544368-1990-4034-9462-a6b32e28e9f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":1273240578},{"type":"v1","addr":"172.21.15.112:6801","nonce":1273240578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":1273240578},{"type":"v1","addr":"172.21.15.112:6803","nonce":1273240578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":1273240578},{"type":"v1","addr":"172.21.15.112:6807","nonce":1273240578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":1273240578},{"type":"v1","addr":"172.21.15.112:6805","nonce":1273240578}]},"public_addr":"172.21.15.112:6801/1273240578","cluster_addr":"172.21.15.112:6803/1273240578","heartbeat_back_addr":"172.21.15.112:6807/1273240578","heartbeat_front_addr":"172.21.15.112:6805/1273240578","state":["exists","up"]},{"osd":2,"uuid":"e1fa99c8-30e7-4f3b-a7af-16ddf8c55f4c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":3588377934},{"type":"v1","addr":"172.21.15.112:6809","nonce":3588377934}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6810","nonce":3588377934},{"type":"v1","addr":"172.21.15.112:6811","nonce":3588377934}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":3588377934},{"type":"v1","addr":"172.21.15.112:6815","nonce":3588377934}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6812","nonce":3588377934},{"type":"v1","addr":"172.21.15.112:6813","nonce":3588377934}]},"public_addr":"172.21.15.112:6809/3588377934","cluster_addr":"172.21.15.112:6811/3588377934","heartbeat_back_addr":"172.21.15.112:6815/3588377934","heartbeat_front_addr":"172.21.15.112:6813/3588377934","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T06:15:04.608603+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T06:15:04.329053+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T06:15:04.503513+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.112:0/2460174515":"2024-09-11T07:50:56.642323+0000","172.21.15.112:0/1772538979":"2024-09-11T07:49:19.627047+0000","172.21.15.112:0/4254349053":"2024-09-11T07:47:54.734535+0000","172.21.15.112:0/2259374114":"2024-09-11T07:40:44.478999+0000","172.21.15.112:0/3229932366":"2024-09-11T07:34:19.803007+0000","172.21.15.112:0/3120127562":"2024-09-11T07:38:50.672553+0000","172.21.15.112:0/1564574846":"2024-09-11T07:32:17.732601+0000","172.21.15.112:0/4203477775":"2024-09-11T07:17:26.486148+0000","172.21.15.112:0/2808250456":"2024-09-11T07:33:39.134677+0000","172.21.15.112:0/3327696888":"2024-09-11T07:35:00.370050+0000","172.21.15.112:0/3848328324":"2024-09-11T07:33:59.155986+0000","172.21.15.112:0/3713798601":"2024-09-11T07:33:26.610511+0000","172.21.15.112:0/2798867900":"2024-09-11T07:35:27.424883+0000","172.21.15.112:0/2668792850":"2024-09-11T07:33:04.761921+0000","172.21.15.112:0/1506207236":"2024-09-11T07:40:07.615470+0000","172.21.15.112:0/35807154":"2024-09-11T07:47:15.114005+0000","172.21.15.112:0/2401435262":"2024-09-11T07:32:47.535562+0000","172.21.15.112:0/4140410344":"2024-09-11T07:51:34.836251+0000","172.21.15.112:0/567893784":"2024-09-11T07:22:39.842694+0000","172.21.15.112:0/1869373832":"2024-09-11T07:28:21.672566+0000","172.21.15.112:0/3311872493":"2024-09-11T07:34:40.466662+0000","172.21.15.112:0/4181417642":"2024-09-11T07:33:46.698377+0000","172.21.15.112:0/787808154":"2024-09-11T07:36:42.710330+0000","172.21.15.112:0/2642475319":"2024-09-11T07:37:24.687004+0000","172.21.15.112:0/282482210":"2024-09-11T07:32:26.184067+0000","172.21.15.112:0/1303555539":"2024-09-11T07:21:16.158093+0000","172.21.15.112:0/2373097925":"2024-09-11T07:33:52.742283+0000","172.21.15.112:0/3906967570":"2024-09-11T07:23:36.090739+0000","172.21.15.112:0/1681720227":"2024-09-11T07:19:01.680994+0000","172.21.15.112:0/1323671982":"2024-09-11T07:46:31.876094+0000","172.21.15.112:0/3750693613":"2024-09-11T07:27:33.444801+0000","172.21.15.112:0/3530695163":"2024-09-11T07:29:14.475291+0000","172.21.15.112:0/2275453036":"2024-09-11T07:34:34.024985+0000","172.21.15.112:0/4174199721":"2024-09-11T07:37:51.147025+0000","172.21.15.112:0/320922338":"2024-09-11T07:25:39.259389+0000","172.21.15.112:0/3985264580":"2024-09-11T07:31:33.464719+0000","172.21.15.112:0/336102942":"2024-09-11T07:33:22.704942+0000","172.21.15.112:0/984902608":"2024-09-11T07:20:03.309695+0000","172.21.15.112:0/2792502219":"2024-09-11T07:35:31.851596+0000","172.21.15.112:0/46432156":"2024-09-11T07:45:10.985313+0000","172.21.15.112:0/3776054687":"2024-09-11T07:35:44.304677+0000","172.21.15.112:0/3369801897":"2024-09-11T07:36:07.337598+0000","172.21.15.112:0/1376861196":"2024-09-11T07:38:32.649823+0000","172.21.15.112:0/840531773":"2024-09-11T07:36:08.142099+0000","172.21.15.112:0/2928236387":"2024-09-11T07:36:13.720574+0000","172.21.15.112:0/411736162":"2024-09-11T07:38:57.983593+0000","172.21.15.112:0/549836860":"2024-09-11T07:36:22.033316+0000","172.21.15.112:0/3806567391":"2024-09-11T07:36:49.034656+0000","172.21.15.112:0/46990434":"2024-09-11T07:37:01.980786+0000","172.21.15.112:0/749106464":"2024-09-11T07:37:07.850954+0000","172.21.15.112:0/3480070670":"2024-09-11T07:37:11.383670+0000","172.21.15.112:0/2354825843":"2024-09-11T07:37:49.225700+0000","172.21.15.112:0/663295647":"2024-09-11T07:50:16.202041+0000","172.21.15.112:0/3019306514":"2024-09-11T07:37:51.993967+0000","172.21.15.112:0/1882223002":"2024-09-11T07:37:57.598767+0000","172.21.15.112:0/1567981817":"2024-09-11T07:35:30.773413+0000","172.21.15.112:0/2367569699":"2024-09-11T07:38:56.689272+0000","172.21.15.112:0/1362423819":"2024-09-11T07:38:08.538696+0000","172.21.15.112:0/2297025889":"2024-09-11T07:48:34.932919+0000","172.21.15.112:0/3186536004":"2024-09-11T07:38:46.680709+0000","172.21.15.112:0/4275195140":"2024-09-11T07:39:02.614496+0000","172.21.15.112:0/3792734753":"2024-09-11T07:45:54.957040+0000","172.21.15.112:0/877077650":"2024-09-11T07:34:05.729537+0000","172.21.15.112:0/1137997695":"2024-09-11T07:39:11.102231+0000","172.21.15.112:0/3470015270":"2024-09-11T07:34:51.224334+0000","172.21.15.112:0/2607760564":"2024-09-11T07:32:03.765254+0000","172.21.15.112:0/363389277":"2024-09-11T07:39:56.179178+0000","172.21.15.112:0/2881449681":"2024-09-11T07:39:50.105111+0000","172.21.15.112:0/2953232248":"2024-09-11T07:40:21.121931+0000","172.21.15.112:0/3646702151":"2024-09-11T07:40:31.500878+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T06:51:48.403 INFO:tasks.ceph:Scrubbing cluster2.osd.0 2024-09-11T06:51:48.403 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-11T06:51:48.564 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-11T06:51:48.565 INFO:teuthology.orchestra.run.smithi112.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-11T06:51:48.565 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-11T06:51:48.581 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd deep-scrub 0 2024-09-11T06:51:48.849 INFO:teuthology.orchestra.run.smithi112.stderr:instructed osd(s) 0 to deep-scrub 2024-09-11T06:51:48.866 INFO:tasks.ceph:Scrubbing cluster2.osd.1 2024-09-11T06:51:48.866 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-11T06:51:49.003 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-11T06:51:49.003 INFO:teuthology.orchestra.run.smithi112.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-11T06:51:49.003 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-11T06:51:49.017 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd deep-scrub 1 2024-09-11T06:51:49.307 INFO:teuthology.orchestra.run.smithi112.stderr:instructed osd(s) 1 to deep-scrub 2024-09-11T06:51:49.326 INFO:tasks.ceph:Scrubbing cluster2.osd.2 2024-09-11T06:51:49.327 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-11T06:51:49.464 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2024-09-11T06:51:49.464 INFO:teuthology.orchestra.run.smithi112.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-11T06:51:49.464 INFO:teuthology.orchestra.run.smithi112.stdout:} 2024-09-11T06:51:49.479 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd deep-scrub 2 2024-09-11T06:51:49.753 INFO:teuthology.orchestra.run.smithi112.stderr:instructed osd(s) 2 to deep-scrub 2024-09-11T06:51:49.770 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-11T06:51:50.043 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-11T06:51:50.044 INFO:teuthology.orchestra.run.smithi112.stderr:dumped all 2024-09-11T06:51:50.060 INFO:teuthology.orchestra.run.smithi112.stdout:{"pg_ready":true,"pg_map":{"version":1486,"stamp":"2024-09-11T06:51:49.725860+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":65,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3650628,"kb_used_data":13572,"kb_used_omap":292,"kb_used_meta":3636699,"kb_avail":279464892,"statfs":{"total":289910292480,"available":286172049408,"internally_reserved":0,"allocated":13897728,"data_stored":8051173,"data_compressed":7472,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":299013,"internal_metadata":3723980795},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19,"num_objects":-6,"num_object_clones":0,"num_object_copies":-12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6,"num_whiteouts":0,"num_read":-2247,"num_read_kb":-2236,"num_write":-150,"num_write_kb":-147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002411"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":780,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-11T06:51:41.044034+0000","last_change":"2024-09-11T06:15:16.158701+0000","last_active":"2024-09-11T06:51:41.044034+0000","last_peered":"2024-09-11T06:51:41.044034+0000","last_clean":"2024-09-11T06:51:41.044034+0000","last_became_active":"2024-09-11T06:15:14.143307+0000","last_became_peered":"2024-09-11T06:15:14.143307+0000","last_unstale":"2024-09-11T06:51:41.044034+0000","last_undegraded":"2024-09-11T06:51:41.044034+0000","last_fullsized":"2024-09-11T06:51:41.044034+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:15:13.030949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:15:13.030949+0000","last_clean_scrub_stamp":"2024-09-11T06:15:13.030949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:41:30.583877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019449249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":780,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-11T06:51:41.046354+0000","last_change":"2024-09-11T06:15:16.157659+0000","last_active":"2024-09-11T06:51:41.046354+0000","last_peered":"2024-09-11T06:51:41.046354+0000","last_clean":"2024-09-11T06:51:41.046354+0000","last_became_active":"2024-09-11T06:15:14.142832+0000","last_became_peered":"2024-09-11T06:15:14.142832+0000","last_unstale":"2024-09-11T06:51:41.046354+0000","last_undegraded":"2024-09-11T06:51:41.046354+0000","last_fullsized":"2024-09-11T06:51:41.046354+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:15:13.030949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:15:13.030949+0000","last_clean_scrub_stamp":"2024-09-11T06:15:13.030949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T17:10:46.757129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001083002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":781,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-11T06:51:41.044324+0000","last_change":"2024-09-11T06:15:16.158167+0000","last_active":"2024-09-11T06:51:41.044324+0000","last_peered":"2024-09-11T06:51:41.044324+0000","last_clean":"2024-09-11T06:51:41.044324+0000","last_became_active":"2024-09-11T06:15:14.143100+0000","last_became_peered":"2024-09-11T06:15:14.143100+0000","last_unstale":"2024-09-11T06:51:41.044324+0000","last_undegraded":"2024-09-11T06:51:41.044324+0000","last_fullsized":"2024-09-11T06:51:41.044324+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:15:13.030949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:15:13.030949+0000","last_clean_scrub_stamp":"2024-09-11T06:15:13.030949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:08:44.555439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001527928,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":781,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-11T06:51:41.041333+0000","last_change":"2024-09-11T06:15:16.158239+0000","last_active":"2024-09-11T06:51:41.041333+0000","last_peered":"2024-09-11T06:51:41.041333+0000","last_clean":"2024-09-11T06:51:41.041333+0000","last_became_active":"2024-09-11T06:15:14.142926+0000","last_became_peered":"2024-09-11T06:15:14.142926+0000","last_unstale":"2024-09-11T06:51:41.041333+0000","last_undegraded":"2024-09-11T06:51:41.041333+0000","last_fullsized":"2024-09-11T06:51:41.041333+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:15:13.030949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:15:13.030949+0000","last_clean_scrub_stamp":"2024-09-11T06:15:13.030949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:52:31.503430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015085280000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":782,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-11T06:51:41.086704+0000","last_change":"2024-09-11T06:15:16.164947+0000","last_active":"2024-09-11T06:51:41.086704+0000","last_peered":"2024-09-11T06:51:41.086704+0000","last_clean":"2024-09-11T06:51:41.086704+0000","last_became_active":"2024-09-11T06:15:14.142347+0000","last_became_peered":"2024-09-11T06:15:14.142347+0000","last_unstale":"2024-09-11T06:51:41.086704+0000","last_undegraded":"2024-09-11T06:51:41.086704+0000","last_fullsized":"2024-09-11T06:51:41.086704+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:15:13.030949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:15:13.030949+0000","last_clean_scrub_stamp":"2024-09-11T06:15:13.030949+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:26:00.950222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022347420000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":780,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-11T06:51:40.889467+0000","last_change":"2024-09-11T06:15:16.457217+0000","last_active":"2024-09-11T06:51:40.889467+0000","last_peered":"2024-09-11T06:51:40.889467+0000","last_clean":"2024-09-11T06:51:40.889467+0000","last_became_active":"2024-09-11T06:15:14.143512+0000","last_became_peered":"2024-09-11T06:15:14.143512+0000","last_unstale":"2024-09-11T06:51:40.889467+0000","last_undegraded":"2024-09-11T06:51:40.889467+0000","last_fullsized":"2024-09-11T06:51:40.889467+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:15:13.030949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:15:13.030949+0000","last_clean_scrub_stamp":"2024-09-11T06:15:13.030949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:31:31.971513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063951500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":781,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-11T06:51:40.890363+0000","last_change":"2024-09-11T06:15:16.457653+0000","last_active":"2024-09-11T06:51:40.890363+0000","last_peered":"2024-09-11T06:51:40.890363+0000","last_clean":"2024-09-11T06:51:40.890363+0000","last_became_active":"2024-09-11T06:15:14.142323+0000","last_became_peered":"2024-09-11T06:15:14.142323+0000","last_unstale":"2024-09-11T06:51:40.890363+0000","last_undegraded":"2024-09-11T06:51:40.890363+0000","last_fullsized":"2024-09-11T06:51:40.890363+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:15:13.030949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:15:13.030949+0000","last_clean_scrub_stamp":"2024-09-11T06:15:13.030949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:28:11.865252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091634599999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":782,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-11T06:51:41.045124+0000","last_change":"2024-09-11T06:15:16.157504+0000","last_active":"2024-09-11T06:51:41.045124+0000","last_peered":"2024-09-11T06:51:41.045124+0000","last_clean":"2024-09-11T06:51:41.045124+0000","last_became_active":"2024-09-11T06:15:14.142073+0000","last_became_peered":"2024-09-11T06:15:14.142073+0000","last_unstale":"2024-09-11T06:51:41.045124+0000","last_undegraded":"2024-09-11T06:51:41.045124+0000","last_fullsized":"2024-09-11T06:51:41.045124+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:15:13.030949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:15:13.030949+0000","last_clean_scrub_stamp":"2024-09-11T06:15:13.030949+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:16:44.214033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010516950000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":908,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-11T06:51:41.045561+0000","last_change":"2024-09-11T06:15:12.513296+0000","last_active":"2024-09-11T06:51:41.045561+0000","last_peered":"2024-09-11T06:51:41.045561+0000","last_clean":"2024-09-11T06:51:41.045561+0000","last_became_active":"2024-09-11T06:15:12.511458+0000","last_became_peered":"2024-09-11T06:15:12.511458+0000","last_unstale":"2024-09-11T06:51:41.045561+0000","last_undegraded":"2024-09-11T06:51:41.045561+0000","last_fullsized":"2024-09-11T06:51:41.045561+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:15:11.022532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:15:11.022532+0000","last_clean_scrub_stamp":"2024-09-11T06:15:11.022532+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:14:38.108521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7472,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738814,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":805464,"kb_used_data":4484,"kb_used_omap":94,"kb_used_meta":800865,"kb_avail":93566376,"statfs":{"total":96636764160,"available":95811969024,"internally_reserved":0,"allocated":4591616,"data_stored":2290133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":96428,"internal_metadata":820086612},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738815,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1289880,"kb_used_data":4544,"kb_used_omap":92,"kb_used_meta":1285219,"kb_avail":93081960,"statfs":{"total":96636764160,"available":95315927040,"internally_reserved":0,"allocated":4653056,"data_stored":2880520,"data_compressed":3736,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":94588,"internal_metadata":1316064900},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738814,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1555284,"kb_used_data":4544,"kb_used_omap":105,"kb_used_meta":1550614,"kb_avail":92816556,"statfs":{"total":96636764160,"available":95044153344,"internally_reserved":0,"allocated":4653056,"data_stored":2880520,"data_compressed":3736,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":107997,"internal_metadata":1587829283},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3736,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3736,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T06:51:50.064 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-11T06:15:13.030949+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=6, tm_min=15, tm_sec=13, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=6, tm_min=51, tm_sec=47, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T06:51:50.064 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-09-11T06:15:13.030949+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=6, tm_min=15, tm_sec=13, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=6, tm_min=51, tm_sec=47, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T06:51:50.064 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-11T06:15:13.030949+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=6, tm_min=15, tm_sec=13, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=6, tm_min=51, tm_sec=47, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T06:51:50.064 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-09-11T06:15:13.030949+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=6, tm_min=15, tm_sec=13, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=6, tm_min=51, tm_sec=47, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T06:51:50.064 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-09-11T06:15:13.030949+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=6, tm_min=15, tm_sec=13, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=6, tm_min=51, tm_sec=47, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T06:51:50.064 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-09-11T06:15:13.030949+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=6, tm_min=15, tm_sec=13, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=6, tm_min=51, tm_sec=47, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T06:51:50.064 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-11T06:15:13.030949+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=6, tm_min=15, tm_sec=13, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=6, tm_min=51, tm_sec=47, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T06:51:50.064 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-09-11T06:15:13.030949+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=6, tm_min=15, tm_sec=13, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=6, tm_min=51, tm_sec=47, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T06:51:50.065 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-11T06:15:11.022532+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=6, tm_min=15, tm_sec=11, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=6, tm_min=51, tm_sec=47, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T06:51:50.065 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-11T06:52:10.066 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-11T06:52:10.378 INFO:teuthology.orchestra.run.smithi112.stdout: 2024-09-11T06:52:10.378 INFO:teuthology.orchestra.run.smithi112.stderr:dumped all 2024-09-11T06:52:10.394 INFO:teuthology.orchestra.run.smithi112.stdout:{"pg_ready":true,"pg_map":{"version":1496,"stamp":"2024-09-11T06:52:09.730814+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":104,"ondisk_log_size":104,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3650612,"kb_used_data":13572,"kb_used_omap":292,"kb_used_meta":3636699,"kb_avail":279464908,"statfs":{"total":289910292480,"available":286172065792,"internally_reserved":0,"allocated":13897728,"data_stored":8051173,"data_compressed":7472,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":299013,"internal_metadata":3723980795},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002985"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":790,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-11T06:51:53.660924+0000","last_change":"2024-09-11T06:51:53.660924+0000","last_active":"2024-09-11T06:51:53.660924+0000","last_peered":"2024-09-11T06:51:53.660924+0000","last_clean":"2024-09-11T06:51:53.660924+0000","last_became_active":"2024-09-11T06:15:14.143307+0000","last_became_peered":"2024-09-11T06:15:14.143307+0000","last_unstale":"2024-09-11T06:51:53.660924+0000","last_undegraded":"2024-09-11T06:51:53.660924+0000","last_fullsized":"2024-09-11T06:51:53.660924+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:51:53.660674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:51:53.660674+0000","last_clean_scrub_stamp":"2024-09-11T06:51:53.660674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T17:30:10.926894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019449249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":790,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-11T06:51:51.635920+0000","last_change":"2024-09-11T06:51:51.635920+0000","last_active":"2024-09-11T06:51:51.635920+0000","last_peered":"2024-09-11T06:51:51.635920+0000","last_clean":"2024-09-11T06:51:51.635920+0000","last_became_active":"2024-09-11T06:15:14.142832+0000","last_became_peered":"2024-09-11T06:15:14.142832+0000","last_unstale":"2024-09-11T06:51:51.635920+0000","last_undegraded":"2024-09-11T06:51:51.635920+0000","last_fullsized":"2024-09-11T06:51:51.635920+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:51:51.635642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:51:51.635642+0000","last_clean_scrub_stamp":"2024-09-11T06:51:51.635642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:29:10.059557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001083002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":790,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-11T06:51:54.650174+0000","last_change":"2024-09-11T06:51:54.650229+0000","last_active":"2024-09-11T06:51:54.650174+0000","last_peered":"2024-09-11T06:51:54.650174+0000","last_clean":"2024-09-11T06:51:54.650174+0000","last_became_active":"2024-09-11T06:15:14.143100+0000","last_became_peered":"2024-09-11T06:15:14.143100+0000","last_unstale":"2024-09-11T06:51:54.650174+0000","last_undegraded":"2024-09-11T06:51:54.650174+0000","last_fullsized":"2024-09-11T06:51:54.650174+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:51:54.649980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:51:54.649980+0000","last_clean_scrub_stamp":"2024-09-11T06:51:54.649980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:33:02.716853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001527928,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":790,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-11T06:51:52.639874+0000","last_change":"2024-09-11T06:51:52.639929+0000","last_active":"2024-09-11T06:51:52.639874+0000","last_peered":"2024-09-11T06:51:52.639874+0000","last_clean":"2024-09-11T06:51:52.639874+0000","last_became_active":"2024-09-11T06:15:14.142926+0000","last_became_peered":"2024-09-11T06:15:14.142926+0000","last_unstale":"2024-09-11T06:51:52.639874+0000","last_undegraded":"2024-09-11T06:51:52.639874+0000","last_fullsized":"2024-09-11T06:51:52.639874+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:51:52.639651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:51:52.639651+0000","last_clean_scrub_stamp":"2024-09-11T06:51:52.639651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:24:54.558470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015085280000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"388'3","reported_seq":792,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-11T06:51:49.358374+0000","last_change":"2024-09-11T06:51:49.358430+0000","last_active":"2024-09-11T06:51:49.358374+0000","last_peered":"2024-09-11T06:51:49.358374+0000","last_clean":"2024-09-11T06:51:49.358374+0000","last_became_active":"2024-09-11T06:15:14.142347+0000","last_became_peered":"2024-09-11T06:15:14.142347+0000","last_unstale":"2024-09-11T06:51:49.358374+0000","last_undegraded":"2024-09-11T06:51:49.358374+0000","last_fullsized":"2024-09-11T06:51:49.358374+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'3","last_scrub_stamp":"2024-09-11T06:51:49.358260+0000","last_deep_scrub":"388'3","last_deep_scrub_stamp":"2024-09-11T06:51:49.358260+0000","last_clean_scrub_stamp":"2024-09-11T06:51:49.358260+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:18:08.783852+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0022347420000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":789,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-11T06:51:51.153909+0000","last_change":"2024-09-11T06:51:51.154009+0000","last_active":"2024-09-11T06:51:51.153909+0000","last_peered":"2024-09-11T06:51:51.153909+0000","last_clean":"2024-09-11T06:51:51.153909+0000","last_became_active":"2024-09-11T06:15:14.143512+0000","last_became_peered":"2024-09-11T06:15:14.143512+0000","last_unstale":"2024-09-11T06:51:51.153909+0000","last_undegraded":"2024-09-11T06:51:51.153909+0000","last_fullsized":"2024-09-11T06:51:51.153909+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:51:51.153599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:51:51.153599+0000","last_clean_scrub_stamp":"2024-09-11T06:51:51.153599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:00:46.707503+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00063951500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":790,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-11T06:51:50.122336+0000","last_change":"2024-09-11T06:51:50.122355+0000","last_active":"2024-09-11T06:51:50.122336+0000","last_peered":"2024-09-11T06:51:50.122336+0000","last_clean":"2024-09-11T06:51:50.122336+0000","last_became_active":"2024-09-11T06:15:14.142323+0000","last_became_peered":"2024-09-11T06:15:14.142323+0000","last_unstale":"2024-09-11T06:51:50.122336+0000","last_undegraded":"2024-09-11T06:51:50.122336+0000","last_fullsized":"2024-09-11T06:51:50.122336+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:51:50.122260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:51:50.122260+0000","last_clean_scrub_stamp":"2024-09-11T06:51:50.122260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:56:50.008262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091634599999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"388'2","reported_seq":792,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-11T06:51:50.664972+0000","last_change":"2024-09-11T06:51:50.665109+0000","last_active":"2024-09-11T06:51:50.664972+0000","last_peered":"2024-09-11T06:51:50.664972+0000","last_clean":"2024-09-11T06:51:50.664972+0000","last_became_active":"2024-09-11T06:15:14.142073+0000","last_became_peered":"2024-09-11T06:15:14.142073+0000","last_unstale":"2024-09-11T06:51:50.664972+0000","last_undegraded":"2024-09-11T06:51:50.664972+0000","last_fullsized":"2024-09-11T06:51:50.664972+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'2","last_scrub_stamp":"2024-09-11T06:51:50.664724+0000","last_deep_scrub":"388'2","last_deep_scrub_stamp":"2024-09-11T06:51:50.664724+0000","last_clean_scrub_stamp":"2024-09-11T06:51:50.664724+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:42:57.187448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010516950000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"388'99","reported_seq":919,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-11T06:51:49.625358+0000","last_change":"2024-09-11T06:51:49.625421+0000","last_active":"2024-09-11T06:51:49.625358+0000","last_peered":"2024-09-11T06:51:49.625358+0000","last_clean":"2024-09-11T06:51:49.625358+0000","last_became_active":"2024-09-11T06:15:12.511458+0000","last_became_peered":"2024-09-11T06:15:12.511458+0000","last_unstale":"2024-09-11T06:51:49.625358+0000","last_undegraded":"2024-09-11T06:51:49.625358+0000","last_fullsized":"2024-09-11T06:51:49.625358+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'99","last_scrub_stamp":"2024-09-11T06:51:49.625276+0000","last_deep_scrub":"388'99","last_deep_scrub_stamp":"2024-09-11T06:51:49.625276+0000","last_clean_scrub_stamp":"2024-09-11T06:51:49.625276+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:36:37.822512+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7472,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738818,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":805460,"kb_used_data":4484,"kb_used_omap":94,"kb_used_meta":800865,"kb_avail":93566380,"statfs":{"total":96636764160,"available":95811973120,"internally_reserved":0,"allocated":4591616,"data_stored":2290133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":96428,"internal_metadata":820086612},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738819,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1289872,"kb_used_data":4544,"kb_used_omap":92,"kb_used_meta":1285219,"kb_avail":93081968,"statfs":{"total":96636764160,"available":95315935232,"internally_reserved":0,"allocated":4653056,"data_stored":2880520,"data_compressed":3736,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":94588,"internal_metadata":1316064900},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738818,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1555280,"kb_used_data":4544,"kb_used_omap":105,"kb_used_meta":1550614,"kb_avail":92816560,"statfs":{"total":96636764160,"available":95044157440,"internally_reserved":0,"allocated":4653056,"data_stored":2880520,"data_compressed":3736,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":107997,"internal_metadata":1587829283},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3736,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3736,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T06:52:10.395 DEBUG:teuthology.orchestra.run.smithi112:> sudo ceph --cluster cluster2 config set global mon_health_to_clog false 2024-09-11T06:52:10.702 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-11T06:52:10.703 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-11T06:52:10.703 DEBUG:tasks.ceph.cluster2.osd.0:waiting for process to exit 2024-09-11T06:52:10.703 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T06:52:14.806 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~0s 2024-09-11T06:52:16.804 INFO:tasks.ceph.cluster2.osd.0:Stopped 2024-09-11T06:52:16.805 DEBUG:tasks.ceph.cluster2.osd.1:waiting for process to exit 2024-09-11T06:52:16.805 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T06:52:20.109 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.1 is failed for ~0s 2024-09-11T06:52:20.110 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 has been restored 2024-09-11T06:52:22.907 INFO:tasks.ceph.cluster2.osd.1:Stopped 2024-09-11T06:52:22.907 DEBUG:tasks.ceph.cluster2.osd.2:waiting for process to exit 2024-09-11T06:52:22.908 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T06:52:25.312 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.2 is failed for ~0s 2024-09-11T06:52:25.312 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.1 has been restored 2024-09-11T06:52:29.010 INFO:tasks.ceph.cluster2.osd.2:Stopped 2024-09-11T06:52:29.010 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-11T06:52:29.011 DEBUG:tasks.ceph.cluster2.mgr.x:waiting for process to exit 2024-09-11T06:52:29.011 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T06:52:29.108 INFO:tasks.ceph.cluster2.mgr.x:Stopped 2024-09-11T06:52:29.108 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-11T06:52:29.108 DEBUG:tasks.ceph.cluster2.mon.a:waiting for process to exit 2024-09-11T06:52:29.108 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T06:52:29.137 INFO:tasks.ceph.cluster2.mon.a:Stopped 2024-09-11T06:52:29.137 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-11T06:52:29.137 DEBUG:teuthology.orchestra.run.smithi112:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/cluster2.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2024-09-11T06:52:29.512 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-0 on ubuntu@smithi112.front.sepia.ceph.com 2024-09-11T06:52:29.512 DEBUG:teuthology.orchestra.run.smithi112:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-0 2024-09-11T06:52:29.786 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-1 on ubuntu@smithi112.front.sepia.ceph.com 2024-09-11T06:52:29.786 DEBUG:teuthology.orchestra.run.smithi112:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-1 2024-09-11T06:52:30.069 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-2 on ubuntu@smithi112.front.sepia.ceph.com 2024-09-11T06:52:30.069 DEBUG:teuthology.orchestra.run.smithi112:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-2 2024-09-11T06:52:30.314 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.2 has been restored 2024-09-11T06:52:30.331 INFO:tasks.ceph:Archiving mon data... 2024-09-11T06:52:30.332 DEBUG:teuthology.misc:Transferring archived files from smithi112:/var/lib/ceph/mon/cluster2-a to /home/teuthworker/archive/teuthology-2024-09-10_21:16:03-rbd-squid-distro-default-smithi/7900605/data/cluster2.mon.a.tgz 2024-09-11T06:52:30.332 DEBUG:teuthology.orchestra.run.smithi112:> mktemp 2024-09-11T06:52:30.337 INFO:teuthology.orchestra.run.smithi112.stdout:/tmp/tmp.CoxEIqrXnb 2024-09-11T06:52:30.337 DEBUG:teuthology.orchestra.run.smithi112:> sudo tar cz -f - -C /var/lib/ceph/mon/cluster2-a -- . > /tmp/tmp.CoxEIqrXnb 2024-09-11T06:52:30.482 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod 0666 /tmp/tmp.CoxEIqrXnb 2024-09-11T06:52:30.617 DEBUG:teuthology.orchestra.remote:smithi112:/tmp/tmp.CoxEIqrXnb is 409KB 2024-09-11T06:52:30.699 DEBUG:teuthology.orchestra.run.smithi112:> rm -fr /tmp/tmp.CoxEIqrXnb 2024-09-11T06:52:30.706 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-11T06:52:30.707 DEBUG:teuthology.orchestra.run.smithi006:> sudo rm -rf -- /etc/ceph/cluster2.conf /etc/ceph/cluster2.keyring /home/ubuntu/cephtest/cluster2.data /home/ubuntu/cephtest/cluster2.monmap /home/ubuntu/cephtest/../*.pid 2024-09-11T06:52:30.711 DEBUG:teuthology.orchestra.run.smithi112:> sudo rm -rf -- /etc/ceph/cluster2.conf /etc/ceph/cluster2.keyring /home/ubuntu/cephtest/cluster2.data /home/ubuntu/cephtest/cluster2.monmap /home/ubuntu/cephtest/../*.pid 2024-09-11T06:52:30.940 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-11T06:52:30.951 INFO:tasks.ceph.ceph_manager.cluster1:waiting for clean 2024-09-11T06:52:30.952 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-11T06:52:31.481 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-11T06:52:31.482 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-09-11T06:52:31.514 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":1464,"stamp":"2024-09-11T06:52:29.793686+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4544972,"kb_used_data":8016,"kb_used_omap":444,"kb_used_meta":4536451,"kb_avail":278570548,"statfs":{"total":289910292480,"available":285256241152,"internally_reserved":0,"allocated":8208384,"data_stored":5436979,"data_compressed":7456,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":454986,"internal_metadata":4645326518},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002905"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":658,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-09-11T06:51:40.690932+0000","last_change":"2024-09-11T06:14:37.124927+0000","last_active":"2024-09-11T06:51:40.690932+0000","last_peered":"2024-09-11T06:51:40.690932+0000","last_clean":"2024-09-11T06:51:40.690932+0000","last_became_active":"2024-09-11T06:14:35.103646+0000","last_became_peered":"2024-09-11T06:14:35.103646+0000","last_unstale":"2024-09-11T06:51:40.690932+0000","last_undegraded":"2024-09-11T06:51:40.690932+0000","last_fullsized":"2024-09-11T06:51:40.690932+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:14:34.076871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:14:34.076871+0000","last_clean_scrub_stamp":"2024-09-11T06:14:34.076871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:17:56.558226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00180295,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":658,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-09-11T06:51:40.689979+0000","last_change":"2024-09-11T06:14:37.124816+0000","last_active":"2024-09-11T06:51:40.689979+0000","last_peered":"2024-09-11T06:51:40.689979+0000","last_clean":"2024-09-11T06:51:40.689979+0000","last_became_active":"2024-09-11T06:14:35.103523+0000","last_became_peered":"2024-09-11T06:14:35.103523+0000","last_unstale":"2024-09-11T06:51:40.689979+0000","last_undegraded":"2024-09-11T06:51:40.689979+0000","last_fullsized":"2024-09-11T06:51:40.689979+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:14:34.076871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:14:34.076871+0000","last_clean_scrub_stamp":"2024-09-11T06:14:34.076871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:49:06.803512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016837309999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":659,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-09-11T06:51:40.693811+0000","last_change":"2024-09-11T06:14:37.125460+0000","last_active":"2024-09-11T06:51:40.693811+0000","last_peered":"2024-09-11T06:51:40.693811+0000","last_clean":"2024-09-11T06:51:40.693811+0000","last_became_active":"2024-09-11T06:14:35.103464+0000","last_became_peered":"2024-09-11T06:14:35.103464+0000","last_unstale":"2024-09-11T06:51:40.693811+0000","last_undegraded":"2024-09-11T06:51:40.693811+0000","last_fullsized":"2024-09-11T06:51:40.693811+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:14:34.076871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:14:34.076871+0000","last_clean_scrub_stamp":"2024-09-11T06:14:34.076871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T17:30:58.380858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022336330000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":659,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-09-11T06:51:40.685353+0000","last_change":"2024-09-11T06:14:37.124096+0000","last_active":"2024-09-11T06:51:40.685353+0000","last_peered":"2024-09-11T06:51:40.685353+0000","last_clean":"2024-09-11T06:51:40.685353+0000","last_became_active":"2024-09-11T06:14:35.101427+0000","last_became_peered":"2024-09-11T06:14:35.101427+0000","last_unstale":"2024-09-11T06:51:40.685353+0000","last_undegraded":"2024-09-11T06:51:40.685353+0000","last_fullsized":"2024-09-11T06:51:40.685353+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:14:34.076871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:14:34.076871+0000","last_clean_scrub_stamp":"2024-09-11T06:14:34.076871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:37:23.166823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001085391,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":661,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-09-11T06:51:41.771597+0000","last_change":"2024-09-11T06:14:37.115786+0000","last_active":"2024-09-11T06:51:41.771597+0000","last_peered":"2024-09-11T06:51:41.771597+0000","last_clean":"2024-09-11T06:51:41.771597+0000","last_became_active":"2024-09-11T06:14:35.105259+0000","last_became_peered":"2024-09-11T06:14:35.105259+0000","last_unstale":"2024-09-11T06:51:41.771597+0000","last_undegraded":"2024-09-11T06:51:41.771597+0000","last_fullsized":"2024-09-11T06:51:41.771597+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:14:34.076871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:14:34.076871+0000","last_clean_scrub_stamp":"2024-09-11T06:14:34.076871+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:33:55.339306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092280599999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":659,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-09-11T06:51:40.440930+0000","last_change":"2024-09-11T06:14:37.343308+0000","last_active":"2024-09-11T06:51:40.440930+0000","last_peered":"2024-09-11T06:51:40.440930+0000","last_clean":"2024-09-11T06:51:40.440930+0000","last_became_active":"2024-09-11T06:14:35.104976+0000","last_became_peered":"2024-09-11T06:14:35.104976+0000","last_unstale":"2024-09-11T06:51:40.440930+0000","last_undegraded":"2024-09-11T06:51:40.440930+0000","last_fullsized":"2024-09-11T06:51:40.440930+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:14:34.076871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:14:34.076871+0000","last_clean_scrub_stamp":"2024-09-11T06:14:34.076871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:33:14.682684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001060088,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":659,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-09-11T06:51:40.433182+0000","last_change":"2024-09-11T06:14:37.343210+0000","last_active":"2024-09-11T06:51:40.433182+0000","last_peered":"2024-09-11T06:51:40.433182+0000","last_clean":"2024-09-11T06:51:40.433182+0000","last_became_active":"2024-09-11T06:14:35.104951+0000","last_became_peered":"2024-09-11T06:14:35.104951+0000","last_unstale":"2024-09-11T06:51:40.433182+0000","last_undegraded":"2024-09-11T06:51:40.433182+0000","last_fullsized":"2024-09-11T06:51:40.433182+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:14:34.076871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:14:34.076871+0000","last_clean_scrub_stamp":"2024-09-11T06:14:34.076871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:18:43.513138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087051999999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":660,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-09-11T06:51:40.695963+0000","last_change":"2024-09-11T06:14:37.124468+0000","last_active":"2024-09-11T06:51:40.695963+0000","last_peered":"2024-09-11T06:51:40.695963+0000","last_clean":"2024-09-11T06:51:40.695963+0000","last_became_active":"2024-09-11T06:14:35.099575+0000","last_became_peered":"2024-09-11T06:14:35.099575+0000","last_unstale":"2024-09-11T06:51:40.695963+0000","last_undegraded":"2024-09-11T06:51:40.695963+0000","last_fullsized":"2024-09-11T06:51:40.695963+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:14:34.076871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:14:34.076871+0000","last_clean_scrub_stamp":"2024-09-11T06:14:34.076871+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:34:43.189610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010604620000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":787,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-09-11T06:51:40.695770+0000","last_change":"2024-09-11T06:14:32.080441+0000","last_active":"2024-09-11T06:51:40.695770+0000","last_peered":"2024-09-11T06:51:40.695770+0000","last_clean":"2024-09-11T06:51:40.695770+0000","last_became_active":"2024-09-11T06:14:32.077706+0000","last_became_peered":"2024-09-11T06:14:32.077706+0000","last_unstale":"2024-09-11T06:51:40.695770+0000","last_undegraded":"2024-09-11T06:51:40.695770+0000","last_fullsized":"2024-09-11T06:51:40.695770+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:14:31.066451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:14:31.066451+0000","last_clean_scrub_stamp":"2024-09-11T06:14:31.066451+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:50:25.120392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7456,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949673421,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1273628,"kb_used_data":2632,"kb_used_omap":159,"kb_used_meta":1270816,"kb_avail":93098212,"statfs":{"total":96636764160,"available":95332569088,"internally_reserved":0,"allocated":2695168,"data_stored":1418735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":162976,"internal_metadata":1301316448},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706126,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1684184,"kb_used_data":2692,"kb_used_omap":134,"kb_used_meta":1681337,"kb_avail":92687656,"statfs":{"total":96636764160,"available":94912159744,"internally_reserved":0,"allocated":2756608,"data_stored":2009122,"data_compressed":3728,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":137978,"internal_metadata":1721689350},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706124,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1587160,"kb_used_data":2692,"kb_used_omap":150,"kb_used_meta":1584297,"kb_avail":92784680,"statfs":{"total":96636764160,"available":95011512320,"internally_reserved":0,"allocated":2756608,"data_stored":2009122,"data_compressed":3728,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":154032,"internal_metadata":1622320720},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3728,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3728,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T06:52:31.515 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-11T06:52:31.820 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-11T06:52:31.820 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-09-11T06:52:31.839 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":1465,"stamp":"2024-09-11T06:52:31.794174+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4544972,"kb_used_data":8016,"kb_used_omap":444,"kb_used_meta":4536451,"kb_avail":278570548,"statfs":{"total":289910292480,"available":285256241152,"internally_reserved":0,"allocated":8208384,"data_stored":5436979,"data_compressed":7456,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":454986,"internal_metadata":4645326518},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002936"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":658,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-09-11T06:51:40.690932+0000","last_change":"2024-09-11T06:14:37.124927+0000","last_active":"2024-09-11T06:51:40.690932+0000","last_peered":"2024-09-11T06:51:40.690932+0000","last_clean":"2024-09-11T06:51:40.690932+0000","last_became_active":"2024-09-11T06:14:35.103646+0000","last_became_peered":"2024-09-11T06:14:35.103646+0000","last_unstale":"2024-09-11T06:51:40.690932+0000","last_undegraded":"2024-09-11T06:51:40.690932+0000","last_fullsized":"2024-09-11T06:51:40.690932+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:14:34.076871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:14:34.076871+0000","last_clean_scrub_stamp":"2024-09-11T06:14:34.076871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:17:56.558226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00180295,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":658,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-09-11T06:51:40.689979+0000","last_change":"2024-09-11T06:14:37.124816+0000","last_active":"2024-09-11T06:51:40.689979+0000","last_peered":"2024-09-11T06:51:40.689979+0000","last_clean":"2024-09-11T06:51:40.689979+0000","last_became_active":"2024-09-11T06:14:35.103523+0000","last_became_peered":"2024-09-11T06:14:35.103523+0000","last_unstale":"2024-09-11T06:51:40.689979+0000","last_undegraded":"2024-09-11T06:51:40.689979+0000","last_fullsized":"2024-09-11T06:51:40.689979+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:14:34.076871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:14:34.076871+0000","last_clean_scrub_stamp":"2024-09-11T06:14:34.076871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:49:06.803512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016837309999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":659,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-09-11T06:51:40.693811+0000","last_change":"2024-09-11T06:14:37.125460+0000","last_active":"2024-09-11T06:51:40.693811+0000","last_peered":"2024-09-11T06:51:40.693811+0000","last_clean":"2024-09-11T06:51:40.693811+0000","last_became_active":"2024-09-11T06:14:35.103464+0000","last_became_peered":"2024-09-11T06:14:35.103464+0000","last_unstale":"2024-09-11T06:51:40.693811+0000","last_undegraded":"2024-09-11T06:51:40.693811+0000","last_fullsized":"2024-09-11T06:51:40.693811+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:14:34.076871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:14:34.076871+0000","last_clean_scrub_stamp":"2024-09-11T06:14:34.076871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T17:30:58.380858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022336330000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":659,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-09-11T06:51:40.685353+0000","last_change":"2024-09-11T06:14:37.124096+0000","last_active":"2024-09-11T06:51:40.685353+0000","last_peered":"2024-09-11T06:51:40.685353+0000","last_clean":"2024-09-11T06:51:40.685353+0000","last_became_active":"2024-09-11T06:14:35.101427+0000","last_became_peered":"2024-09-11T06:14:35.101427+0000","last_unstale":"2024-09-11T06:51:40.685353+0000","last_undegraded":"2024-09-11T06:51:40.685353+0000","last_fullsized":"2024-09-11T06:51:40.685353+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:14:34.076871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:14:34.076871+0000","last_clean_scrub_stamp":"2024-09-11T06:14:34.076871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:37:23.166823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001085391,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":661,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-09-11T06:51:41.771597+0000","last_change":"2024-09-11T06:14:37.115786+0000","last_active":"2024-09-11T06:51:41.771597+0000","last_peered":"2024-09-11T06:51:41.771597+0000","last_clean":"2024-09-11T06:51:41.771597+0000","last_became_active":"2024-09-11T06:14:35.105259+0000","last_became_peered":"2024-09-11T06:14:35.105259+0000","last_unstale":"2024-09-11T06:51:41.771597+0000","last_undegraded":"2024-09-11T06:51:41.771597+0000","last_fullsized":"2024-09-11T06:51:41.771597+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:14:34.076871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:14:34.076871+0000","last_clean_scrub_stamp":"2024-09-11T06:14:34.076871+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:33:55.339306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092280599999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":659,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-09-11T06:51:40.440930+0000","last_change":"2024-09-11T06:14:37.343308+0000","last_active":"2024-09-11T06:51:40.440930+0000","last_peered":"2024-09-11T06:51:40.440930+0000","last_clean":"2024-09-11T06:51:40.440930+0000","last_became_active":"2024-09-11T06:14:35.104976+0000","last_became_peered":"2024-09-11T06:14:35.104976+0000","last_unstale":"2024-09-11T06:51:40.440930+0000","last_undegraded":"2024-09-11T06:51:40.440930+0000","last_fullsized":"2024-09-11T06:51:40.440930+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:14:34.076871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:14:34.076871+0000","last_clean_scrub_stamp":"2024-09-11T06:14:34.076871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:33:14.682684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001060088,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":659,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-09-11T06:51:40.433182+0000","last_change":"2024-09-11T06:14:37.343210+0000","last_active":"2024-09-11T06:51:40.433182+0000","last_peered":"2024-09-11T06:51:40.433182+0000","last_clean":"2024-09-11T06:51:40.433182+0000","last_became_active":"2024-09-11T06:14:35.104951+0000","last_became_peered":"2024-09-11T06:14:35.104951+0000","last_unstale":"2024-09-11T06:51:40.433182+0000","last_undegraded":"2024-09-11T06:51:40.433182+0000","last_fullsized":"2024-09-11T06:51:40.433182+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:14:34.076871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:14:34.076871+0000","last_clean_scrub_stamp":"2024-09-11T06:14:34.076871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:18:43.513138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087051999999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":660,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-09-11T06:51:40.695963+0000","last_change":"2024-09-11T06:14:37.124468+0000","last_active":"2024-09-11T06:51:40.695963+0000","last_peered":"2024-09-11T06:51:40.695963+0000","last_clean":"2024-09-11T06:51:40.695963+0000","last_became_active":"2024-09-11T06:14:35.099575+0000","last_became_peered":"2024-09-11T06:14:35.099575+0000","last_unstale":"2024-09-11T06:51:40.695963+0000","last_undegraded":"2024-09-11T06:51:40.695963+0000","last_fullsized":"2024-09-11T06:51:40.695963+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:14:34.076871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:14:34.076871+0000","last_clean_scrub_stamp":"2024-09-11T06:14:34.076871+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:34:43.189610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010604620000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":787,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-09-11T06:51:40.695770+0000","last_change":"2024-09-11T06:14:32.080441+0000","last_active":"2024-09-11T06:51:40.695770+0000","last_peered":"2024-09-11T06:51:40.695770+0000","last_clean":"2024-09-11T06:51:40.695770+0000","last_became_active":"2024-09-11T06:14:32.077706+0000","last_became_peered":"2024-09-11T06:14:32.077706+0000","last_unstale":"2024-09-11T06:51:40.695770+0000","last_undegraded":"2024-09-11T06:51:40.695770+0000","last_fullsized":"2024-09-11T06:51:40.695770+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:14:31.066451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:14:31.066451+0000","last_clean_scrub_stamp":"2024-09-11T06:14:31.066451+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:50:25.120392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7456,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949673421,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1273628,"kb_used_data":2632,"kb_used_omap":159,"kb_used_meta":1270816,"kb_avail":93098212,"statfs":{"total":96636764160,"available":95332569088,"internally_reserved":0,"allocated":2695168,"data_stored":1418735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":162976,"internal_metadata":1301316448},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706127,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1684184,"kb_used_data":2692,"kb_used_omap":134,"kb_used_meta":1681337,"kb_avail":92687656,"statfs":{"total":96636764160,"available":94912159744,"internally_reserved":0,"allocated":2756608,"data_stored":2009122,"data_compressed":3728,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":137978,"internal_metadata":1721689350},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706125,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1587160,"kb_used_data":2692,"kb_used_omap":150,"kb_used_meta":1584297,"kb_avail":92784680,"statfs":{"total":96636764160,"available":95011512320,"internally_reserved":0,"allocated":2756608,"data_stored":2009122,"data_compressed":3728,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":154032,"internal_metadata":1622320720},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3728,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3728,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T06:52:31.840 INFO:tasks.ceph.ceph_manager.cluster1:clean! 2024-09-11T06:52:31.840 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-11T06:52:32.142 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-11T06:52:32.142 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-09-11T06:52:32.159 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":1465,"stamp":"2024-09-11T06:52:31.794174+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4544972,"kb_used_data":8016,"kb_used_omap":444,"kb_used_meta":4536451,"kb_avail":278570548,"statfs":{"total":289910292480,"available":285256241152,"internally_reserved":0,"allocated":8208384,"data_stored":5436979,"data_compressed":7456,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":454986,"internal_metadata":4645326518},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002936"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":658,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-09-11T06:51:40.690932+0000","last_change":"2024-09-11T06:14:37.124927+0000","last_active":"2024-09-11T06:51:40.690932+0000","last_peered":"2024-09-11T06:51:40.690932+0000","last_clean":"2024-09-11T06:51:40.690932+0000","last_became_active":"2024-09-11T06:14:35.103646+0000","last_became_peered":"2024-09-11T06:14:35.103646+0000","last_unstale":"2024-09-11T06:51:40.690932+0000","last_undegraded":"2024-09-11T06:51:40.690932+0000","last_fullsized":"2024-09-11T06:51:40.690932+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:14:34.076871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:14:34.076871+0000","last_clean_scrub_stamp":"2024-09-11T06:14:34.076871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:17:56.558226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00180295,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":658,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-09-11T06:51:40.689979+0000","last_change":"2024-09-11T06:14:37.124816+0000","last_active":"2024-09-11T06:51:40.689979+0000","last_peered":"2024-09-11T06:51:40.689979+0000","last_clean":"2024-09-11T06:51:40.689979+0000","last_became_active":"2024-09-11T06:14:35.103523+0000","last_became_peered":"2024-09-11T06:14:35.103523+0000","last_unstale":"2024-09-11T06:51:40.689979+0000","last_undegraded":"2024-09-11T06:51:40.689979+0000","last_fullsized":"2024-09-11T06:51:40.689979+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:14:34.076871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:14:34.076871+0000","last_clean_scrub_stamp":"2024-09-11T06:14:34.076871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:49:06.803512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016837309999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":659,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-09-11T06:51:40.693811+0000","last_change":"2024-09-11T06:14:37.125460+0000","last_active":"2024-09-11T06:51:40.693811+0000","last_peered":"2024-09-11T06:51:40.693811+0000","last_clean":"2024-09-11T06:51:40.693811+0000","last_became_active":"2024-09-11T06:14:35.103464+0000","last_became_peered":"2024-09-11T06:14:35.103464+0000","last_unstale":"2024-09-11T06:51:40.693811+0000","last_undegraded":"2024-09-11T06:51:40.693811+0000","last_fullsized":"2024-09-11T06:51:40.693811+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:14:34.076871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:14:34.076871+0000","last_clean_scrub_stamp":"2024-09-11T06:14:34.076871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T17:30:58.380858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022336330000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":659,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-09-11T06:51:40.685353+0000","last_change":"2024-09-11T06:14:37.124096+0000","last_active":"2024-09-11T06:51:40.685353+0000","last_peered":"2024-09-11T06:51:40.685353+0000","last_clean":"2024-09-11T06:51:40.685353+0000","last_became_active":"2024-09-11T06:14:35.101427+0000","last_became_peered":"2024-09-11T06:14:35.101427+0000","last_unstale":"2024-09-11T06:51:40.685353+0000","last_undegraded":"2024-09-11T06:51:40.685353+0000","last_fullsized":"2024-09-11T06:51:40.685353+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:14:34.076871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:14:34.076871+0000","last_clean_scrub_stamp":"2024-09-11T06:14:34.076871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:37:23.166823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001085391,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":661,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-09-11T06:51:41.771597+0000","last_change":"2024-09-11T06:14:37.115786+0000","last_active":"2024-09-11T06:51:41.771597+0000","last_peered":"2024-09-11T06:51:41.771597+0000","last_clean":"2024-09-11T06:51:41.771597+0000","last_became_active":"2024-09-11T06:14:35.105259+0000","last_became_peered":"2024-09-11T06:14:35.105259+0000","last_unstale":"2024-09-11T06:51:41.771597+0000","last_undegraded":"2024-09-11T06:51:41.771597+0000","last_fullsized":"2024-09-11T06:51:41.771597+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:14:34.076871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:14:34.076871+0000","last_clean_scrub_stamp":"2024-09-11T06:14:34.076871+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:33:55.339306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092280599999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":659,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-09-11T06:51:40.440930+0000","last_change":"2024-09-11T06:14:37.343308+0000","last_active":"2024-09-11T06:51:40.440930+0000","last_peered":"2024-09-11T06:51:40.440930+0000","last_clean":"2024-09-11T06:51:40.440930+0000","last_became_active":"2024-09-11T06:14:35.104976+0000","last_became_peered":"2024-09-11T06:14:35.104976+0000","last_unstale":"2024-09-11T06:51:40.440930+0000","last_undegraded":"2024-09-11T06:51:40.440930+0000","last_fullsized":"2024-09-11T06:51:40.440930+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:14:34.076871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:14:34.076871+0000","last_clean_scrub_stamp":"2024-09-11T06:14:34.076871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:33:14.682684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001060088,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":659,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-09-11T06:51:40.433182+0000","last_change":"2024-09-11T06:14:37.343210+0000","last_active":"2024-09-11T06:51:40.433182+0000","last_peered":"2024-09-11T06:51:40.433182+0000","last_clean":"2024-09-11T06:51:40.433182+0000","last_became_active":"2024-09-11T06:14:35.104951+0000","last_became_peered":"2024-09-11T06:14:35.104951+0000","last_unstale":"2024-09-11T06:51:40.433182+0000","last_undegraded":"2024-09-11T06:51:40.433182+0000","last_fullsized":"2024-09-11T06:51:40.433182+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:14:34.076871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:14:34.076871+0000","last_clean_scrub_stamp":"2024-09-11T06:14:34.076871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:18:43.513138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087051999999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":660,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-09-11T06:51:40.695963+0000","last_change":"2024-09-11T06:14:37.124468+0000","last_active":"2024-09-11T06:51:40.695963+0000","last_peered":"2024-09-11T06:51:40.695963+0000","last_clean":"2024-09-11T06:51:40.695963+0000","last_became_active":"2024-09-11T06:14:35.099575+0000","last_became_peered":"2024-09-11T06:14:35.099575+0000","last_unstale":"2024-09-11T06:51:40.695963+0000","last_undegraded":"2024-09-11T06:51:40.695963+0000","last_fullsized":"2024-09-11T06:51:40.695963+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:14:34.076871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:14:34.076871+0000","last_clean_scrub_stamp":"2024-09-11T06:14:34.076871+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:34:43.189610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010604620000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":787,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-09-11T06:51:40.695770+0000","last_change":"2024-09-11T06:14:32.080441+0000","last_active":"2024-09-11T06:51:40.695770+0000","last_peered":"2024-09-11T06:51:40.695770+0000","last_clean":"2024-09-11T06:51:40.695770+0000","last_became_active":"2024-09-11T06:14:32.077706+0000","last_became_peered":"2024-09-11T06:14:32.077706+0000","last_unstale":"2024-09-11T06:51:40.695770+0000","last_undegraded":"2024-09-11T06:51:40.695770+0000","last_fullsized":"2024-09-11T06:51:40.695770+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:14:31.066451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:14:31.066451+0000","last_clean_scrub_stamp":"2024-09-11T06:14:31.066451+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:50:25.120392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7456,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949673421,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1273628,"kb_used_data":2632,"kb_used_omap":159,"kb_used_meta":1270816,"kb_avail":93098212,"statfs":{"total":96636764160,"available":95332569088,"internally_reserved":0,"allocated":2695168,"data_stored":1418735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":162976,"internal_metadata":1301316448},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706127,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1684184,"kb_used_data":2692,"kb_used_omap":134,"kb_used_meta":1681337,"kb_avail":92687656,"statfs":{"total":96636764160,"available":94912159744,"internally_reserved":0,"allocated":2756608,"data_stored":2009122,"data_compressed":3728,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":137978,"internal_metadata":1721689350},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706125,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1587160,"kb_used_data":2692,"kb_used_omap":150,"kb_used_meta":1584297,"kb_avail":92784680,"statfs":{"total":96636764160,"available":95011512320,"internally_reserved":0,"allocated":2756608,"data_stored":2009122,"data_compressed":3728,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":154032,"internal_metadata":1622320720},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3728,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3728,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T06:52:32.160 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-09-11T06:52:32.464 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-11T06:52:32.464 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":327,"fsid":"c4197d0c-e10d-4a46-b3cc-95a1d7af9683","created":"2024-09-11T06:14:17.992688+0000","modified":"2024-09-11T06:51:40.423092+0000","last_up_change":"2024-09-11T06:14:31.066451+0000","last_in_change":"2024-09-11T06:14:18.910683+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":3,"require_min_compat_client":"octopus","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-11T06:14:31.066308+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-11T06:14:33.321843+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9351a1cd-aa93-4d78-a30d-8a0d7a3285f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":2281847065},{"type":"v1","addr":"172.21.15.6:6809","nonce":2281847065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":2281847065},{"type":"v1","addr":"172.21.15.6:6811","nonce":2281847065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":2281847065},{"type":"v1","addr":"172.21.15.6:6815","nonce":2281847065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":2281847065},{"type":"v1","addr":"172.21.15.6:6813","nonce":2281847065}]},"public_addr":"172.21.15.6:6809/2281847065","cluster_addr":"172.21.15.6:6811/2281847065","heartbeat_back_addr":"172.21.15.6:6815/2281847065","heartbeat_front_addr":"172.21.15.6:6813/2281847065","state":["exists","up"]},{"osd":1,"uuid":"d43f49e2-ee49-43c1-8a34-5a967034a8ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":3363216727},{"type":"v1","addr":"172.21.15.6:6801","nonce":3363216727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":3363216727},{"type":"v1","addr":"172.21.15.6:6803","nonce":3363216727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":3363216727},{"type":"v1","addr":"172.21.15.6:6807","nonce":3363216727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":3363216727},{"type":"v1","addr":"172.21.15.6:6805","nonce":3363216727}]},"public_addr":"172.21.15.6:6801/3363216727","cluster_addr":"172.21.15.6:6803/3363216727","heartbeat_back_addr":"172.21.15.6:6807/3363216727","heartbeat_front_addr":"172.21.15.6:6805/3363216727","state":["exists","up"]},{"osd":2,"uuid":"fd7eafd1-7f01-4be9-8977-8a7fad1990da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":3564543485},{"type":"v1","addr":"172.21.15.6:6817","nonce":3564543485}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":3564543485},{"type":"v1","addr":"172.21.15.6:6819","nonce":3564543485}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":3564543485},{"type":"v1","addr":"172.21.15.6:6823","nonce":3564543485}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":3564543485},{"type":"v1","addr":"172.21.15.6:6821","nonce":3564543485}]},"public_addr":"172.21.15.6:6817/3564543485","cluster_addr":"172.21.15.6:6819/3564543485","heartbeat_back_addr":"172.21.15.6:6823/3564543485","heartbeat_front_addr":"172.21.15.6:6821/3564543485","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T06:14:25.351853+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T06:14:25.565559+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T06:14:26.221801+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T06:52:33.484 INFO:tasks.ceph:Scrubbing cluster1.osd.0 2024-09-11T06:52:33.485 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-11T06:52:33.649 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-11T06:52:33.650 INFO:teuthology.orchestra.run.smithi006.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-11T06:52:33.650 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-11T06:52:33.667 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 0 2024-09-11T06:52:33.966 INFO:teuthology.orchestra.run.smithi006.stderr:instructed osd(s) 0 to deep-scrub 2024-09-11T06:52:33.986 INFO:tasks.ceph:Scrubbing cluster1.osd.1 2024-09-11T06:52:33.986 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-11T06:52:34.141 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-11T06:52:34.141 INFO:teuthology.orchestra.run.smithi006.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-11T06:52:34.141 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-11T06:52:34.157 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 1 2024-09-11T06:52:34.455 INFO:teuthology.orchestra.run.smithi006.stderr:instructed osd(s) 1 to deep-scrub 2024-09-11T06:52:34.475 INFO:tasks.ceph:Scrubbing cluster1.osd.2 2024-09-11T06:52:34.476 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-11T06:52:34.633 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-11T06:52:34.633 INFO:teuthology.orchestra.run.smithi006.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-11T06:52:34.633 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-11T06:52:34.650 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 2 2024-09-11T06:52:34.945 INFO:teuthology.orchestra.run.smithi006.stderr:instructed osd(s) 2 to deep-scrub 2024-09-11T06:52:34.965 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-11T06:52:35.274 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-11T06:52:35.275 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-09-11T06:52:35.293 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":1466,"stamp":"2024-09-11T06:52:33.794736+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4544972,"kb_used_data":8016,"kb_used_omap":444,"kb_used_meta":4536451,"kb_avail":278570548,"statfs":{"total":289910292480,"available":285256241152,"internally_reserved":0,"allocated":8208384,"data_stored":5436979,"data_compressed":7456,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":454986,"internal_metadata":4645326518},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003052"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":658,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-09-11T06:51:40.690932+0000","last_change":"2024-09-11T06:14:37.124927+0000","last_active":"2024-09-11T06:51:40.690932+0000","last_peered":"2024-09-11T06:51:40.690932+0000","last_clean":"2024-09-11T06:51:40.690932+0000","last_became_active":"2024-09-11T06:14:35.103646+0000","last_became_peered":"2024-09-11T06:14:35.103646+0000","last_unstale":"2024-09-11T06:51:40.690932+0000","last_undegraded":"2024-09-11T06:51:40.690932+0000","last_fullsized":"2024-09-11T06:51:40.690932+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:14:34.076871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:14:34.076871+0000","last_clean_scrub_stamp":"2024-09-11T06:14:34.076871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:17:56.558226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00180295,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":658,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-09-11T06:51:40.689979+0000","last_change":"2024-09-11T06:14:37.124816+0000","last_active":"2024-09-11T06:51:40.689979+0000","last_peered":"2024-09-11T06:51:40.689979+0000","last_clean":"2024-09-11T06:51:40.689979+0000","last_became_active":"2024-09-11T06:14:35.103523+0000","last_became_peered":"2024-09-11T06:14:35.103523+0000","last_unstale":"2024-09-11T06:51:40.689979+0000","last_undegraded":"2024-09-11T06:51:40.689979+0000","last_fullsized":"2024-09-11T06:51:40.689979+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:14:34.076871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:14:34.076871+0000","last_clean_scrub_stamp":"2024-09-11T06:14:34.076871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:49:06.803512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016837309999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":659,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-09-11T06:51:40.693811+0000","last_change":"2024-09-11T06:14:37.125460+0000","last_active":"2024-09-11T06:51:40.693811+0000","last_peered":"2024-09-11T06:51:40.693811+0000","last_clean":"2024-09-11T06:51:40.693811+0000","last_became_active":"2024-09-11T06:14:35.103464+0000","last_became_peered":"2024-09-11T06:14:35.103464+0000","last_unstale":"2024-09-11T06:51:40.693811+0000","last_undegraded":"2024-09-11T06:51:40.693811+0000","last_fullsized":"2024-09-11T06:51:40.693811+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:14:34.076871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:14:34.076871+0000","last_clean_scrub_stamp":"2024-09-11T06:14:34.076871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T17:30:58.380858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022336330000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":659,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-09-11T06:51:40.685353+0000","last_change":"2024-09-11T06:14:37.124096+0000","last_active":"2024-09-11T06:51:40.685353+0000","last_peered":"2024-09-11T06:51:40.685353+0000","last_clean":"2024-09-11T06:51:40.685353+0000","last_became_active":"2024-09-11T06:14:35.101427+0000","last_became_peered":"2024-09-11T06:14:35.101427+0000","last_unstale":"2024-09-11T06:51:40.685353+0000","last_undegraded":"2024-09-11T06:51:40.685353+0000","last_fullsized":"2024-09-11T06:51:40.685353+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:14:34.076871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:14:34.076871+0000","last_clean_scrub_stamp":"2024-09-11T06:14:34.076871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:37:23.166823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001085391,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":661,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-09-11T06:51:41.771597+0000","last_change":"2024-09-11T06:14:37.115786+0000","last_active":"2024-09-11T06:51:41.771597+0000","last_peered":"2024-09-11T06:51:41.771597+0000","last_clean":"2024-09-11T06:51:41.771597+0000","last_became_active":"2024-09-11T06:14:35.105259+0000","last_became_peered":"2024-09-11T06:14:35.105259+0000","last_unstale":"2024-09-11T06:51:41.771597+0000","last_undegraded":"2024-09-11T06:51:41.771597+0000","last_fullsized":"2024-09-11T06:51:41.771597+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:14:34.076871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:14:34.076871+0000","last_clean_scrub_stamp":"2024-09-11T06:14:34.076871+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:33:55.339306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092280599999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":659,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-09-11T06:51:40.440930+0000","last_change":"2024-09-11T06:14:37.343308+0000","last_active":"2024-09-11T06:51:40.440930+0000","last_peered":"2024-09-11T06:51:40.440930+0000","last_clean":"2024-09-11T06:51:40.440930+0000","last_became_active":"2024-09-11T06:14:35.104976+0000","last_became_peered":"2024-09-11T06:14:35.104976+0000","last_unstale":"2024-09-11T06:51:40.440930+0000","last_undegraded":"2024-09-11T06:51:40.440930+0000","last_fullsized":"2024-09-11T06:51:40.440930+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:14:34.076871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:14:34.076871+0000","last_clean_scrub_stamp":"2024-09-11T06:14:34.076871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:33:14.682684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001060088,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":659,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-09-11T06:51:40.433182+0000","last_change":"2024-09-11T06:14:37.343210+0000","last_active":"2024-09-11T06:51:40.433182+0000","last_peered":"2024-09-11T06:51:40.433182+0000","last_clean":"2024-09-11T06:51:40.433182+0000","last_became_active":"2024-09-11T06:14:35.104951+0000","last_became_peered":"2024-09-11T06:14:35.104951+0000","last_unstale":"2024-09-11T06:51:40.433182+0000","last_undegraded":"2024-09-11T06:51:40.433182+0000","last_fullsized":"2024-09-11T06:51:40.433182+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:14:34.076871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:14:34.076871+0000","last_clean_scrub_stamp":"2024-09-11T06:14:34.076871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:18:43.513138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087051999999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":660,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-09-11T06:51:40.695963+0000","last_change":"2024-09-11T06:14:37.124468+0000","last_active":"2024-09-11T06:51:40.695963+0000","last_peered":"2024-09-11T06:51:40.695963+0000","last_clean":"2024-09-11T06:51:40.695963+0000","last_became_active":"2024-09-11T06:14:35.099575+0000","last_became_peered":"2024-09-11T06:14:35.099575+0000","last_unstale":"2024-09-11T06:51:40.695963+0000","last_undegraded":"2024-09-11T06:51:40.695963+0000","last_fullsized":"2024-09-11T06:51:40.695963+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:14:34.076871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:14:34.076871+0000","last_clean_scrub_stamp":"2024-09-11T06:14:34.076871+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:34:43.189610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010604620000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":787,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-09-11T06:51:40.695770+0000","last_change":"2024-09-11T06:14:32.080441+0000","last_active":"2024-09-11T06:51:40.695770+0000","last_peered":"2024-09-11T06:51:40.695770+0000","last_clean":"2024-09-11T06:51:40.695770+0000","last_became_active":"2024-09-11T06:14:32.077706+0000","last_became_peered":"2024-09-11T06:14:32.077706+0000","last_unstale":"2024-09-11T06:51:40.695770+0000","last_undegraded":"2024-09-11T06:51:40.695770+0000","last_fullsized":"2024-09-11T06:51:40.695770+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:14:31.066451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:14:31.066451+0000","last_clean_scrub_stamp":"2024-09-11T06:14:31.066451+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:50:25.120392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7456,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949673422,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1273628,"kb_used_data":2632,"kb_used_omap":159,"kb_used_meta":1270816,"kb_avail":93098212,"statfs":{"total":96636764160,"available":95332569088,"internally_reserved":0,"allocated":2695168,"data_stored":1418735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":162976,"internal_metadata":1301316448},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706127,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1684184,"kb_used_data":2692,"kb_used_omap":134,"kb_used_meta":1681337,"kb_avail":92687656,"statfs":{"total":96636764160,"available":94912159744,"internally_reserved":0,"allocated":2756608,"data_stored":2009122,"data_compressed":3728,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":137978,"internal_metadata":1721689350},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706125,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1587160,"kb_used_data":2692,"kb_used_omap":150,"kb_used_meta":1584297,"kb_avail":92784680,"statfs":{"total":96636764160,"available":95011512320,"internally_reserved":0,"allocated":2756608,"data_stored":2009122,"data_compressed":3728,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":154032,"internal_metadata":1622320720},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3728,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3728,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T06:52:35.294 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-11T06:14:34.076871+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=6, tm_min=14, tm_sec=34, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=6, tm_min=52, tm_sec=32, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T06:52:35.294 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-09-11T06:14:34.076871+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=6, tm_min=14, tm_sec=34, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=6, tm_min=52, tm_sec=32, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T06:52:35.294 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-11T06:14:34.076871+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=6, tm_min=14, tm_sec=34, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=6, tm_min=52, tm_sec=32, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T06:52:35.295 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-09-11T06:14:34.076871+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=6, tm_min=14, tm_sec=34, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=6, tm_min=52, tm_sec=32, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T06:52:35.295 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-09-11T06:14:34.076871+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=6, tm_min=14, tm_sec=34, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=6, tm_min=52, tm_sec=32, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T06:52:35.295 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-09-11T06:14:34.076871+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=6, tm_min=14, tm_sec=34, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=6, tm_min=52, tm_sec=32, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T06:52:35.295 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-11T06:14:34.076871+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=6, tm_min=14, tm_sec=34, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=6, tm_min=52, tm_sec=32, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T06:52:35.295 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-09-11T06:14:34.076871+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=6, tm_min=14, tm_sec=34, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=6, tm_min=52, tm_sec=32, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T06:52:35.295 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-11T06:14:31.066451+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=6, tm_min=14, tm_sec=31, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=6, tm_min=52, tm_sec=32, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T06:52:35.295 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-11T06:52:55.296 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-11T06:52:55.611 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-11T06:52:55.611 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-09-11T06:52:55.629 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":1476,"stamp":"2024-09-11T06:52:53.799634+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":104,"ondisk_log_size":104,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4544960,"kb_used_data":8016,"kb_used_omap":444,"kb_used_meta":4536451,"kb_avail":278570560,"statfs":{"total":289910292480,"available":285256253440,"internally_reserved":0,"allocated":8208384,"data_stored":5436979,"data_compressed":7456,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":454986,"internal_metadata":4645326518},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002995"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":667,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-09-11T06:52:39.180559+0000","last_change":"2024-09-11T06:52:39.180614+0000","last_active":"2024-09-11T06:52:39.180559+0000","last_peered":"2024-09-11T06:52:39.180559+0000","last_clean":"2024-09-11T06:52:39.180559+0000","last_became_active":"2024-09-11T06:14:35.103646+0000","last_became_peered":"2024-09-11T06:14:35.103646+0000","last_unstale":"2024-09-11T06:52:39.180559+0000","last_undegraded":"2024-09-11T06:52:39.180559+0000","last_fullsized":"2024-09-11T06:52:39.180559+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:52:39.180355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:52:39.180355+0000","last_clean_scrub_stamp":"2024-09-11T06:52:39.180355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:13:19.467358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00180295,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":667,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-09-11T06:52:40.220510+0000","last_change":"2024-09-11T06:52:40.220571+0000","last_active":"2024-09-11T06:52:40.220510+0000","last_peered":"2024-09-11T06:52:40.220510+0000","last_clean":"2024-09-11T06:52:40.220510+0000","last_became_active":"2024-09-11T06:14:35.103523+0000","last_became_peered":"2024-09-11T06:14:35.103523+0000","last_unstale":"2024-09-11T06:52:40.220510+0000","last_undegraded":"2024-09-11T06:52:40.220510+0000","last_fullsized":"2024-09-11T06:52:40.220510+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:52:40.220282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:52:40.220282+0000","last_clean_scrub_stamp":"2024-09-11T06:52:40.220282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:52:52.187090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016837309999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":668,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-09-11T06:52:38.209151+0000","last_change":"2024-09-11T06:52:38.209206+0000","last_active":"2024-09-11T06:52:38.209151+0000","last_peered":"2024-09-11T06:52:38.209151+0000","last_clean":"2024-09-11T06:52:38.209151+0000","last_became_active":"2024-09-11T06:14:35.103464+0000","last_became_peered":"2024-09-11T06:14:35.103464+0000","last_unstale":"2024-09-11T06:52:38.209151+0000","last_undegraded":"2024-09-11T06:52:38.209151+0000","last_fullsized":"2024-09-11T06:52:38.209151+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:52:38.208931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:52:38.208931+0000","last_clean_scrub_stamp":"2024-09-11T06:52:38.208931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:19:56.564397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022336330000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":668,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-09-11T06:52:37.231998+0000","last_change":"2024-09-11T06:52:37.232049+0000","last_active":"2024-09-11T06:52:37.231998+0000","last_peered":"2024-09-11T06:52:37.231998+0000","last_clean":"2024-09-11T06:52:37.231998+0000","last_became_active":"2024-09-11T06:14:35.101427+0000","last_became_peered":"2024-09-11T06:14:35.101427+0000","last_unstale":"2024-09-11T06:52:37.231998+0000","last_undegraded":"2024-09-11T06:52:37.231998+0000","last_fullsized":"2024-09-11T06:52:37.231998+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:52:37.231782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:52:37.231782+0000","last_clean_scrub_stamp":"2024-09-11T06:52:37.231782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:48:22.898026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001085391,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"327'3","reported_seq":671,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-09-11T06:52:34.408302+0000","last_change":"2024-09-11T06:52:34.408379+0000","last_active":"2024-09-11T06:52:34.408302+0000","last_peered":"2024-09-11T06:52:34.408302+0000","last_clean":"2024-09-11T06:52:34.408302+0000","last_became_active":"2024-09-11T06:14:35.105259+0000","last_became_peered":"2024-09-11T06:14:35.105259+0000","last_unstale":"2024-09-11T06:52:34.408302+0000","last_undegraded":"2024-09-11T06:52:34.408302+0000","last_fullsized":"2024-09-11T06:52:34.408302+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'3","last_scrub_stamp":"2024-09-11T06:52:34.408090+0000","last_deep_scrub":"327'3","last_deep_scrub_stamp":"2024-09-11T06:52:34.408090+0000","last_clean_scrub_stamp":"2024-09-11T06:52:34.408090+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:44:19.422967+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00092280599999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":668,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-09-11T06:52:35.935025+0000","last_change":"2024-09-11T06:52:35.935076+0000","last_active":"2024-09-11T06:52:35.935025+0000","last_peered":"2024-09-11T06:52:35.935025+0000","last_clean":"2024-09-11T06:52:35.935025+0000","last_became_active":"2024-09-11T06:14:35.104976+0000","last_became_peered":"2024-09-11T06:14:35.104976+0000","last_unstale":"2024-09-11T06:52:35.935025+0000","last_undegraded":"2024-09-11T06:52:35.935025+0000","last_fullsized":"2024-09-11T06:52:35.935025+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:52:35.934826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:52:35.934826+0000","last_clean_scrub_stamp":"2024-09-11T06:52:35.934826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:52:13.404792+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001060088,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":669,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-09-11T06:52:36.949061+0000","last_change":"2024-09-11T06:52:36.949061+0000","last_active":"2024-09-11T06:52:36.949061+0000","last_peered":"2024-09-11T06:52:36.949061+0000","last_clean":"2024-09-11T06:52:36.949061+0000","last_became_active":"2024-09-11T06:14:35.104951+0000","last_became_peered":"2024-09-11T06:14:35.104951+0000","last_unstale":"2024-09-11T06:52:36.949061+0000","last_undegraded":"2024-09-11T06:52:36.949061+0000","last_fullsized":"2024-09-11T06:52:36.949061+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T06:52:36.948776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T06:52:36.948776+0000","last_clean_scrub_stamp":"2024-09-11T06:52:36.948776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:24:23.174966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087051999999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"327'2","reported_seq":670,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-09-11T06:52:36.202526+0000","last_change":"2024-09-11T06:52:36.202590+0000","last_active":"2024-09-11T06:52:36.202526+0000","last_peered":"2024-09-11T06:52:36.202526+0000","last_clean":"2024-09-11T06:52:36.202526+0000","last_became_active":"2024-09-11T06:14:35.099575+0000","last_became_peered":"2024-09-11T06:14:35.099575+0000","last_unstale":"2024-09-11T06:52:36.202526+0000","last_undegraded":"2024-09-11T06:52:36.202526+0000","last_fullsized":"2024-09-11T06:52:36.202526+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'2","last_scrub_stamp":"2024-09-11T06:52:36.202196+0000","last_deep_scrub":"327'2","last_deep_scrub_stamp":"2024-09-11T06:52:36.202196+0000","last_clean_scrub_stamp":"2024-09-11T06:52:36.202196+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:33:25.508059+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0010604620000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"327'99","reported_seq":799,"reported_epoch":327,"state":"active+clean","last_fresh":"2024-09-11T06:52:35.177912+0000","last_change":"2024-09-11T06:52:35.177912+0000","last_active":"2024-09-11T06:52:35.177912+0000","last_peered":"2024-09-11T06:52:35.177912+0000","last_clean":"2024-09-11T06:52:35.177912+0000","last_became_active":"2024-09-11T06:14:32.077706+0000","last_became_peered":"2024-09-11T06:14:32.077706+0000","last_unstale":"2024-09-11T06:52:35.177912+0000","last_undegraded":"2024-09-11T06:52:35.177912+0000","last_fullsized":"2024-09-11T06:52:35.177912+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'99","last_scrub_stamp":"2024-09-11T06:52:35.177692+0000","last_deep_scrub":"327'99","last_deep_scrub_stamp":"2024-09-11T06:52:35.177692+0000","last_clean_scrub_stamp":"2024-09-11T06:52:35.177692+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:35:10.202887+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":7456,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949673426,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1273624,"kb_used_data":2632,"kb_used_omap":159,"kb_used_meta":1270816,"kb_avail":93098216,"statfs":{"total":96636764160,"available":95332573184,"internally_reserved":0,"allocated":2695168,"data_stored":1418735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":162976,"internal_metadata":1301316448},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706131,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1684180,"kb_used_data":2692,"kb_used_omap":134,"kb_used_meta":1681337,"kb_avail":92687660,"statfs":{"total":96636764160,"available":94912163840,"internally_reserved":0,"allocated":2756608,"data_stored":2009122,"data_compressed":3728,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":137978,"internal_metadata":1721689350},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706129,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1587156,"kb_used_data":2692,"kb_used_omap":150,"kb_used_meta":1584297,"kb_avail":92784684,"statfs":{"total":96636764160,"available":95011516416,"internally_reserved":0,"allocated":2756608,"data_stored":2009122,"data_compressed":3728,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":154032,"internal_metadata":1622320720},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3728,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3728,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T06:52:55.631 DEBUG:teuthology.orchestra.run.smithi006:> sudo ceph --cluster cluster1 config set global mon_health_to_clog false 2024-09-11T06:52:55.955 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-11T06:52:55.956 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-11T06:52:55.956 DEBUG:tasks.ceph.cluster1.osd.0:waiting for process to exit 2024-09-11T06:52:55.956 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T06:52:58.387 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster1.osd.0 is failed for ~0s 2024-09-11T06:53:02.057 INFO:tasks.ceph.cluster1.osd.0:Stopped 2024-09-11T06:53:02.058 DEBUG:tasks.ceph.cluster1.osd.1:waiting for process to exit 2024-09-11T06:53:02.058 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T06:53:03.691 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster1.osd.1 is failed for ~0s 2024-09-11T06:53:03.691 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster1.osd.0 has been restored 2024-09-11T06:53:08.160 INFO:tasks.ceph.cluster1.osd.1:Stopped 2024-09-11T06:53:08.160 DEBUG:tasks.ceph.cluster1.osd.2:waiting for process to exit 2024-09-11T06:53:08.160 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T06:53:08.893 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster1.osd.2 is failed for ~0s 2024-09-11T06:53:08.893 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster1.osd.1 has been restored 2024-09-11T06:53:14.096 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster1.osd.2 is failed for ~5s 2024-09-11T06:53:14.263 INFO:tasks.ceph.cluster1.osd.2:Stopped 2024-09-11T06:53:14.263 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-11T06:53:14.263 DEBUG:tasks.ceph.cluster1.mgr.x:waiting for process to exit 2024-09-11T06:53:14.263 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T06:53:14.294 INFO:tasks.ceph.cluster1.mgr.x:Stopped 2024-09-11T06:53:14.294 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-11T06:53:14.295 DEBUG:tasks.ceph.cluster1.mon.a:waiting for process to exit 2024-09-11T06:53:14.295 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T06:53:14.353 INFO:tasks.ceph.cluster1.mon.a:Stopped 2024-09-11T06:53:14.353 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-11T06:53:14.353 DEBUG:teuthology.orchestra.run.smithi006:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/cluster1.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2024-09-11T06:53:14.434 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-0 on ubuntu@smithi006.front.sepia.ceph.com 2024-09-11T06:53:14.434 DEBUG:teuthology.orchestra.run.smithi006:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-0 2024-09-11T06:53:14.947 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-1 on ubuntu@smithi006.front.sepia.ceph.com 2024-09-11T06:53:14.947 DEBUG:teuthology.orchestra.run.smithi006:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-1 2024-09-11T06:53:15.304 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-2 on ubuntu@smithi006.front.sepia.ceph.com 2024-09-11T06:53:15.305 DEBUG:teuthology.orchestra.run.smithi006:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-2 2024-09-11T06:53:15.756 INFO:tasks.ceph:Archiving mon data... 2024-09-11T06:53:15.757 DEBUG:teuthology.misc:Transferring archived files from smithi006:/var/lib/ceph/mon/cluster1-a to /home/teuthworker/archive/teuthology-2024-09-10_21:16:03-rbd-squid-distro-default-smithi/7900605/data/cluster1.mon.a.tgz 2024-09-11T06:53:15.757 DEBUG:teuthology.orchestra.run.smithi006:> mktemp 2024-09-11T06:53:15.775 INFO:teuthology.orchestra.run.smithi006.stdout:/tmp/tmp.mk3QbkHCC8 2024-09-11T06:53:15.775 DEBUG:teuthology.orchestra.run.smithi006:> sudo tar cz -f - -C /var/lib/ceph/mon/cluster1-a -- . > /tmp/tmp.mk3QbkHCC8 2024-09-11T06:53:15.933 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 0666 /tmp/tmp.mk3QbkHCC8 2024-09-11T06:53:16.035 DEBUG:teuthology.orchestra.remote:smithi006:/tmp/tmp.mk3QbkHCC8 is 435KB 2024-09-11T06:53:16.095 DEBUG:teuthology.orchestra.run.smithi006:> rm -fr /tmp/tmp.mk3QbkHCC8 2024-09-11T06:53:16.100 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-11T06:53:16.100 DEBUG:teuthology.orchestra.run.smithi006:> sudo rm -rf -- /etc/ceph/cluster1.conf /etc/ceph/cluster1.keyring /home/ubuntu/cephtest/cluster1.data /home/ubuntu/cephtest/cluster1.monmap /home/ubuntu/cephtest/../*.pid 2024-09-11T06:53:16.143 DEBUG:teuthology.orchestra.run.smithi112:> sudo rm -rf -- /etc/ceph/cluster1.conf /etc/ceph/cluster1.keyring /home/ubuntu/cephtest/cluster1.data /home/ubuntu/cephtest/cluster1.monmap /home/ubuntu/cephtest/../*.pid 2024-09-11T06:53:16.213 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-11T06:53:16.305 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-11T06:53:16.305 INFO:tasks.ceph:Archiving crash dumps... 2024-09-11T06:53:16.307 DEBUG:teuthology.misc:Transferring archived files from smithi006:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-10_21:16:03-rbd-squid-distro-default-smithi/7900605/remote/smithi006/crash 2024-09-11T06:53:16.308 DEBUG:teuthology.orchestra.run.smithi006:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-11T06:53:16.322 DEBUG:teuthology.misc:Transferring archived files from smithi112:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-10_21:16:03-rbd-squid-distro-default-smithi/7900605/remote/smithi112/crash 2024-09-11T06:53:16.323 DEBUG:teuthology.orchestra.run.smithi112:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-11T06:53:16.337 INFO:tasks.ceph:Compressing logs... 2024-09-11T06:53:16.337 DEBUG:teuthology.orchestra.run.smithi006:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-11T06:53:16.367 DEBUG:teuthology.orchestra.run.smithi112:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-11T06:53:16.387 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28486.log 2024-09-11T06:53:16.388 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1.audit.log 2024-09-11T06:53:16.388 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/cluster1-client.admin.28486.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28486.log.gz 2024-09-11T06:53:16.388 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29119.log 2024-09-11T06:53:16.390 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/cluster1.audit.log: gzip -5 --verbose -- /var/log/ceph/cluster1-mon.a.log 2024-09-11T06:53:16.390 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.25136.log 2024-09-11T06:53:16.390 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/cluster1-client.admin.29119.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29119.log.gz 2024-09-11T06:53:16.390 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/cluster1-mon.a.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28154.log 2024-09-11T06:53:16.390 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/cluster1-client.admin.25136.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.25136.log.gz 2024-09-11T06:53:16.390 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29171.log 2024-09-11T06:53:16.390 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose --/var/log/ceph/cluster1-client.admin.28154.log: /var/log/ceph/cluster1-client.admin.29429.log 2024-09-11T06:53:16.393 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/cluster1-client.admin.29171.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29171.log.gz 2024-09-11T06:53:16.393 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28512.log 2024-09-11T06:53:16.393 INFO:teuthology.orchestra.run.smithi006.stderr: 80.2% -- replaced with /var/log/ceph/cluster1-client.admin.28154.log.gz 2024-09-11T06:53:16.393 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/cluster1-client.admin.29429.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29301.log 2024-09-11T06:53:16.393 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29429.log.gz 2024-09-11T06:53:16.393 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/cluster1-client.admin.28512.log: gzip -5 --verbose -- 0.0% /var/log/ceph/cluster2-mgr.x.log -- replaced with /var/log/ceph/cluster1-client.admin.28512.log.gz 2024-09-11T06:53:16.393 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-11T06:53:16.393 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/cluster1-client.admin.29301.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28330.log 2024-09-11T06:53:16.393 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29301.log.gz 2024-09-11T06:53:16.393 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/cluster2-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.25284.log 2024-09-11T06:53:16.394 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/cluster1-client.admin.28330.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28259.log 2024-09-11T06:53:16.394 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28330.log.gz 2024-09-11T06:53:16.394 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/cluster1-client.admin.25284.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.25284.log.gz 2024-09-11T06:53:16.394 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-mgr.x.log 2024-09-11T06:53:16.400 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/cluster1-client.admin.28259.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.25160.log 2024-09-11T06:53:16.400 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28259.log.gz 2024-09-11T06:53:16.400 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/cluster1-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28416.log 2024-09-11T06:53:16.400 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-osd.2.log 2024-09-11T06:53:16.400 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/cluster1-client.admin.25160.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.25160.log.gz 2024-09-11T06:53:16.400 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/cluster1-client.admin.28416.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28337.log 2024-09-11T06:53:16.400 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28416.log.gz 2024-09-11T06:53:16.400 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/cluster1-osd.2.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.25218.log 2024-09-11T06:53:16.400 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose --/var/log/ceph/cluster1-client.admin.28337.log: /var/log/ceph/cluster1-client.admin.28129.log 2024-09-11T06:53:16.400 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28337.log.gz 2024-09-11T06:53:16.400 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/cluster1-client.admin.25218.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28538.log 2024-09-11T06:53:16.401 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.25218.log.gz 2024-09-11T06:53:16.401 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/cluster1-client.admin.28129.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28129.log.gz 2024-09-11T06:53:16.401 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-osd.0.log 2024-09-11T06:53:16.401 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/cluster1-client.admin.28538.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29145.log 2024-09-11T06:53:16.401 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28538.log.gz 2024-09-11T06:53:16.401 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/cluster1-osd.0.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28258.log 2024-09-11T06:53:16.401 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/cluster1-client.admin.29145.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.25184.log 2024-09-11T06:53:16.401 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29145.log.gz 2024-09-11T06:53:16.401 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/cluster1-client.admin.28258.log: 0.0%gzip -- replaced with /var/log/ceph/cluster1-client.admin.28258.log.gz 2024-09-11T06:53:16.401 INFO:teuthology.orchestra.run.smithi006.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.admin.28096.log 2024-09-11T06:53:16.401 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/cluster1-client.admin.25184.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28338.log 2024-09-11T06:53:16.401 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.25184.log.gz 2024-09-11T06:53:16.402 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/cluster1-client.admin.28096.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28096.log.gz 2024-09-11T06:53:16.402 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29249.log 2024-09-11T06:53:16.402 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/cluster1-client.admin.28338.log: gzip -5 --verbose -- 0.0% /var/log/ceph/cluster1-client.admin.29197.log -- replaced with /var/log/ceph/cluster1-client.admin.28338.log.gz 2024-09-11T06:53:16.402 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-11T06:53:16.402 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/cluster1-client.admin.29249.log: gzip -5 --verbose -- 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29249.log.gz /var/log/ceph/cluster1-client.admin.29275.log 2024-09-11T06:53:16.402 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-11T06:53:16.402 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/cluster1-client.admin.29197.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29197.log.gz 2024-09-11T06:53:16.402 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29379.log 2024-09-11T06:53:16.402 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/cluster1-client.admin.29275.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28408.log 0.0% 2024-09-11T06:53:16.402 INFO:teuthology.orchestra.run.smithi006.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.29275.log.gz 2024-09-11T06:53:16.402 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/cluster1-client.admin.29379.log: 0.0%gzip -- replaced with /var/log/ceph/cluster1-client.admin.29379.log.gz 2024-09-11T06:53:16.403 INFO:teuthology.orchestra.run.smithi006.stderr: -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.21430.log 2024-09-11T06:53:16.403 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/cluster1-client.admin.28408.log: gzip 0.0% -5 -- replaced with /var/log/ceph/cluster1-client.admin.28408.log.gz --verbose 2024-09-11T06:53:16.403 INFO:teuthology.orchestra.run.smithi006.stderr: -- /var/log/ceph/cluster1-client.admin.28200.log 2024-09-11T06:53:16.407 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph.tmp-client.admin.21430.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.21430.log.gz 2024-09-11T06:53:16.408 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.25042.log 2024-09-11T06:53:16.408 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/cluster1-client.admin.28200.log: 0.0%gzip -- replaced with /var/log/ceph/cluster1-client.admin.28200.log.gz 2024-09-11T06:53:16.408 INFO:teuthology.orchestra.run.smithi006.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.admin.29405.log 2024-09-11T06:53:16.408 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/cluster1-client.admin.25042.log: gzip -5 0.0% --verbose -- replaced with /var/log/ceph/cluster1-client.admin.25042.log.gz -- 2024-09-11T06:53:16.408 INFO:teuthology.orchestra.run.smithi006.stderr: /var/log/ceph/cluster1-client.admin.28174.log 2024-09-11T06:53:16.408 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/cluster1-client.admin.29405.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29405.log.gz 2024-09-11T06:53:16.408 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29353.log 2024-09-11T06:53:16.408 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/cluster1-client.admin.28174.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28174.log.gz 2024-09-11T06:53:16.408 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1.log 2024-09-11T06:53:16.408 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/cluster1-client.admin.29353.log: 0.0%gzip -- replaced with /var/log/ceph/cluster1-client.admin.29353.log.gz 2024-09-11T06:53:16.408 INFO:teuthology.orchestra.run.smithi006.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.admin.28415.log 2024-09-11T06:53:16.408 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/cluster1.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29223.log 2024-09-11T06:53:16.408 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/cluster1-client.admin.28415.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28415.log.gz 2024-09-11T06:53:16.409 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28226.log 2024-09-11T06:53:16.409 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.27829.log 2024-09-11T06:53:16.409 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/cluster1-client.admin.29223.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29223.log.gz 2024-09-11T06:53:16.409 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/cluster1-client.admin.28226.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28226.log.gz 2024-09-11T06:53:16.409 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29327.log 2024-09-11T06:53:16.409 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/cluster1-client.admin.27829.log: gzip -5 --verbose -- 0.0% /var/log/ceph/cluster1-osd.1.log -- replaced with /var/log/ceph/cluster1-client.admin.27829.log.gz 2024-09-11T06:53:16.409 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-11T06:53:16.409 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28260.log 2024-09-11T06:53:16.409 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/cluster1-client.admin.29327.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29327.log.gz 2024-09-11T06:53:16.409 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/cluster1-osd.1.log: /var/log/ceph/cluster1-client.admin.28260.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28260.log.gz 2024-09-11T06:53:16.410 INFO:teuthology.orchestra.run.smithi006.stderr: 87.5% -- replaced with /var/log/ceph/cluster1.log.gz 2024-09-11T06:53:16.410 INFO:teuthology.orchestra.run.smithi006.stderr: 94.2% -- replaced with /var/log/ceph/cluster1.audit.log.gz 2024-09-11T06:53:16.410 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123618.log 2024-09-11T06:53:16.411 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31399.log 2024-09-11T06:53:16.411 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.123618.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88862.log 2024-09-11T06:53:16.411 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.31399.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84143.log 2024-09-11T06:53:16.411 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.88862.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120381.log 2024-09-11T06:53:16.413 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.84143.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63821.log 2024-09-11T06:53:16.413 INFO:teuthology.orchestra.run.smithi112.stderr: 90.8% -- replaced with /var/log/ceph/cluster2-client.mirror.123618.log.gz 2024-09-11T06:53:16.413 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.120381.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84387.log 2024-09-11T06:53:16.413 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.63821.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62174.log 2024-09-11T06:53:16.413 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.84387.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.80971.log 2024-09-11T06:53:16.413 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.62174.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127959.log 2024-09-11T06:53:16.413 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.admin.80971.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115318.log 2024-09-11T06:53:16.414 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.127959.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76456.log 2024-09-11T06:53:16.415 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.115318.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.127959.log.gz 2024-09-11T06:53:16.415 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44371.log 2024-09-11T06:53:16.415 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123547.log 2024-09-11T06:53:16.415 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.76456.log: /var/log/ceph/cluster2-client.mirror.44371.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85934.log 2024-09-11T06:53:16.416 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54076.log 2024-09-11T06:53:16.416 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.123547.log: /var/log/ceph/cluster2-client.mirror.85934.log: 91.2% -- replaced with /var/log/ceph/cluster1-client.mirror.115318.log.gz 2024-09-11T06:53:16.416 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110756.log 2024-09-11T06:53:16.416 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose --/var/log/ceph/cluster2-client.mirror.54076.log: /var/log/ceph/cluster2-client.mirror.120247.log 2024-09-11T06:53:16.421 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.31399.log.gz 2024-09-11T06:53:16.421 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.110756.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106637.log 2024-09-11T06:53:16.421 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.120247.log: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.120381.log.gz 2024-09-11T06:53:16.421 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75394.log 2024-09-11T06:53:16.421 INFO:teuthology.orchestra.run.smithi112.stderr: 90.8% -- replaced with /var/log/ceph/cluster2-client.mirror.123547.log.gz 2024-09-11T06:53:16.421 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.120247.log.gz 2024-09-11T06:53:16.421 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.106637.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63484.log 2024-09-11T06:53:16.422 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.76456.log.gz 2024-09-11T06:53:16.422 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.75394.log: 88.9% -- replaced with /var/log/ceph/cluster1-client.admin.80971.log.gz 2024-09-11T06:53:16.422 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107437.log 2024-09-11T06:53:16.422 INFO:teuthology.orchestra.run.smithi112.stderr: 92.0% -- replaced with /var/log/ceph/cluster1-client.mirror.110756.log.gz 2024-09-11T06:53:16.422 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.63484.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60961.log 2024-09-11T06:53:16.422 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.107437.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.127742.log 2024-09-11T06:53:16.422 INFO:teuthology.orchestra.run.smithi112.stderr: 88.0%/var/log/ceph/cluster2-client.mirror.60961.log: -- replaced with /var/log/ceph/cluster2-client.mirror.107437.log.gz 2024-09-11T06:53:16.422 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68484.log 2024-09-11T06:53:16.422 INFO:teuthology.orchestra.run.smithi112.stderr: 92.0% -- replaced with /var/log/ceph/cluster2-client.mirror.106637.log.gz 2024-09-11T06:53:16.422 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.0.127742.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64511.log 2024-09-11T06:53:16.422 INFO:teuthology.orchestra.run.smithi112.stderr: 18.9% -- replaced with /var/log/ceph/cluster1-client.mirror.0.127742.log.gz 2024-09-11T06:53:16.422 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.68484.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68612.log 2024-09-11T06:53:16.422 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.64511.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75115.log 2024-09-11T06:53:16.423 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127725.log 2024-09-11T06:53:16.423 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.68612.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.63821.log.gz 2024-09-11T06:53:16.423 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128040.log 2024-09-11T06:53:16.423 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.75115.log: /var/log/ceph/cluster2-client.mirror.127725.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36631.log 2024-09-11T06:53:16.423 INFO:teuthology.orchestra.run.smithi112.stderr: 53.1% -- replaced with /var/log/ceph/cluster2-client.mirror.127725.log.gz 2024-09-11T06:53:16.423 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.128040.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127099.log 2024-09-11T06:53:16.434 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.36631.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126131.log 2024-09-11T06:53:16.434 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.128040.log.gz -- replaced with /var/log/ceph/cluster2-client.mirror.64511.log.gz 2024-09-11T06:53:16.434 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-11T06:53:16.434 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.127099.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121782.log 2024-09-11T06:53:16.434 INFO:teuthology.orchestra.run.smithi112.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.127099.log.gz 2024-09-11T06:53:16.435 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.126131.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.131018.log 2024-09-11T06:53:16.435 INFO:teuthology.orchestra.run.smithi112.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.126131.log.gz 2024-09-11T06:53:16.435 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.121782.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109586.log 2024-09-11T06:53:16.435 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.admin.131018.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119053.log 2024-09-11T06:53:16.435 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.131018.log.gz 2024-09-11T06:53:16.435 INFO:teuthology.orchestra.run.smithi112.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.121782.log.gz 2024-09-11T06:53:16.435 INFO:teuthology.orchestra.run.smithi112.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.75115.log.gz 2024-09-11T06:53:16.435 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.109586.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78851.log 2024-09-11T06:53:16.435 INFO:teuthology.orchestra.run.smithi112.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.75394.log.gz 2024-09-11T06:53:16.435 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.119053.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52120.log 2024-09-11T06:53:16.435 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.63484.log.gz 2024-09-11T06:53:16.435 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.78851.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64100.log 2024-09-11T06:53:16.435 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52457.log 2024-09-11T06:53:16.435 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.52120.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.119053.log.gz 2024-09-11T06:53:16.436 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.64100.log: 92.0% -- replaced with /var/log/ceph/cluster2-client.mirror.109586.log.gz 2024-09-11T06:53:16.436 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106012.log 2024-09-11T06:53:16.436 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.62174.log.gz 2024-09-11T06:53:16.436 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.52457.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51430.log 2024-09-11T06:53:16.436 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.106012.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.29757.log 2024-09-11T06:53:16.436 INFO:teuthology.orchestra.run.smithi112.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.64100.log.gz 2024-09-11T06:53:16.436 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.51430.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.106012.log.gz 2024-09-11T06:53:16.436 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.119981.log 2024-09-11T06:53:16.436 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.29757.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69232.log 2024-09-11T06:53:16.436 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.119981.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127121.log 2024-09-11T06:53:16.436 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.69232.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56012.log 2024-09-11T06:53:16.436 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.127121.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78479.log 2024-09-11T06:53:16.436 INFO:teuthology.orchestra.run.smithi112.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.127121.log.gz 2024-09-11T06:53:16.437 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.56012.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.68484.log.gz 2024-09-11T06:53:16.437 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77731.log 2024-09-11T06:53:16.437 INFO:teuthology.orchestra.run.smithi112.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.68612.log.gz 2024-09-11T06:53:16.437 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107494.log 2024-09-11T06:53:16.437 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.78479.log: 91.0% -- replaced with /var/log/ceph/cluster1-client.mirror.119981.log.gz 2024-09-11T06:53:16.437 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.77731.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50215.log 2024-09-11T06:53:16.437 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.107494.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44557.log 2024-09-11T06:53:16.437 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.50215.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28766.log 2024-09-11T06:53:16.437 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.44557.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54981.log 2024-09-11T06:53:16.437 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.admin.28766.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123718.log 2024-09-11T06:53:16.438 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28766.log.gz 2024-09-11T06:53:16.438 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.54981.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63263.log 2024-09-11T06:53:16.438 INFO:teuthology.orchestra.run.smithi112.stderr: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.107494.log.gz 2024-09-11T06:53:16.438 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51430.log.gz 2024-09-11T06:53:16.438 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.123718.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53147.log 2024-09-11T06:53:16.438 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.63263.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118633.log 2024-09-11T06:53:16.438 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.53147.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.123718.log.gz 2024-09-11T06:53:16.438 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80776.log 2024-09-11T06:53:16.438 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.118633.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34201.log 2024-09-11T06:53:16.438 INFO:teuthology.orchestra.run.smithi112.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.56012.log.gz 2024-09-11T06:53:16.438 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.80776.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.90549.log 2024-09-11T06:53:16.438 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.34201.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.63263.log.gz 2024-09-11T06:53:16.438 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121344.log 2024-09-11T06:53:16.439 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.90549.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75429.log 2024-09-11T06:53:16.439 INFO:teuthology.orchestra.run.smithi112.stderr: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.118633.log.gz 2024-09-11T06:53:16.439 INFO:teuthology.orchestra.run.smithi112.stderr: 84.5% -- replaced with /var/log/ceph/cluster2-client.mirror.90549.log.gz 2024-09-11T06:53:16.439 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.121344.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114012.log 2024-09-11T06:53:16.439 INFO:teuthology.orchestra.run.smithi112.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.121344.log.gz 2024-09-11T06:53:16.439 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.75429.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63856.log 2024-09-11T06:53:16.439 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.114012.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.50215.log.gz 2024-09-11T06:53:16.439 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120813.log 2024-09-11T06:53:16.439 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.63856.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68670.log 2024-09-11T06:53:16.448 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.120813.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85228.log 2024-09-11T06:53:16.449 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.68670.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.63856.log.gz 2024-09-11T06:53:16.449 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.131287.log 2024-09-11T06:53:16.449 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.85228.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59899.log 2024-09-11T06:53:16.449 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86135.log 2024-09-11T06:53:16.449 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.admin.131287.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.131287.log.gz 2024-09-11T06:53:16.449 INFO:teuthology.orchestra.run.smithi112.stderr: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.120813.log.gz 2024-09-11T06:53:16.449 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.59899.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28791.log 2024-09-11T06:53:16.449 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76797.log 2024-09-11T06:53:16.449 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.86135.log: gzip/var/log/ceph/cluster1-client.admin.28791.log: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44654.log 2024-09-11T06:53:16.449 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28791.log.gz 2024-09-11T06:53:16.449 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.76797.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71321.log 2024-09-11T06:53:16.449 INFO:teuthology.orchestra.run.smithi112.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.69232.log.gz 2024-09-11T06:53:16.450 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.44654.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84891.log 2024-09-11T06:53:16.450 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33023.log 2024-09-11T06:53:16.450 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.71321.log: /var/log/ceph/cluster2-client.mirror.84891.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40324.log 2024-09-11T06:53:16.450 INFO:teuthology.orchestra.run.smithi112.stderr: 96.2% -- replaced with /var/log/ceph/cluster2-client.mirror.114012.log.gz 2024-09-11T06:53:16.450 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.33023.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.34201.log.gz 2024-09-11T06:53:16.450 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31306.log 2024-09-11T06:53:16.450 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.40324.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130705.log 2024-09-11T06:53:16.450 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.31306.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60178.log 2024-09-11T06:53:16.450 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.130705.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.44654.log.gz 2024-09-11T06:53:16.450 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127009.log 2024-09-11T06:53:16.450 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.60178.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87317.log 2024-09-11T06:53:16.450 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.127009.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106487.log 2024-09-11T06:53:16.451 INFO:teuthology.orchestra.run.smithi112.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.127009.log.gz 2024-09-11T06:53:16.451 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.87317.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.90968.log 2024-09-11T06:53:16.451 INFO:teuthology.orchestra.run.smithi112.stderr: 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.130705.log.gz 2024-09-11T06:53:16.451 INFO:teuthology.orchestra.run.smithi112.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.78851.log.gz 2024-09-11T06:53:16.451 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.29637.log 2024-09-11T06:53:16.451 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.106487.log: /var/log/ceph/cluster2-client.mirror.90968.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79634.log 2024-09-11T06:53:16.451 INFO:teuthology.orchestra.run.smithi112.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.106487.log.gz 2024-09-11T06:53:16.451 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.29637.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98598.log 2024-09-11T06:53:16.451 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.79634.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.59899.log.gz 2024-09-11T06:53:16.451 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44464.log 2024-09-11T06:53:16.451 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87788.log 2024-09-11T06:53:16.451 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.98598.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.60178.log.gz 2024-09-11T06:53:16.451 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.44464.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35228.log 2024-09-11T06:53:16.451 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.87788.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65689.log 2024-09-11T06:53:16.452 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.35228.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54501.log 2024-09-11T06:53:16.452 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.65689.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130973.log 2024-09-11T06:53:16.452 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.54501.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.71321.log.gz 2024-09-11T06:53:16.452 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.29586.log 2024-09-11T06:53:16.452 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.130973.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73340.log 2024-09-11T06:53:16.452 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.130973.log.gz 2024-09-11T06:53:16.452 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.29586.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66472.log 2024-09-11T06:53:16.452 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46291.log 2024-09-11T06:53:16.452 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.73340.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.79634.log.gz 2024-09-11T06:53:16.452 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.66472.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80853.log 2024-09-11T06:53:16.452 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53054.log 2024-09-11T06:53:16.452 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.46291.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126829.log 2024-09-11T06:53:16.452 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.80853.log: /var/log/ceph/cluster2-client.mirror.53054.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130230.log 2024-09-11T06:53:16.453 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.126829.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48066.log 2024-09-11T06:53:16.456 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.130230.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.126829.log.gz 2024-09-11T06:53:16.456 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87237.log 2024-09-11T06:53:16.456 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.78479.log.gz 2024-09-11T06:53:16.456 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.48066.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40603.log 2024-09-11T06:53:16.456 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.87237.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85891.log 2024-09-11T06:53:16.456 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111920.log 2024-09-11T06:53:16.456 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.40603.log: /var/log/ceph/cluster2-client.mirror.85891.log: 91.9% -- replaced with /var/log/ceph/cluster2-client.mirror.130230.log.gz 2024-09-11T06:53:16.456 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.31306.log.gz 2024-09-11T06:53:16.457 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93598.log 2024-09-11T06:53:16.457 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.111920.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130060.log 2024-09-11T06:53:16.457 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.93598.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60833.log 2024-09-11T06:53:16.457 INFO:teuthology.orchestra.run.smithi112.stderr: 84.3% -- replaced with /var/log/ceph/cluster2-client.mirror.93598.log.gz 2024-09-11T06:53:16.457 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.130060.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.85228.log.gz 2024-09-11T06:53:16.457 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78421.log 2024-09-11T06:53:16.457 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.60833.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.130060.log.gz 2024-09-11T06:53:16.457 INFO:teuthology.orchestra.run.smithi112.stderr: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.111920.log.gz 2024-09-11T06:53:16.457 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113206.log 2024-09-11T06:53:16.457 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.78421.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71379.log 2024-09-11T06:53:16.457 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.113206.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65034.log 2024-09-11T06:53:16.457 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.71379.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66844.log 2024-09-11T06:53:16.457 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85534.log 2024-09-11T06:53:16.457 INFO:teuthology.orchestra.run.smithi112.stderr: 88.0%/var/log/ceph/cluster2-client.mirror.65034.log: -- replaced with /var/log/ceph/cluster2-client.mirror.35228.log.gz 2024-09-11T06:53:16.458 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.66844.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129642.log 2024-09-11T06:53:16.458 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77638.log 2024-09-11T06:53:16.458 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.85534.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.113206.log.gz 2024-09-11T06:53:16.458 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.129642.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.29729.log 2024-09-11T06:53:16.458 INFO:teuthology.orchestra.run.smithi112.stderr: 83.3% -- replaced with /var/log/ceph/cluster2-client.mirror.129642.log.gz 2024-09-11T06:53:16.458 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.77638.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55508.log 2024-09-11T06:53:16.458 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68705.log 2024-09-11T06:53:16.458 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.29729.log: /var/log/ceph/cluster2-client.mirror.55508.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28816.log 2024-09-11T06:53:16.458 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.68705.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83174.log 2024-09-11T06:53:16.458 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.admin.28816.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.28816.log.gz 2024-09-11T06:53:16.459 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28664.log 2024-09-11T06:53:16.459 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.83174.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129248.log 2024-09-11T06:53:16.471 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.admin.28664.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.28664.log.gz 2024-09-11T06:53:16.472 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.44371.log.gz 2024-09-11T06:53:16.472 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.44464.log.gz 2024-09-11T06:53:16.472 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.129248.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.129248.log.gz 2024-09-11T06:53:16.472 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125659.log 2024-09-11T06:53:16.472 INFO:teuthology.orchestra.run.smithi112.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.44557.log.gz 2024-09-11T06:53:16.472 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129869.log 2024-09-11T06:53:16.472 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67127.log 2024-09-11T06:53:16.472 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.125659.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.52120.log.gz 2024-09-11T06:53:16.472 INFO:teuthology.orchestra.run.smithi112.stderr:gzip 86.9%/var/log/ceph/cluster1-client.mirror.129869.log: -- replaced with /var/log/ceph/cluster1-client.mirror.125659.log.gz 2024-09-11T06:53:16.472 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.67127.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.129869.log.gz 2024-09-11T06:53:16.472 INFO:teuthology.orchestra.run.smithi112.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.52457.log.gz 2024-09-11T06:53:16.472 INFO:teuthology.orchestra.run.smithi112.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.admin.28360.log 2024-09-11T06:53:16.472 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111878.log 2024-09-11T06:53:16.473 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.admin.28360.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.131391.log 2024-09-11T06:53:16.473 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.28360.log.gz 2024-09-11T06:53:16.473 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.111878.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45912.log 2024-09-11T06:53:16.473 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.admin.131391.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128534.log 2024-09-11T06:53:16.473 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.131391.log.gz 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.40324.log.gz 2024-09-11T06:53:16.473 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.45912.log: 2024-09-11T06:53:16.473 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85801.log 2024-09-11T06:53:16.473 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.128534.log: gzip 91.8% -- replaced with /var/log/ceph/cluster2-client.mirror.111878.log.gz 2024-09-11T06:53:16.473 INFO:teuthology.orchestra.run.smithi112.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.128534.log.gz 2024-09-11T06:53:16.473 INFO:teuthology.orchestra.run.smithi112.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125309.log 2024-09-11T06:53:16.473 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.85801.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.68670.log.gz 2024-09-11T06:53:16.473 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123527.log 2024-09-11T06:53:16.474 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.125309.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110736.log 2024-09-11T06:53:16.474 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.123527.log: gzip -5 88.2% --verbose -- /var/log/ceph/cluster2-client.mirror.123214.log 2024-09-11T06:53:16.474 INFO:teuthology.orchestra.run.smithi112.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.123527.log.gz 2024-09-11T06:53:16.474 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.110736.log: gzip -5 --verbose -- 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.77731.log.gz 2024-09-11T06:53:16.474 INFO:teuthology.orchestra.run.smithi112.stderr: /var/log/ceph/cluster2-client.mirror.115241.log 2024-09-11T06:53:16.474 INFO:teuthology.orchestra.run.smithi112.stderr: 90.7%/var/log/ceph/cluster2-client.mirror.123214.log: -- replaced with /var/log/ceph/cluster2-client.mirror.125309.log.gz 2024-09-11T06:53:16.474 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47694.log 2024-09-11T06:53:16.474 INFO:teuthology.orchestra.run.smithi112.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.123214.log.gz 2024-09-11T06:53:16.474 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.115241.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65503.log 2024-09-11T06:53:16.474 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.47694.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.115241.log.gz 2024-09-11T06:53:16.474 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.53147.log.gz 2024-09-11T06:53:16.474 INFO:teuthology.orchestra.run.smithi112.stderr: 92.0% -- replaced with /var/log/ceph/cluster2-client.mirror.110736.log.gz 2024-09-11T06:53:16.475 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127639.log 2024-09-11T06:53:16.475 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.65503.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81980.log 2024-09-11T06:53:16.475 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.127639.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58345.log 2024-09-11T06:53:16.475 INFO:teuthology.orchestra.run.smithi112.stderr: 87.0%/var/log/ceph/cluster2-client.mirror.81980.log: -- replaced with /var/log/ceph/cluster1-client.mirror.127639.log.gz 2024-09-11T06:53:16.475 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128429.log 2024-09-11T06:53:16.475 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.58345.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128932.log 2024-09-11T06:53:16.475 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128586.log 2024-09-11T06:53:16.475 INFO:teuthology.orchestra.run.smithi112.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.73340.log.gz 2024-09-11T06:53:16.475 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.128429.log: 83.4% -- replaced with /var/log/ceph/cluster2-client.mirror.128429.log.gz 2024-09-11T06:53:16.475 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.128932.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84577.log 2024-09-11T06:53:16.475 INFO:teuthology.orchestra.run.smithi112.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.128932.log.gz 2024-09-11T06:53:16.476 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.128586.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.80776.log.gz 2024-09-11T06:53:16.476 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82242.log 2024-09-11T06:53:16.476 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.84577.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50682.log 2024-09-11T06:53:16.476 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.82242.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38720.log 2024-09-11T06:53:16.476 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.50682.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128561.log 2024-09-11T06:53:16.476 INFO:teuthology.orchestra.run.smithi112.stderr: 84.7% -- replaced with /var/log/ceph/cluster2-client.mirror.87237.log.gz 2024-09-11T06:53:16.476 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.38720.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120771.log 2024-09-11T06:53:16.476 INFO:teuthology.orchestra.run.smithi112.stderr: 91.5% -- replaced with /var/log/ceph/cluster2-client.mirror.128586.log.gz 2024-09-11T06:53:16.476 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.128561.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114754.log 2024-09-11T06:53:16.476 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.78421.log.gz 2024-09-11T06:53:16.476 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.120771.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124175.log 2024-09-11T06:53:16.476 INFO:teuthology.orchestra.run.smithi112.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.128561.log.gz 2024-09-11T06:53:16.476 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.114754.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34236.log 2024-09-11T06:53:16.477 INFO:teuthology.orchestra.run.smithi112.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.120771.log.gz 2024-09-11T06:53:16.477 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.124175.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84705.log 2024-09-11T06:53:16.477 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.34236.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.124175.log.gz 2024-09-11T06:53:16.477 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5%gzip -- replaced with /var/log/ceph/cluster2-client.mirror.45912.log.gz -5 2024-09-11T06:53:16.477 INFO:teuthology.orchestra.run.smithi112.stderr: --verbose -- /var/log/ceph/cluster1-client.mirror.127054.log 2024-09-11T06:53:16.477 INFO:teuthology.orchestra.run.smithi112.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.66844.log.gz 2024-09-11T06:53:16.477 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.84705.log: 92.0% -- replaced with /var/log/ceph/cluster1-client.mirror.114754.log.gz 2024-09-11T06:53:16.482 INFO:teuthology.orchestra.run.smithi112.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.71379.log.gz 2024-09-11T06:53:16.482 INFO:teuthology.orchestra.run.smithi112.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.81980.log.gz 2024-09-11T06:53:16.482 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.83174.log.gz 2024-09-11T06:53:16.482 INFO:teuthology.orchestra.run.smithi112.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.68705.log.gz 2024-09-11T06:53:16.482 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.127054.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65724.log 2024-09-11T06:53:16.482 INFO:teuthology.orchestra.run.smithi112.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.127054.log.gz 2024-09-11T06:53:16.482 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89248.log 2024-09-11T06:53:16.482 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86295.log 2024-09-11T06:53:16.483 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.65724.log: /var/log/ceph/cluster2-client.mirror.89248.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96848.log 2024-09-11T06:53:16.483 INFO:teuthology.orchestra.run.smithi112.stderr: 84.5% -- replaced with /var/log/ceph/cluster2-client.mirror.89248.log.gz 2024-09-11T06:53:16.483 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.86295.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42538.log 2024-09-11T06:53:16.483 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112803.log 2024-09-11T06:53:16.483 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.96848.log: /var/log/ceph/cluster2-client.mirror.42538.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50403.log 2024-09-11T06:53:16.483 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.65689.log.gz 2024-09-11T06:53:16.483 INFO:teuthology.orchestra.run.smithi112.stderr: 84.5% -- replaced with /var/log/ceph/cluster2-client.mirror.96848.log.gz 2024-09-11T06:53:16.483 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.112803.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.29876.log 2024-09-11T06:53:16.483 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.50403.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44313.log 2024-09-11T06:53:16.483 INFO:teuthology.orchestra.run.smithi112.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.112803.log.gz 2024-09-11T06:53:16.483 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.29876.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130087.log 2024-09-11T06:53:16.484 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.44313.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119518.log 2024-09-11T06:53:16.484 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.130087.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128697.log 2024-09-11T06:53:16.484 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.25418.log 2024-09-11T06:53:16.484 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.119518.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.130087.log.gz 2024-09-11T06:53:16.484 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.76797.log.gz 2024-09-11T06:53:16.484 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.128697.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.119518.log.gz 2024-09-11T06:53:16.484 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.128697.log.gz 2024-09-11T06:53:16.484 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.77638.log.gz 2024-09-11T06:53:16.484 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125196.log 2024-09-11T06:53:16.484 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.admin.25418.log: gzip 0.0% -5 -- replaced with /var/log/ceph/cluster2-client.admin.25418.log.gz --verbose 2024-09-11T06:53:16.484 INFO:teuthology.orchestra.run.smithi112.stderr: -- /var/log/ceph/cluster2-client.mirror.113500.log 2024-09-11T06:53:16.484 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.125196.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54183.log 2024-09-11T06:53:16.484 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.113500.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78979.log 2024-09-11T06:53:16.485 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.53054.log.gz 2024-09-11T06:53:16.485 INFO:teuthology.orchestra.run.smithi112.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.125196.log.gz 2024-09-11T06:53:16.485 INFO:teuthology.orchestra.run.smithi112.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.113500.log.gz 2024-09-11T06:53:16.485 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.54183.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85401.log 2024-09-11T06:53:16.485 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.78979.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86612.log 2024-09-11T06:53:16.489 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.85401.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69139.log 2024-09-11T06:53:16.489 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.86612.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75863.log 2024-09-11T06:53:16.489 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77297.log 2024-09-11T06:53:16.489 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.69139.log: /var/log/ceph/cluster2-client.mirror.75863.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83643.log 2024-09-11T06:53:16.489 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68391.log 2024-09-11T06:53:16.489 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.77297.log: /var/log/ceph/cluster2-client.mirror.83643.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50027.log 2024-09-11T06:53:16.490 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50810.log 2024-09-11T06:53:16.490 INFO:teuthology.orchestra.run.smithi112.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.60833.log.gz 2024-09-11T06:53:16.490 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.50027.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.60961.log.gz 2024-09-11T06:53:16.490 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.68391.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28866.log 2024-09-11T06:53:16.490 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.50810.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69697.log 2024-09-11T06:53:16.490 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.admin.28866.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47694.log.gz 2024-09-11T06:53:16.490 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.28866.log.gz 2024-09-11T06:53:16.490 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.69697.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80304.log 2024-09-11T06:53:16.490 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79037.log 2024-09-11T06:53:16.491 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.80304.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38406.log 2024-09-11T06:53:16.491 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.79037.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123430.log 2024-09-11T06:53:16.491 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.46291.log.gz 2024-09-11T06:53:16.491 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127795.log 2024-09-11T06:53:16.491 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.65503.log.gz 2024-09-11T06:53:16.491 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.38406.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40940.log 2024-09-11T06:53:16.491 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.123430.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47973.log 2024-09-11T06:53:16.491 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.127795.log: 88.5%/var/log/ceph/cluster2-client.mirror.40940.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86044.log 2024-09-11T06:53:16.491 INFO:teuthology.orchestra.run.smithi112.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.123430.log.gz 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.127795.log.gz 2024-09-11T06:53:16.491 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-11T06:53:16.491 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.47973.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55729.log 2024-09-11T06:53:16.491 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116082.log 2024-09-11T06:53:16.491 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.86044.log: /var/log/ceph/cluster2-client.mirror.55729.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124397.log 2024-09-11T06:53:16.495 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68298.log 2024-09-11T06:53:16.495 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.116082.log: /var/log/ceph/cluster2-client.mirror.124397.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67389.log 2024-09-11T06:53:16.495 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.68298.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.29552.log 2024-09-11T06:53:16.495 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.67389.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.81002.log 2024-09-11T06:53:16.495 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.29552.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128676.log 2024-09-11T06:53:16.495 INFO:teuthology.orchestra.run.smithi112.stderr: 91.6% -- replaced with /var/log/ceph/cluster1-client.mirror.116082.log.gz 2024-09-11T06:53:16.495 INFO:teuthology.orchestra.run.smithi112.stderr: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.124397.log.gz 2024-09-11T06:53:16.495 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.admin.81002.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130611.log 2024-09-11T06:53:16.495 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63728.log 2024-09-11T06:53:16.495 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.128676.log: /var/log/ceph/cluster2-client.mirror.130611.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111586.log 2024-09-11T06:53:16.496 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.128676.log.gz 2024-09-11T06:53:16.496 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72875.log 2024-09-11T06:53:16.496 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.63728.log: /var/log/ceph/cluster2-client.mirror.111586.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127031.log 2024-09-11T06:53:16.496 INFO:teuthology.orchestra.run.smithi112.stderr: 88.2%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88021.log 2024-09-11T06:53:16.496 INFO:teuthology.orchestra.run.smithi112.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.111586.log.gz/var/log/ceph/cluster2-client.mirror.72875.log: 2024-09-11T06:53:16.496 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.127031.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33864.log 2024-09-11T06:53:16.496 INFO:teuthology.orchestra.run.smithi112.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.127031.log.gz 2024-09-11T06:53:16.497 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.88021.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.54981.log.gz 2024-09-11T06:53:16.497 INFO:teuthology.orchestra.run.smithi112.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.65724.log.gz 2024-09-11T06:53:16.498 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123689.log 2024-09-11T06:53:16.499 INFO:teuthology.orchestra.run.smithi112.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.84143.log.gz 2024-09-11T06:53:16.499 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.33864.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124863.log 2024-09-11T06:53:16.499 INFO:teuthology.orchestra.run.smithi112.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.84387.log.gz 2024-09-11T06:53:16.499 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.123689.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95758.log 2024-09-11T06:53:16.500 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.124863.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.84577.log.gz 2024-09-11T06:53:16.500 INFO:teuthology.orchestra.run.smithi112.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.84705.log.gz 2024-09-11T06:53:16.500 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85421.log 2024-09-11T06:53:16.500 INFO:teuthology.orchestra.run.smithi112.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.84891.log.gz 2024-09-11T06:53:16.500 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.95758.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129038.log 2024-09-11T06:53:16.501 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.85421.log: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.123689.log.gz 2024-09-11T06:53:16.501 INFO:teuthology.orchestra.run.smithi112.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.77297.log.gz 2024-09-11T06:53:16.501 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84201.log 2024-09-11T06:53:16.501 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.129038.log: 83.5% -- replaced with /var/log/ceph/cluster2-client.mirror.129038.log.gz 2024-09-11T06:53:16.503 INFO:teuthology.orchestra.run.smithi112.stderr: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.124863.log.gz 2024-09-11T06:53:16.503 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129105.log 2024-09-11T06:53:16.503 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.68298.log.gz 2024-09-11T06:53:16.503 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54283.log 2024-09-11T06:53:16.503 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.84201.log: /var/log/ceph/cluster2-client.mirror.129105.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87159.log 2024-09-11T06:53:16.504 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.69697.log.gz 2024-09-11T06:53:16.504 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47636.log 2024-09-11T06:53:16.504 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.68391.log.gz 2024-09-11T06:53:16.505 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.87159.log: /var/log/ceph/cluster2-client.mirror.54283.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49969.log 2024-09-11T06:53:16.505 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.47636.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.63728.log.gz 2024-09-11T06:53:16.505 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78289.log 2024-09-11T06:53:16.506 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.49969.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.65034.log.gz 2024-09-11T06:53:16.507 INFO:teuthology.orchestra.run.smithi112.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.66472.log.gz 2024-09-11T06:53:16.507 INFO:teuthology.orchestra.run.smithi112.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.69139.log.gz 2024-09-11T06:53:16.509 INFO:teuthology.orchestra.run.smithi112.stderr: 88.7% -- replaced with /var/log/ceph/cluster1-client.admin.81002.log.gz 2024-09-11T06:53:16.510 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45983.log 2024-09-11T06:53:16.511 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.78289.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.84201.log.gz 2024-09-11T06:53:16.512 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74402.log 2024-09-11T06:53:16.513 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.45983.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36387.log 2024-09-11T06:53:16.513 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.74402.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37379.log 2024-09-11T06:53:16.514 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.36387.log: /var/log/ceph/cluster2-client.mirror.37379.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125289.log 2024-09-11T06:53:16.515 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84050.log 2024-09-11T06:53:16.515 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.125289.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127571.log 2024-09-11T06:53:16.516 INFO:teuthology.orchestra.run.smithi112.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.125289.log.gz 2024-09-11T06:53:16.516 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83081.log 2024-09-11T06:53:16.516 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.84050.log: /var/log/ceph/cluster1-client.mirror.127571.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.127571.log.gz 2024-09-11T06:53:16.517 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61709.log 2024-09-11T06:53:16.517 INFO:teuthology.orchestra.run.smithi112.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.36631.log.gz 2024-09-11T06:53:16.518 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.83081.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129489.log 2024-09-11T06:53:16.518 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.61709.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49562.log 2024-09-11T06:53:16.518 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125951.log 2024-09-11T06:53:16.519 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.129489.log: /var/log/ceph/cluster2-client.mirror.49562.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108201.log 2024-09-11T06:53:16.519 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.125951.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.125951.log.gz 2024-09-11T06:53:16.520 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67457.log 2024-09-11T06:53:16.521 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.108201.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.108201.log.gz 2024-09-11T06:53:16.522 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69790.log 2024-09-11T06:53:16.522 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.67457.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48756.log 2024-09-11T06:53:16.523 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.69790.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129269.log 2024-09-11T06:53:16.523 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.42538.log.gz 2024-09-11T06:53:16.524 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.48756.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.36387.log.gz 2024-09-11T06:53:16.526 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89103.log 2024-09-11T06:53:16.527 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.129269.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.129269.log.gz 2024-09-11T06:53:16.527 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45119.log 2024-09-11T06:53:16.538 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.89103.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35418.log 2024-09-11T06:53:16.538 INFO:teuthology.orchestra.run.smithi112.stderr: 88.4% -- replaced with /var/log/ceph/cluster1-client.mirror.67127.log.gz 2024-09-11T06:53:16.539 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.45119.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.55508.log.gz 2024-09-11T06:53:16.539 INFO:teuthology.orchestra.run.smithi112.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.55729.log.gz 2024-09-11T06:53:16.541 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.58345.log.gz 2024-09-11T06:53:16.549 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.75429.log.gz 2024-09-11T06:53:16.549 INFO:teuthology.orchestra.run.smithi112.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.75863.log.gz 2024-09-11T06:53:16.551 INFO:teuthology.orchestra.run.smithi112.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.74402.log.gz 2024-09-11T06:53:16.554 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105658.log 2024-09-11T06:53:16.561 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.35418.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.40603.log.gz 2024-09-11T06:53:16.561 INFO:teuthology.orchestra.run.smithi112.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.40940.log.gz 2024-09-11T06:53:16.562 INFO:teuthology.orchestra.run.smithi112.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.34236.log.gz 2024-09-11T06:53:16.563 INFO:teuthology.orchestra.run.smithi112.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.67457.log.gz 2024-09-11T06:53:16.564 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122151.log 2024-09-11T06:53:16.564 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.105658.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.80304.log.gz 2024-09-11T06:53:16.564 INFO:teuthology.orchestra.run.smithi112.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.105658.log.gz 2024-09-11T06:53:16.565 INFO:teuthology.orchestra.run.smithi112.stderr: 95.1% -- replaced with /var/log/ceph/cluster2-client.mirror.88862.log.gz 2024-09-11T06:53:16.566 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72348.log 2024-09-11T06:53:16.566 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49562.log.gz 2024-09-11T06:53:16.568 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.122151.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49969.log.gz 2024-09-11T06:53:16.568 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50027.log.gz 2024-09-11T06:53:16.568 INFO:teuthology.orchestra.run.smithi112.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.122151.log.gz 2024-09-11T06:53:16.568 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.50403.log.gz 2024-09-11T06:53:16.568 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62329.log 2024-09-11T06:53:16.569 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50682.log.gz 2024-09-11T06:53:16.569 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.72348.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.50810.log.gz 2024-09-11T06:53:16.569 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53612.log 2024-09-11T06:53:16.569 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.62329.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47636.log.gz 2024-09-11T06:53:16.574 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84763.log 2024-09-11T06:53:16.575 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.53612.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126784.log 2024-09-11T06:53:16.575 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.84763.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86832.log 2024-09-11T06:53:16.576 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.126784.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.126784.log.gz 2024-09-11T06:53:16.577 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38441.log 2024-09-11T06:53:16.577 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.86832.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53922.log 2024-09-11T06:53:16.578 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.38441.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.48066.log.gz 2024-09-11T06:53:16.578 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30593.log 2024-09-11T06:53:16.579 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.53922.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.47973.log.gz 2024-09-11T06:53:16.579 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125360.log 2024-09-11T06:53:16.580 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.30593.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126311.log 2024-09-11T06:53:16.580 INFO:teuthology.orchestra.run.smithi112.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.78289.log.gz 2024-09-11T06:53:16.581 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.125360.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.45983.log.gz 2024-09-11T06:53:16.582 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.125360.log.gz 2024-09-11T06:53:16.582 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65538.log 2024-09-11T06:53:16.582 INFO:teuthology.orchestra.run.smithi112.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.83081.log.gz 2024-09-11T06:53:16.582 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.126311.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.126311.log.gz 2024-09-11T06:53:16.583 INFO:teuthology.orchestra.run.smithi112.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.79037.log.gz 2024-09-11T06:53:16.583 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.78979.log.gz 2024-09-11T06:53:16.583 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.62329.log.gz 2024-09-11T06:53:16.584 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112736.log 2024-09-11T06:53:16.584 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.65538.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130438.log 2024-09-11T06:53:16.585 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.112736.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.112736.log.gz 2024-09-11T06:53:16.586 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53948.log 2024-09-11T06:53:16.586 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.130438.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.130438.log.gz 2024-09-11T06:53:16.587 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50903.log 2024-09-11T06:53:16.588 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.53948.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84108.log 2024-09-11T06:53:16.588 INFO:teuthology.orchestra.run.smithi112.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.84050.log.gz 2024-09-11T06:53:16.588 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.50903.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110711.log 2024-09-11T06:53:16.589 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.84108.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30372.log 2024-09-11T06:53:16.590 INFO:teuthology.orchestra.run.smithi112.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.72348.log.gz 2024-09-11T06:53:16.590 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.110711.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.110711.log.gz 2024-09-11T06:53:16.591 INFO:teuthology.orchestra.run.smithi112.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.85534.log.gz 2024-09-11T06:53:16.591 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71941.log 2024-09-11T06:53:16.591 INFO:teuthology.orchestra.run.smithi112.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.84763.log.gz 2024-09-11T06:53:16.592 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.30372.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126199.log 2024-09-11T06:53:16.593 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.71941.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116423.log 2024-09-11T06:53:16.594 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.126199.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.126199.log.gz 2024-09-11T06:53:16.594 INFO:teuthology.orchestra.run.smithi112.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.86832.log.gz 2024-09-11T06:53:16.598 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71600.log 2024-09-11T06:53:16.599 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.116423.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.116423.log.gz 2024-09-11T06:53:16.599 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71786.log 2024-09-11T06:53:16.600 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.71600.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76235.log 2024-09-11T06:53:16.601 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.71786.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76177.log 2024-09-11T06:53:16.602 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.76235.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68054.log 2024-09-11T06:53:16.602 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.76177.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126851.log 2024-09-11T06:53:16.603 INFO:teuthology.orchestra.run.smithi112.stderr: 91.2% -- replaced with /var/log/ceph/cluster2-client.mirror.87159.log.gz 2024-09-11T06:53:16.603 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.68054.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61395.log 2024-09-11T06:53:16.605 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.126851.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.126851.log.gz 2024-09-11T06:53:16.605 INFO:teuthology.orchestra.run.smithi112.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.37379.log.gz 2024-09-11T06:53:16.605 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.130994.log 2024-09-11T06:53:16.605 INFO:teuthology.orchestra.run.smithi112.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.38720.log.gz 2024-09-11T06:53:16.606 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.61395.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.44313.log.gz 2024-09-11T06:53:16.606 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124792.log 2024-09-11T06:53:16.606 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.admin.130994.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.130994.log.gz 95.2% -- replaced with /var/log/ceph/cluster2-client.mirror.129489.log.gz 2024-09-11T06:53:16.607 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-11T06:53:16.607 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30407.log 2024-09-11T06:53:16.607 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54303.log 2024-09-11T06:53:16.608 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.53612.log.gz 2024-09-11T06:53:16.608 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.124792.log: /var/log/ceph/cluster2-client.mirror.30407.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86769.log 2024-09-11T06:53:16.608 INFO:teuthology.orchestra.run.smithi112.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.54183.log.gz 2024-09-11T06:53:16.609 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.54303.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62139.log 2024-09-11T06:53:16.610 INFO:teuthology.orchestra.run.smithi112.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.50903.log.gz 2024-09-11T06:53:16.610 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.86769.log: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.124792.log.gz 2024-09-11T06:53:16.610 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104561.log 2024-09-11T06:53:16.610 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44220.log 2024-09-11T06:53:16.610 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.62139.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.65538.log.gz 2024-09-11T06:53:16.611 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.104561.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.71600.log.gz 2024-09-11T06:53:16.611 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.61709.log.gz 2024-09-11T06:53:16.611 INFO:teuthology.orchestra.run.smithi112.stderr: 91.8% -- replaced with /var/log/ceph/cluster1-client.mirror.104561.log.gz 2024-09-11T06:53:16.612 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74650.log 2024-09-11T06:53:16.612 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.44220.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128102.log 2024-09-11T06:53:16.613 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.74650.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.48756.log.gz 2024-09-11T06:53:16.613 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111156.log 2024-09-11T06:53:16.614 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.128102.log: 83.4% -- replaced with /var/log/ceph/cluster2-client.mirror.128102.log.gz 2024-09-11T06:53:16.614 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66530.log 2024-09-11T06:53:16.615 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104919.log 2024-09-11T06:53:16.615 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.111156.log: /var/log/ceph/cluster2-client.mirror.66530.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99486.log 2024-09-11T06:53:16.616 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.104919.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.72875.log.gz 2024-09-11T06:53:16.617 INFO:teuthology.orchestra.run.smithi112.stderr: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.104919.log.gz 2024-09-11T06:53:16.618 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81666.log 2024-09-11T06:53:16.618 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.99486.log: 1.3% -- replaced with /var/log/ceph/cluster2-client.mirror.99486.log.gz 2024-09-11T06:53:16.618 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32333.log 2024-09-11T06:53:16.619 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.81666.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34294.log 2024-09-11T06:53:16.620 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.32333.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.69790.log.gz 2024-09-11T06:53:16.621 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38855.log 2024-09-11T06:53:16.621 INFO:teuthology.orchestra.run.smithi112.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.84108.log.gz 2024-09-11T06:53:16.621 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.34294.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117280.log 2024-09-11T06:53:16.622 INFO:teuthology.orchestra.run.smithi112.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.71786.log.gz 2024-09-11T06:53:16.622 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.38855.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.71941.log.gz 2024-09-11T06:53:16.624 INFO:teuthology.orchestra.run.smithi112.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.86769.log.gz 2024-09-11T06:53:16.624 INFO:teuthology.orchestra.run.smithi112.stderr: 96.7% -- replaced with /var/log/ceph/cluster2-client.mirror.111156.log.gz 2024-09-11T06:53:16.624 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35825.log 2024-09-11T06:53:16.625 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.117280.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129931.log 2024-09-11T06:53:16.625 INFO:teuthology.orchestra.run.smithi112.stderr: 88.0%/var/log/ceph/cluster2-client.mirror.35825.log: -- replaced with /var/log/ceph/cluster1-client.mirror.117280.log.gz 2024-09-11T06:53:16.627 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.129931.log: 93.3% -- replaced with /var/log/ceph/cluster2-client.mirror.85401.log.gz 2024-09-11T06:53:16.627 INFO:teuthology.orchestra.run.smithi112.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.98598.log.gz 2024-09-11T06:53:16.627 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.67361.log 2024-09-11T06:53:16.628 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80525.log 2024-09-11T06:53:16.628 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.68054.log.gz 2024-09-11T06:53:16.628 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.44220.log.gz 2024-09-11T06:53:16.628 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.admin.67361.log: 91.7% -- replaced with /var/log/ceph/cluster2-client.mirror.129931.log.gz 2024-09-11T06:53:16.629 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93621.log 2024-09-11T06:53:16.629 INFO:teuthology.orchestra.run.smithi112.stderr: 55.3% -- replaced with /var/log/ceph/cluster2-client.mirror.29552.log.gz 2024-09-11T06:53:16.629 INFO:teuthology.orchestra.run.smithi112.stderr: 2.3% -- replaced with /var/log/ceph/cluster2-client.mirror.29586.log.gz 2024-09-11T06:53:16.630 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121222.log 2024-09-11T06:53:16.630 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.80525.log: /var/log/ceph/cluster2-client.mirror.93621.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104155.log 2024-09-11T06:53:16.630 INFO:teuthology.orchestra.run.smithi112.stderr: 56.0% -- replaced with /var/log/ceph/cluster2-client.mirror.29637.log.gz 2024-09-11T06:53:16.630 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.121222.log: 84.5% -- replaced with /var/log/ceph/cluster2-client.mirror.29876.log.gz 2024-09-11T06:53:16.630 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81511.log 2024-09-11T06:53:16.631 INFO:teuthology.orchestra.run.smithi112.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.61395.log.gz 2024-09-11T06:53:16.631 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.104155.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.66530.log.gz 2024-09-11T06:53:16.633 INFO:teuthology.orchestra.run.smithi112.stderr: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.121222.log.gz 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.35825.log.gz 2024-09-11T06:53:16.633 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-11T06:53:16.633 INFO:teuthology.orchestra.run.smithi112.stderr: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.104155.log.gz 2024-09-11T06:53:16.633 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39948.log 2024-09-11T06:53:16.633 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.81511.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130848.log 2024-09-11T06:53:16.635 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83922.log 2024-09-11T06:53:16.635 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.39948.log: /var/log/ceph/cluster1-client.mirror.130848.log: 0.0% 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.62139.log.gz 2024-09-11T06:53:16.635 INFO:teuthology.orchestra.run.smithi112.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.130848.log.gz 2024-09-11T06:53:16.635 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53834.log 2024-09-11T06:53:16.635 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128448.log 2024-09-11T06:53:16.635 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.83922.log: /var/log/ceph/cluster1-client.mirror.53834.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125167.log 2024-09-11T06:53:16.636 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.128448.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.38406.log.gz 2024-09-11T06:53:16.636 INFO:teuthology.orchestra.run.smithi112.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.38441.log.gz 2024-09-11T06:53:16.637 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45340.log 2024-09-11T06:53:16.637 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45119.log.gz 2024-09-11T06:53:16.637 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.125167.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.128448.log.gz 2024-09-11T06:53:16.637 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112306.log 2024-09-11T06:53:16.638 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.45340.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31996.log 2024-09-11T06:53:16.638 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.112306.log: 90.8% -- replaced with /var/log/ceph/cluster2-client.mirror.125167.log.gz 2024-09-11T06:53:16.639 INFO:teuthology.orchestra.run.smithi112.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.81666.log.gz 2024-09-11T06:53:16.641 INFO:teuthology.orchestra.run.smithi112.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.83643.log.gz 2024-09-11T06:53:16.642 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114442.log 2024-09-11T06:53:16.643 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.31996.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111972.log 2024-09-11T06:53:16.643 INFO:teuthology.orchestra.run.smithi112.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.32333.log.gz 2024-09-11T06:53:16.643 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.114442.log: 88.0% 88.6% -- replaced with /var/log/ceph/cluster1-client.admin.67361.log.gz 2024-09-11T06:53:16.644 INFO:teuthology.orchestra.run.smithi112.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.114442.log.gz 2024-09-11T06:53:16.645 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60492.log 2024-09-11T06:53:16.645 INFO:teuthology.orchestra.run.smithi112.stderr: 96.8% -- replaced with /var/log/ceph/cluster2-client.mirror.112306.log.gz 2024-09-11T06:53:16.645 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.111972.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49690.log 2024-09-11T06:53:16.645 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.60492.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.53834.log.gz 2024-09-11T06:53:16.645 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.111972.log.gz 2024-09-11T06:53:16.645 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39541.log 2024-09-11T06:53:16.646 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30465.log 2024-09-11T06:53:16.647 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.49690.log: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.80853.log.gz 2024-09-11T06:53:16.647 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.39541.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105248.log 2024-09-11T06:53:16.647 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75770.log 2024-09-11T06:53:16.648 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.30465.log: /var/log/ceph/cluster2-client.mirror.105248.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80618.log 2024-09-11T06:53:16.648 INFO:teuthology.orchestra.run.smithi112.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.105248.log.gz 2024-09-11T06:53:16.648 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.75770.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.74650.log.gz 2024-09-11T06:53:16.650 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108166.log 2024-09-11T06:53:16.650 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.80618.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63298.log 2024-09-11T06:53:16.651 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82461.log 2024-09-11T06:53:16.651 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.108166.log: /var/log/ceph/cluster2-client.mirror.63298.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.108166.log.gz 2024-09-11T06:53:16.654 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.45340.log.gz 2024-09-11T06:53:16.654 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75898.log 2024-09-11T06:53:16.655 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.82461.log: 95.4% -- replaced with /var/log/ceph/cluster2-client.mirror.53948.log.gz 2024-09-11T06:53:16.655 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50996.log 2024-09-11T06:53:16.656 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.75898.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124821.log 2024-09-11T06:53:16.656 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.33864.log.gz 2024-09-11T06:53:16.656 INFO:teuthology.orchestra.run.smithi112.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.39541.log.gz 2024-09-11T06:53:16.656 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.50996.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123143.log 2024-09-11T06:53:16.657 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85490.log 2024-09-11T06:53:16.657 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.124821.log: /var/log/ceph/cluster2-client.mirror.123143.log: 88.2% 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.80525.log.gz 2024-09-11T06:53:16.657 INFO:teuthology.orchestra.run.smithi112.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.123143.log.gz 2024-09-11T06:53:16.658 INFO:teuthology.orchestra.run.smithi112.stderr: 89.2%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70724.log 2024-09-11T06:53:16.658 INFO:teuthology.orchestra.run.smithi112.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.124821.log.gz 2024-09-11T06:53:16.659 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.85490.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.131313.log 2024-09-11T06:53:16.659 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.70724.log: 88.8% 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.76177.log.gz 2024-09-11T06:53:16.659 INFO:teuthology.orchestra.run.smithi112.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.76235.log.gz 2024-09-11T06:53:16.659 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128997.log 2024-09-11T06:53:16.659 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.49690.log.gz 2024-09-11T06:53:16.660 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.admin.131313.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.131313.log.gz 2024-09-11T06:53:16.660 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78196.log 2024-09-11T06:53:16.661 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32709.log 2024-09-11T06:53:16.661 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.128997.log: /var/log/ceph/cluster2-client.mirror.78196.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.38855.log.gz 2024-09-11T06:53:16.661 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.128997.log.gz 2024-09-11T06:53:16.661 INFO:teuthology.orchestra.run.smithi112.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.50996.log.gz 2024-09-11T06:53:16.661 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129709.log 2024-09-11T06:53:16.662 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79785.log 2024-09-11T06:53:16.662 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.32709.log: /var/log/ceph/cluster2-client.mirror.129709.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94109.log 2024-09-11T06:53:16.663 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.79785.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83829.log 2024-09-11T06:53:16.666 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.94109.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.60492.log.gz 2024-09-11T06:53:16.666 INFO:teuthology.orchestra.run.smithi112.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.80618.log.gz 2024-09-11T06:53:16.666 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.63298.log.gz 2024-09-11T06:53:16.667 INFO:teuthology.orchestra.run.smithi112.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.83922.log.gz 2024-09-11T06:53:16.667 INFO:teuthology.orchestra.run.smithi112.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.75770.log.gz 2024-09-11T06:53:16.670 INFO:teuthology.orchestra.run.smithi112.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.81511.log.gz 2024-09-11T06:53:16.671 INFO:teuthology.orchestra.run.smithi112.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.82461.log.gz 2024-09-11T06:53:16.673 INFO:teuthology.orchestra.run.smithi112.stderr: 92.8% -- replaced with /var/log/ceph/cluster1-client.mirror.53922.log.gz 2024-09-11T06:53:16.673 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 88.5% --verbose -- replaced with /var/log/ceph/cluster2-client.mirror.78196.log.gz -- 2024-09-11T06:53:16.673 INFO:teuthology.orchestra.run.smithi112.stderr: /var/log/ceph/cluster1-client.mirror.130873.log 2024-09-11T06:53:16.674 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.83829.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.75898.log.gz 2024-09-11T06:53:16.679 INFO:teuthology.orchestra.run.smithi112.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.39948.log.gz 2024-09-11T06:53:16.686 INFO:teuthology.orchestra.run.smithi112.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.83829.log.gz 2024-09-11T06:53:16.693 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.70724.log.gz 2024-09-11T06:53:16.694 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70949.log 2024-09-11T06:53:16.694 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.130873.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.130873.log.gz 2024-09-11T06:53:16.695 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50496.log 2024-09-11T06:53:16.700 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.70949.log: 92.9% -- replaced with /var/log/ceph/cluster2-client.mirror.54283.log.gz 2024-09-11T06:53:16.701 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86162.log 2024-09-11T06:53:16.702 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.50496.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.30372.log.gz 2024-09-11T06:53:16.702 INFO:teuthology.orchestra.run.smithi112.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.30593.log.gz 2024-09-11T06:53:16.702 INFO:teuthology.orchestra.run.smithi112.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.30407.log.gz 2024-09-11T06:53:16.702 INFO:teuthology.orchestra.run.smithi112.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.30465.log.gz 2024-09-11T06:53:16.702 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.50496.log.gz 2024-09-11T06:53:16.710 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.70949.log.gz 2024-09-11T06:53:16.711 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105347.log 2024-09-11T06:53:16.711 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.86162.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77390.log 2024-09-11T06:53:16.712 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.105347.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.79785.log.gz 2024-09-11T06:53:16.719 INFO:teuthology.orchestra.run.smithi112.stderr: 96.9% -- replaced with /var/log/ceph/cluster2-client.mirror.105347.log.gz 2024-09-11T06:53:16.724 INFO:teuthology.orchestra.run.smithi112.stderr: 92.8% -- replaced with /var/log/ceph/cluster2-client.mirror.85891.log.gz 2024-09-11T06:53:16.726 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39576.log 2024-09-11T06:53:16.727 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.77390.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.77390.log.gz 2024-09-11T06:53:16.734 INFO:teuthology.orchestra.run.smithi112.stderr: 93.2% -- replaced with /var/log/ceph/cluster1-client.mirror.85421.log.gz 2024-09-11T06:53:16.742 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- 92.8% /var/log/ceph/cluster2-client.mirror.70445.log -- replaced with /var/log/ceph/cluster1-client.mirror.54303.log.gz 2024-09-11T06:53:16.743 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-11T06:53:16.743 INFO:teuthology.orchestra.run.smithi112.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.31996.log.gz 2024-09-11T06:53:16.743 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.39576.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.39576.log.gz 2024-09-11T06:53:16.751 INFO:teuthology.orchestra.run.smithi112.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.34294.log.gz 2024-09-11T06:53:16.758 INFO:teuthology.orchestra.run.smithi112.stderr: 95.9% -- replaced with /var/log/ceph/cluster2-client.mirror.89103.log.gz 2024-09-11T06:53:16.758 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53205.log 2024-09-11T06:53:16.758 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.70445.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130649.log 2024-09-11T06:53:16.759 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.53205.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122264.log 2024-09-11T06:53:16.759 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.53205.log.gz 2024-09-11T06:53:16.760 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.130649.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.130649.log.gz 2024-09-11T06:53:16.761 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63356.log 2024-09-11T06:53:16.763 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.122264.log: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.122264.log.gz 2024-09-11T06:53:16.763 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61523.log 2024-09-11T06:53:16.764 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.63356.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129971.log 2024-09-11T06:53:16.766 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.61523.log: 95.4% -- replaced with /var/log/ceph/cluster2-client.mirror.129105.log.gz 2024-09-11T06:53:16.767 INFO:teuthology.orchestra.run.smithi112.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.63356.log.gz 2024-09-11T06:53:16.767 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126446.log 2024-09-11T06:53:16.767 INFO:teuthology.orchestra.run.smithi112.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.61523.log.gz 2024-09-11T06:53:16.768 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.129971.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.129971.log.gz 2024-09-11T06:53:16.768 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40289.log 2024-09-11T06:53:16.770 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112828.log 2024-09-11T06:53:16.770 INFO:teuthology.orchestra.run.smithi112.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.35418.log.gz 2024-09-11T06:53:16.770 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.126446.log: /var/log/ceph/cluster2-client.mirror.40289.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.126446.log.gz 2024-09-11T06:53:16.770 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35356.log 2024-09-11T06:53:16.770 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.112828.log: 95.8% -- replaced with /var/log/ceph/cluster2-client.mirror.90968.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80945.log 2024-09-11T06:53:16.770 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-11T06:53:16.772 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.35356.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31775.log 2024-09-11T06:53:16.772 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.80945.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79413.log 2024-09-11T06:53:16.772 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76553.log 2024-09-11T06:53:16.772 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.31775.log: 91.5% -- replaced with /var/log/ceph/cluster2-client.mirror.112828.log.gz 2024-09-11T06:53:16.772 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31213.log 2024-09-11T06:53:16.772 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.79413.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.70445.log.gz 2024-09-11T06:53:16.773 INFO:teuthology.orchestra.run.smithi112.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.35356.log.gz 2024-09-11T06:53:16.773 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.76553.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51744.log 2024-09-11T06:53:16.773 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.25266.log 2024-09-11T06:53:16.775 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.31213.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70666.log 2024-09-11T06:53:16.775 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.51744.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56911.log 2024-09-11T06:53:16.775 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.admin.25266.log: /var/log/ceph/cluster2-client.mirror.70666.log: 0.0%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59000.log 2024-09-11T06:53:16.775 INFO:teuthology.orchestra.run.smithi112.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.25266.log.gz 2024-09-11T06:53:16.775 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48477.log 2024-09-11T06:53:16.775 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.56911.log: /var/log/ceph/cluster2-client.mirror.59000.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110014.log 2024-09-11T06:53:16.776 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45588.log 2024-09-11T06:53:16.776 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.48477.log: /var/log/ceph/cluster2-client.mirror.110014.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59372.log 2024-09-11T06:53:16.776 INFO:teuthology.orchestra.run.smithi112.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.79413.log.gz 2024-09-11T06:53:16.777 INFO:teuthology.orchestra.run.smithi112.stderr: 94.4% -- replaced with /var/log/ceph/cluster2-client.mirror.110014.log.gz 2024-09-11T06:53:16.777 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.45588.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60868.log 2024-09-11T06:53:16.777 INFO:teuthology.orchestra.run.smithi112.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.76553.log.gz 2024-09-11T06:53:16.778 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.59372.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67199.log 2024-09-11T06:53:16.778 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129580.log 2024-09-11T06:53:16.778 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.60868.log: /var/log/ceph/cluster2-client.mirror.67199.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130923.log 2024-09-11T06:53:16.778 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125794.log 2024-09-11T06:53:16.779 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.129580.log: /var/log/ceph/cluster2-client.mirror.130923.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.130923.log.gz 2024-09-11T06:53:16.779 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.129580.log.gz 2024-09-11T06:53:16.780 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51744.log.gz 2024-09-11T06:53:16.781 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102987.log 2024-09-11T06:53:16.781 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.125794.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.125794.log.gz 2024-09-11T06:53:16.781 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69732.log 2024-09-11T06:53:16.783 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.102987.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114064.log 2024-09-11T06:53:16.783 INFO:teuthology.orchestra.run.smithi112.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.56911.log.gz 2024-09-11T06:53:16.783 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45588.log.gz 2024-09-11T06:53:16.783 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.69732.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.102987.log.gz 2024-09-11T06:53:16.783 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75243.log 2024-09-11T06:53:16.783 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.119372.log 2024-09-11T06:53:16.786 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.114064.log: /var/log/ceph/cluster2-client.mirror.75243.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92947.log 2024-09-11T06:53:16.786 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.119372.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.114064.log.gz 2024-09-11T06:53:16.786 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.60868.log.gz 2024-09-11T06:53:16.786 INFO:teuthology.orchestra.run.smithi112.stderr: 88.1%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31341.log 2024-09-11T06:53:16.786 INFO:teuthology.orchestra.run.smithi112.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.119372.log.gz 2024-09-11T06:53:16.786 INFO:teuthology.orchestra.run.smithi112.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.59000.log.gz 2024-09-11T06:53:16.786 INFO:teuthology.orchestra.run.smithi112.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.75243.log.gz 2024-09-11T06:53:16.786 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.92947.log: 84.6% -- replaced with /var/log/ceph/cluster2-client.mirror.92947.log.gz 2024-09-11T06:53:16.786 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67957.log 2024-09-11T06:53:16.786 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123760.log 2024-09-11T06:53:16.786 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.31341.log: /var/log/ceph/cluster2-client.mirror.67957.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80922.log 2024-09-11T06:53:16.787 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.123760.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38778.log 2024-09-11T06:53:16.788 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.80922.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48159.log 2024-09-11T06:53:16.788 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32554.log 2024-09-11T06:53:16.788 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.69732.log.gz 2024-09-11T06:53:16.788 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.38778.log: /var/log/ceph/cluster2-client.mirror.48159.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.119936.log 2024-09-11T06:53:16.789 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.32554.log: 90.8% -- replaced with /var/log/ceph/cluster2-client.mirror.123760.log.gz 2024-09-11T06:53:16.789 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123021.log 2024-09-11T06:53:16.789 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.119936.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.70666.log.gz 2024-09-11T06:53:16.789 INFO:teuthology.orchestra.run.smithi112.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.119936.log.gz 2024-09-11T06:53:16.790 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121853.log 2024-09-11T06:53:16.790 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.123021.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104449.log 2024-09-11T06:53:16.790 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.121853.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.121853.log.gz 2024-09-11T06:53:16.790 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127256.log 2024-09-11T06:53:16.791 INFO:teuthology.orchestra.run.smithi112.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.31213.log.gz 2024-09-11T06:53:16.791 INFO:teuthology.orchestra.run.smithi112.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.31341.log.gz 2024-09-11T06:53:16.791 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.104449.log: 88.3% 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.123021.log.gz 2024-09-11T06:53:16.792 INFO:teuthology.orchestra.run.smithi112.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.104449.log.gz 2024-09-11T06:53:16.793 INFO:teuthology.orchestra.run.smithi112.stderr: 88.1%gzip -- replaced with /var/log/ceph/cluster2-client.mirror.31775.log.gz -5 2024-09-11T06:53:16.793 INFO:teuthology.orchestra.run.smithi112.stderr: --verbose -- /var/log/ceph/cluster1-client.mirror.80098.log 2024-09-11T06:53:16.793 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46097.log 2024-09-11T06:53:16.793 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.127256.log: /var/log/ceph/cluster1-client.mirror.80098.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74460.log 2024-09-11T06:53:16.793 INFO:teuthology.orchestra.run.smithi112.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.127256.log.gz 2024-09-11T06:53:16.793 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.46097.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103685.log 2024-09-11T06:53:16.795 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.74460.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.40289.log.gz 2024-09-11T06:53:16.795 INFO:teuthology.orchestra.run.smithi112.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.38778.log.gz 2024-09-11T06:53:16.795 INFO:teuthology.orchestra.run.smithi112.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.32554.log.gz 2024-09-11T06:53:16.795 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92045.log 2024-09-11T06:53:16.796 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82130.log 2024-09-11T06:53:16.796 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.103685.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84798.log 2024-09-11T06:53:16.796 INFO:teuthology.orchestra.run.smithi112.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.103685.log.gz 2024-09-11T06:53:16.796 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.92045.log: 84.5% -- replaced with /var/log/ceph/cluster2-client.mirror.92045.log.gz 2024-09-11T06:53:16.796 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121435.log 2024-09-11T06:53:16.796 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.82130.log: /var/log/ceph/cluster2-client.mirror.84798.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.74460.log.gz 2024-09-11T06:53:16.796 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86115.log 2024-09-11T06:53:16.799 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64383.log 2024-09-11T06:53:16.799 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.121435.log: /var/log/ceph/cluster2-client.mirror.86115.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119775.log 2024-09-11T06:53:16.799 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.64383.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105391.log 2024-09-11T06:53:16.799 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.119775.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112064.log 2024-09-11T06:53:16.799 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.105391.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.105391.log.gz 2024-09-11T06:53:16.799 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79939.log 2024-09-11T06:53:16.799 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.112064.log: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.121435.log.gz 2024-09-11T06:53:16.799 INFO:teuthology.orchestra.run.smithi112.stderr: 91.9% -- replaced with /var/log/ceph/cluster2-client.mirror.119775.log.gz 2024-09-11T06:53:16.799 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130408.log 2024-09-11T06:53:16.800 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.79939.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65259.log 2024-09-11T06:53:16.800 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.130408.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35170.log 2024-09-11T06:53:16.801 INFO:teuthology.orchestra.run.smithi112.stderr: 91.9% -- replaced with /var/log/ceph/cluster2-client.mirror.112064.log.gz 2024-09-11T06:53:16.801 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.65259.log: 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.130408.log.gz 2024-09-11T06:53:16.801 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46071.log 2024-09-11T06:53:16.802 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.35170.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117844.log 2024-09-11T06:53:16.802 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63201.log 2024-09-11T06:53:16.802 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.46071.log: /var/log/ceph/cluster1-client.mirror.117844.log: gzip -5 --verbose -- /var/log/ceph/cluster2-osd.1.log 2024-09-11T06:53:16.803 INFO:teuthology.orchestra.run.smithi112.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.117844.log.gz 2024-09-11T06:53:16.803 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.63201.log: 96.0% -- replaced with /var/log/ceph/cluster2-client.mirror.130611.log.gz 2024-09-11T06:53:16.803 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107536.log 2024-09-11T06:53:16.803 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47415.log 2024-09-11T06:53:16.804 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48477.log.gz 2024-09-11T06:53:16.804 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63635.log 2024-09-11T06:53:16.805 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-osd.1.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.84798.log.gz 2024-09-11T06:53:16.806 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.47415.log: /var/log/ceph/cluster2-client.mirror.107536.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71883.log 2024-09-11T06:53:16.806 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.64383.log.gz 2024-09-11T06:53:16.806 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.63635.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31585.log 2024-09-11T06:53:16.806 INFO:teuthology.orchestra.run.smithi112.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.65259.log.gz 2024-09-11T06:53:16.806 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.71883.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107788.log 2024-09-11T06:53:16.806 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.31585.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51802.log 2024-09-11T06:53:16.808 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67095.log 2024-09-11T06:53:16.808 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.107788.log: /var/log/ceph/cluster2-client.mirror.51802.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.119750.log 2024-09-11T06:53:16.808 INFO:teuthology.orchestra.run.smithi112.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.107788.log.gz 2024-09-11T06:53:16.808 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.67095.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52643.log 2024-09-11T06:53:16.808 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.119750.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95735.log 2024-09-11T06:53:16.808 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.52643.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95153.log 2024-09-11T06:53:16.808 INFO:teuthology.orchestra.run.smithi112.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.119750.log.gz 2024-09-11T06:53:16.809 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.95735.log: 84.4% -- replaced with /var/log/ceph/cluster2-client.mirror.95735.log.gz 2024-09-11T06:53:16.810 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72968.log 2024-09-11T06:53:16.810 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.95153.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50717.log 2024-09-11T06:53:16.810 INFO:teuthology.orchestra.run.smithi112.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.59372.log.gz 2024-09-11T06:53:16.810 INFO:teuthology.orchestra.run.smithi112.stderr: 84.7% -- replaced with /var/log/ceph/cluster2-client.mirror.95153.log.gz 2024-09-11T06:53:16.812 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.72968.log: 96.8% -- replaced with /var/log/ceph/cluster2-client.mirror.107536.log.gz 2024-09-11T06:53:16.812 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40475.log 2024-09-11T06:53:16.813 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109529.log 2024-09-11T06:53:16.813 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.50717.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.79939.log.gz 2024-09-11T06:53:16.813 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.40475.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88431.log 2024-09-11T06:53:16.814 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.109529.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128976.log 2024-09-11T06:53:16.816 INFO:teuthology.orchestra.run.smithi112.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.109529.log.gz 2024-09-11T06:53:16.816 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.88431.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87212.log 2024-09-11T06:53:16.816 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.128976.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52023.log 2024-09-11T06:53:16.816 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.87212.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.128976.log.gz 2024-09-11T06:53:16.816 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125614.log 2024-09-11T06:53:16.816 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.52023.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86941.log 2024-09-11T06:53:16.816 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.125614.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34577.log 2024-09-11T06:53:16.816 INFO:teuthology.orchestra.run.smithi112.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.125614.log.gz 2024-09-11T06:53:16.816 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.86941.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79862.log 2024-09-11T06:53:16.817 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.34577.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77952.log 2024-09-11T06:53:16.817 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose/var/log/ceph/cluster2-client.mirror.79862.log: -- /var/log/ceph/cluster2-client.mirror.105426.log 2024-09-11T06:53:16.818 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.77952.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59713.log 2024-09-11T06:53:16.818 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48628.log 2024-09-11T06:53:16.818 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.105426.log: /var/log/ceph/cluster2-client.mirror.59713.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.105426.log.gz 2024-09-11T06:53:16.818 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52771.log 2024-09-11T06:53:16.818 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.48628.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85042.log 2024-09-11T06:53:16.819 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69887.log 2024-09-11T06:53:16.819 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.85042.log: /var/log/ceph/cluster2-client.mirror.52771.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109022.log 2024-09-11T06:53:16.819 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.63201.log.gz 2024-09-11T06:53:16.820 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.69887.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28430.log 2024-09-11T06:53:16.821 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.109022.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57190.log 2024-09-11T06:53:16.821 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.admin.28430.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.28430.log.gz 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.63635.log.gz 2024-09-11T06:53:16.821 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-11T06:53:16.821 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.57190.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108233.log 2024-09-11T06:53:16.821 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94455.log 2024-09-11T06:53:16.822 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.108233.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51151.log 2024-09-11T06:53:16.822 INFO:teuthology.orchestra.run.smithi112.stderr: 91.9% -- replaced with /var/log/ceph/cluster2-client.mirror.109022.log.gz 2024-09-11T06:53:16.822 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.94455.log: 88.0%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102755.log 2024-09-11T06:53:16.823 INFO:teuthology.orchestra.run.smithi112.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.108233.log.gz 2024-09-11T06:53:16.823 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63577.log 2024-09-11T06:53:16.823 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.51151.log: /var/log/ceph/cluster2-client.mirror.102755.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.102755.log.gz 2024-09-11T06:53:16.823 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130292.log 2024-09-11T06:53:16.824 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67241.log 2024-09-11T06:53:16.824 INFO:teuthology.orchestra.run.smithi112.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.71883.log.gz 2024-09-11T06:53:16.824 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.63577.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.72968.log.gz 2024-09-11T06:53:16.824 INFO:teuthology.orchestra.run.smithi112.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.87212.log.gz 2024-09-11T06:53:16.824 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.130292.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.130292.log.gz 2024-09-11T06:53:16.825 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46051.log 2024-09-11T06:53:16.825 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.67241.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35263.log 2024-09-11T06:53:16.826 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41351.log 2024-09-11T06:53:16.826 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.46051.log: /var/log/ceph/cluster2-client.mirror.35263.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92660.log 2024-09-11T06:53:16.826 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72255.log 2024-09-11T06:53:16.827 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.41351.log: /var/log/ceph/cluster2-client.mirror.92660.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38890.log 2024-09-11T06:53:16.827 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.72255.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127324.log 2024-09-11T06:53:16.828 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.38890.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56291.log 2024-09-11T06:53:16.828 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.127324.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.127324.log.gz 2024-09-11T06:53:16.828 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48159.log.gz 2024-09-11T06:53:16.829 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86495.log 2024-09-11T06:53:16.829 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.56291.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75677.log 2024-09-11T06:53:16.830 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.86495.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70538.log 2024-09-11T06:53:16.830 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.75677.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73154.log 2024-09-11T06:53:16.831 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.70538.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55136.log 2024-09-11T06:53:16.832 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.73154.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43472.log 2024-09-11T06:53:16.832 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.55136.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123476.log 2024-09-11T06:53:16.833 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.43472.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58403.log 2024-09-11T06:53:16.835 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.123476.log: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.123476.log.gz 2024-09-11T06:53:16.836 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123962.log 2024-09-11T06:53:16.836 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.58403.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61240.log 2024-09-11T06:53:16.836 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128388.log 2024-09-11T06:53:16.837 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.123962.log: /var/log/ceph/cluster2-client.mirror.61240.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.123962.log.gz 2024-09-11T06:53:16.838 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123505.log 2024-09-11T06:53:16.839 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.128388.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.52643.log.gz 2024-09-11T06:53:16.839 INFO:teuthology.orchestra.run.smithi112.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.128388.log.gz 2024-09-11T06:53:16.839 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43379.log 2024-09-11T06:53:16.841 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.123505.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.123505.log.gz 2024-09-11T06:53:16.841 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.131339.log 2024-09-11T06:53:16.842 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71100.log 2024-09-11T06:53:16.842 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.43379.log: /var/log/ceph/cluster2-client.admin.131339.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.131339.log.gz 2024-09-11T06:53:16.842 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67152.log 2024-09-11T06:53:16.842 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.71100.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56946.log 2024-09-11T06:53:16.844 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.67152.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.35170.log.gz 2024-09-11T06:53:16.844 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.50717.log.gz 2024-09-11T06:53:16.845 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55322.log 2024-09-11T06:53:16.845 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.56946.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128955.log 2024-09-11T06:53:16.846 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.55322.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28205.log 2024-09-11T06:53:16.848 INFO:teuthology.orchestra.run.smithi112.stderr: 92.8% -- replaced with /var/log/ceph/cluster2-client.mirror.85934.log.gz 2024-09-11T06:53:16.849 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.128955.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.128955.log.gz 2024-09-11T06:53:16.849 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59337.log 2024-09-11T06:53:16.849 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.admin.28205.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.28205.log.gz 2024-09-11T06:53:16.849 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-mon.a.log 2024-09-11T06:53:16.850 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.59337.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130145.log 2024-09-11T06:53:16.850 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-mon.a.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37131.log 2024-09-11T06:53:16.850 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48721.log 2024-09-11T06:53:16.851 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.130145.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.130145.log.gz 2024-09-11T06:53:16.851 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.37131.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.131519.log 2024-09-11T06:53:16.852 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.48721.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43941.log 2024-09-11T06:53:16.852 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.admin.131519.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.131519.log.gz 2024-09-11T06:53:16.852 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58717.log 2024-09-11T06:53:16.853 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.43941.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87709.log 2024-09-11T06:53:16.854 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.58717.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.131469.log 2024-09-11T06:53:16.854 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.87709.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34612.log 2024-09-11T06:53:16.855 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.admin.131469.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.131469.log.gz 2024-09-11T06:53:16.855 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68519.log 2024-09-11T06:53:16.856 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.34612.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95176.log 2024-09-11T06:53:16.856 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.68519.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72162.log 2024-09-11T06:53:16.858 INFO:teuthology.orchestra.run.smithi006.stderr: 91.3% -- replaced with /var/log/ceph/cluster1-mgr.x.log.gz 2024-09-11T06:53:16.859 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.95176.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83116.log 2024-09-11T06:53:16.860 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.72162.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108619.log 2024-09-11T06:53:16.861 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.83116.log: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.94109.log.gz 2024-09-11T06:53:16.861 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46702.log 2024-09-11T06:53:16.861 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.108619.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122810.log 2024-09-11T06:53:16.862 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.46702.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.108619.log.gz 2024-09-11T06:53:16.862 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103433.log 2024-09-11T06:53:16.863 INFO:teuthology.orchestra.run.smithi112.stderr: 95.8%/var/log/ceph/cluster2-client.mirror.122810.log: -- replaced with /var/log/ceph/cluster2-client.mirror.95758.log.gz 2024-09-11T06:53:16.863 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126424.log 2024-09-11T06:53:16.864 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.122810.log.gz 2024-09-11T06:53:16.864 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.103433.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42067.log 2024-09-11T06:53:16.864 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.126424.log: 87.1% -- replaced with /var/log/ceph/cluster1-client.mirror.126424.log.gz 2024-09-11T06:53:16.867 INFO:teuthology.orchestra.run.smithi112.stderr: 95.9% -- replaced with /var/log/ceph/cluster2-client.mirror.103433.log.gz 2024-09-11T06:53:16.870 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.29041.log 2024-09-11T06:53:16.870 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.42067.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126536.log 2024-09-11T06:53:16.871 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.admin.29041.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129398.log 2024-09-11T06:53:16.872 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.29041.log.gz 2024-09-11T06:53:16.872 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.126536.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51837.log 2024-09-11T06:53:16.872 INFO:teuthology.orchestra.run.smithi112.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.126536.log.gz 2024-09-11T06:53:16.872 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.67330.log 2024-09-11T06:53:16.872 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.129398.log: /var/log/ceph/cluster2-client.mirror.51837.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31248.log 2024-09-11T06:53:16.873 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.admin.67330.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.77952.log.gz 2024-09-11T06:53:16.873 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73619.log 2024-09-11T06:53:16.873 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.31248.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75712.log 2024-09-11T06:53:16.873 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.73619.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124011.log 2024-09-11T06:53:16.874 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.75712.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125096.log 2024-09-11T06:53:16.875 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.124011.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73654.log 2024-09-11T06:53:16.875 INFO:teuthology.orchestra.run.smithi112.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.85042.log.gz 2024-09-11T06:53:16.875 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47415.log.gz 2024-09-11T06:53:16.875 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.125096.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.124011.log.gz 2024-09-11T06:53:16.876 INFO:teuthology.orchestra.run.smithi112.stderr: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.125096.log.gz 2024-09-11T06:53:16.879 INFO:teuthology.orchestra.run.smithi112.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48628.log.gz 2024-09-11T06:53:16.879 INFO:teuthology.orchestra.run.smithi112.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.67095.log.gz 2024-09-11T06:53:16.881 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.69887.log.gz 2024-09-11T06:53:16.881 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.70538.log.gz 2024-09-11T06:53:16.881 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.52771.log.gz 2024-09-11T06:53:16.881 INFO:teuthology.orchestra.run.smithi112.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.63577.log.gz 2024-09-11T06:53:16.882 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.59337.log.gz 2024-09-11T06:53:16.882 INFO:teuthology.orchestra.run.smithi112.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.72255.log.gz 2024-09-11T06:53:16.882 INFO:teuthology.orchestra.run.smithi112.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.75677.log.gz 2024-09-11T06:53:16.882 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.75712.log.gz 2024-09-11T06:53:16.887 INFO:teuthology.orchestra.run.smithi006.stderr: 91.7% -- replaced with /var/log/ceph/cluster2-mgr.x.log.gz 2024-09-11T06:53:16.889 INFO:teuthology.orchestra.run.smithi112.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.55322.log.gz 2024-09-11T06:53:16.890 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77580.log 2024-09-11T06:53:16.890 INFO:teuthology.orchestra.run.smithi112.stderr: 95.8% -- replaced with /var/log/ceph/cluster2-client.mirror.129709.log.gz 2024-09-11T06:53:16.891 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.73654.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41130.log 2024-09-11T06:53:16.891 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.77580.log: 93.0% -- replaced with /var/log/ceph/cluster2-client.mirror.67199.log.gz 2024-09-11T06:53:16.891 INFO:teuthology.orchestra.run.smithi112.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.73154.log.gz 2024-09-11T06:53:16.894 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59748.log 2024-09-11T06:53:16.894 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.41130.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56663.log 2024-09-11T06:53:16.895 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.59748.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76611.log 2024-09-11T06:53:16.895 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.56663.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88612.log 2024-09-11T06:53:16.896 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.76611.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110644.log 2024-09-11T06:53:16.896 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.88612.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52306.log 2024-09-11T06:53:16.897 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.110644.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.110644.log.gz 2024-09-11T06:53:16.899 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122193.log 2024-09-11T06:53:16.899 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.52306.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130563.log 2024-09-11T06:53:16.900 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115659.log 2024-09-11T06:53:16.901 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.122193.log: /var/log/ceph/cluster2-client.mirror.130563.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.130563.log.gz 2024-09-11T06:53:16.909 INFO:teuthology.orchestra.run.smithi112.stderr: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.122193.log.gz 2024-09-11T06:53:16.909 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.29654.log 2024-09-11T06:53:16.909 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.67957.log.gz 2024-09-11T06:53:16.909 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.115659.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87260.log 2024-09-11T06:53:16.909 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.29654.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.29518.log 2024-09-11T06:53:16.909 INFO:teuthology.orchestra.run.smithi112.stderr: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.115659.log.gz 2024-09-11T06:53:16.909 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74495.log 2024-09-11T06:53:16.909 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.admin.29518.log: /var/log/ceph/cluster2-client.mirror.87260.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49093.log 2024-09-11T06:53:16.909 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.74495.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105703.log 2024-09-11T06:53:16.910 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.49093.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106931.log 2024-09-11T06:53:16.910 INFO:teuthology.orchestra.run.smithi112.stderr: 93.2% -- replaced with /var/log/ceph/cluster1-client.mirror.80098.log.gz 2024-09-11T06:53:16.910 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.105703.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36348.log 2024-09-11T06:53:16.910 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.106931.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81887.log 2024-09-11T06:53:16.910 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.36348.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.106931.log.gz 2024-09-11T06:53:16.910 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102409.log 2024-09-11T06:53:16.910 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.81887.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86342.log 2024-09-11T06:53:16.910 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.102409.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28180.log 2024-09-11T06:53:16.910 INFO:teuthology.orchestra.run.smithi112.stderr: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.105703.log.gz 2024-09-11T06:53:16.910 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.86342.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.102409.log.gz 2024-09-11T06:53:16.910 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111208.log 2024-09-11T06:53:16.910 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.admin.28180.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130735.log 2024-09-11T06:53:16.910 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.28180.log.gz 2024-09-11T06:53:16.911 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.111208.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112417.log 2024-09-11T06:53:16.911 INFO:teuthology.orchestra.run.smithi112.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.111208.log.gz 2024-09-11T06:53:16.911 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.130735.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.130735.log.gz 2024-09-11T06:53:16.914 INFO:teuthology.orchestra.run.smithi112.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.31585.log.gz 2024-09-11T06:53:16.914 INFO:teuthology.orchestra.run.smithi112.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.40475.log.gz 2024-09-11T06:53:16.914 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44278.log 2024-09-11T06:53:16.914 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.112417.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129601.log 2024-09-11T06:53:16.914 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.44278.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39390.log 2024-09-11T06:53:16.914 INFO:teuthology.orchestra.run.smithi112.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.112417.log.gz 2024-09-11T06:53:16.915 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121753.log 2024-09-11T06:53:16.915 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.129601.log: /var/log/ceph/cluster2-client.mirror.39390.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70914.log 2024-09-11T06:53:16.915 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.121753.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121733.log 2024-09-11T06:53:16.915 INFO:teuthology.orchestra.run.smithi112.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.129601.log.gz 2024-09-11T06:53:16.915 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.70914.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40006.log 2024-09-11T06:53:16.915 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.121733.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88976.log 2024-09-11T06:53:16.915 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.40006.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.121733.log.gz 2024-09-11T06:53:16.915 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113226.log 2024-09-11T06:53:16.915 INFO:teuthology.orchestra.run.smithi112.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.41130.log.gz 2024-09-11T06:53:16.915 INFO:teuthology.orchestra.run.smithi112.stderr: 90.8% -- replaced with /var/log/ceph/cluster2-client.mirror.121753.log.gz 2024-09-11T06:53:16.915 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.88976.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.41351.log.gz 2024-09-11T06:53:16.916 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33957.log 2024-09-11T06:53:16.916 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66623.log 2024-09-11T06:53:16.917 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.113226.log: /var/log/ceph/cluster2-client.mirror.33957.log: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.113226.log.gz 2024-09-11T06:53:16.917 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.25383.log 2024-09-11T06:53:16.918 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79727.log 2024-09-11T06:53:16.918 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.66623.log: /var/log/ceph/cluster2-client.admin.25383.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.25383.log.gz 2024-09-11T06:53:16.918 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126671.log 2024-09-11T06:53:16.919 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94636.log 2024-09-11T06:53:16.921 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.79727.log: /var/log/ceph/cluster1-client.mirror.126671.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.126671.log.gz 2024-09-11T06:53:16.922 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111475.log 2024-09-11T06:53:16.922 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83864.log 2024-09-11T06:53:16.924 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.94636.log: /var/log/ceph/cluster1-client.mirror.111475.log: 84.6% -- replaced with /var/log/ceph/cluster2-client.mirror.94636.log.gz 2024-09-11T06:53:16.924 INFO:teuthology.orchestra.run.smithi112.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.111475.log.gz 2024-09-11T06:53:16.924 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42236.log 2024-09-11T06:53:16.924 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85263.log 2024-09-11T06:53:16.926 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.83864.log: /var/log/ceph/cluster2-client.mirror.42236.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47132.log 2024-09-11T06:53:16.927 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.85263.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66902.log 2024-09-11T06:53:16.927 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.47132.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54215.log 2024-09-11T06:53:16.928 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130383.log 2024-09-11T06:53:16.928 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.66902.log: /var/log/ceph/cluster1-client.mirror.54215.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127986.log 2024-09-11T06:53:16.929 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63449.log 2024-09-11T06:53:16.930 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.130383.log: /var/log/ceph/cluster1-client.mirror.127986.log: 87.4% 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.127986.log.gz 2024-09-11T06:53:16.930 INFO:teuthology.orchestra.run.smithi112.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.130383.log.gz 2024-09-11T06:53:16.930 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81139.log 2024-09-11T06:53:16.930 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.63449.log: 88.9% -- replaced with /var/log/ceph/cluster1-client.admin.67330.log.gz 2024-09-11T06:53:16.931 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.81139.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86315.log 2024-09-11T06:53:16.932 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34050.log 2024-09-11T06:53:16.933 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.86315.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.73654.log.gz 2024-09-11T06:53:16.933 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126626.log 2024-09-11T06:53:16.933 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.34050.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107043.log 2024-09-11T06:53:16.934 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.126626.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.126626.log.gz 2024-09-11T06:53:16.934 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85662.log 2024-09-11T06:53:16.935 INFO:teuthology.orchestra.run.smithi112.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.34577.log.gz 2024-09-11T06:53:16.935 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.107043.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.34612.log.gz 2024-09-11T06:53:16.935 INFO:teuthology.orchestra.run.smithi112.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.76611.log.gz 2024-09-11T06:53:16.937 INFO:teuthology.orchestra.run.smithi112.stderr: 91.9% -- replaced with /var/log/ceph/cluster1-client.mirror.107043.log.gz 2024-09-11T06:53:16.937 INFO:teuthology.orchestra.run.smithi112.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.77580.log.gz 2024-09-11T06:53:16.937 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56849.log 2024-09-11T06:53:16.938 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.85662.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47880.log 2024-09-11T06:53:16.938 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.79862.log.gz 2024-09-11T06:53:16.938 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.56849.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61895.log 2024-09-11T06:53:16.939 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.47880.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128901.log 2024-09-11T06:53:16.939 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.61895.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40196.log 2024-09-11T06:53:16.940 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.128901.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.83116.log.gz 2024-09-11T06:53:16.941 INFO:teuthology.orchestra.run.smithi112.stderr: 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.128901.log.gz 2024-09-11T06:53:16.942 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104894.log 2024-09-11T06:53:16.942 INFO:teuthology.orchestra.run.smithi112.stderr: 92.8% -- replaced with /var/log/ceph/cluster1-client.mirror.46071.log.gz 2024-09-11T06:53:16.942 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.40196.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.48721.log.gz 2024-09-11T06:53:16.943 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127189.log 2024-09-11T06:53:16.943 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.104894.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125839.log 2024-09-11T06:53:16.944 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.127189.log: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.92660.log.gz 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.104894.log.gz 2024-09-11T06:53:16.944 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-11T06:53:16.944 INFO:teuthology.orchestra.run.smithi112.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.127189.log.gz 2024-09-11T06:53:16.944 INFO:teuthology.orchestra.run.smithi112.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.70914.log.gz 2024-09-11T06:53:16.944 INFO:teuthology.orchestra.run.smithi112.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.71100.log.gz 2024-09-11T06:53:16.944 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55171.log 2024-09-11T06:53:16.945 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.125839.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51089.log 2024-09-11T06:53:16.946 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.55171.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.63449.log.gz 2024-09-11T06:53:16.946 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45433.log 2024-09-11T06:53:16.946 INFO:teuthology.orchestra.run.smithi112.stderr:gzip 87.0% -5 -- replaced with /var/log/ceph/cluster1-client.mirror.125839.log.gz --verbose 2024-09-11T06:53:16.946 INFO:teuthology.orchestra.run.smithi112.stderr: -- /var/log/ceph/cluster1-client.admin.54442.log 2024-09-11T06:53:16.946 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.51089.log: /var/log/ceph/cluster2-client.mirror.45433.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34387.log 2024-09-11T06:53:16.946 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.admin.54442.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76142.log 2024-09-11T06:53:16.946 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.34387.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68019.log 2024-09-11T06:53:16.947 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.76142.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81759.log 2024-09-11T06:53:16.948 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118190.log 2024-09-11T06:53:16.948 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.68019.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86744.log 2024-09-11T06:53:16.948 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.81759.log: /var/log/ceph/cluster2-client.mirror.118190.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74553.log 2024-09-11T06:53:16.948 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.86744.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.29769.log 2024-09-11T06:53:16.948 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.118190.log.gz 2024-09-11T06:53:16.950 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.74553.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85844.log 2024-09-11T06:53:16.950 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55787.log 2024-09-11T06:53:16.952 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.85844.log: /var/log/ceph/cluster2-client.mirror.0.29769.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129661.log 2024-09-11T06:53:16.952 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.55787.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121251.log 2024-09-11T06:53:16.953 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.129661.log: /var/log/ceph/cluster2-client.mirror.121251.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.129661.log.gz 2024-09-11T06:53:16.953 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47318.log 2024-09-11T06:53:16.953 INFO:teuthology.orchestra.run.smithi112.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.121251.log.gz 2024-09-11T06:53:16.956 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.43472.log.gz 2024-09-11T06:53:16.957 INFO:teuthology.orchestra.run.smithi112.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.49093.log.gz 2024-09-11T06:53:16.958 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72782.log 2024-09-11T06:53:16.958 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.47318.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125034.log 2024-09-11T06:53:16.959 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.72782.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125996.log 2024-09-11T06:53:16.959 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127234.log 2024-09-11T06:53:16.960 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.125034.log: /var/log/ceph/cluster1-client.mirror.125996.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103333.log 2024-09-11T06:53:16.960 INFO:teuthology.orchestra.run.smithi112.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.125996.log.gz 2024-09-11T06:53:16.960 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.127234.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.125034.log.gz 2024-09-11T06:53:16.961 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86589.log 2024-09-11T06:53:16.961 INFO:teuthology.orchestra.run.smithi112.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.127234.log.gz/var/log/ceph/cluster2-client.mirror.103333.log: 2024-09-11T06:53:16.961 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96879.log 2024-09-11T06:53:16.962 INFO:teuthology.orchestra.run.smithi112.stderr:gzip 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.103333.log.gz -5 2024-09-11T06:53:16.962 INFO:teuthology.orchestra.run.smithi112.stderr: --verbose -- /var/log/ceph/cluster1-client.admin.42312.log 2024-09-11T06:53:16.962 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.86589.log: /var/log/ceph/cluster2-client.mirror.96879.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56477.log 2024-09-11T06:53:16.962 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.admin.42312.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128876.log 2024-09-11T06:53:16.962 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104516.log 2024-09-11T06:53:16.964 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.128876.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.128876.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48384.log 2024-09-11T06:53:16.964 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-11T06:53:16.964 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.56477.log: /var/log/ceph/cluster1-client.mirror.104516.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.29862.log 2024-09-11T06:53:16.964 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.48384.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.104516.log.gz 2024-09-11T06:53:16.964 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125380.log 2024-09-11T06:53:16.965 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.29862.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62922.log 2024-09-11T06:53:16.965 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.125380.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53670.log 2024-09-11T06:53:16.965 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.62922.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86899.log 2024-09-11T06:53:16.966 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.53670.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.85662.log.gz 2024-09-11T06:53:16.966 INFO:teuthology.orchestra.run.smithi112.stderr: 95.9% -- replaced with /var/log/ceph/cluster2-client.mirror.93621.log.gz 2024-09-11T06:53:16.966 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48942.log 2024-09-11T06:53:16.967 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.86899.log: gzip 90.8% -5 --verbose -- replaced with /var/log/ceph/cluster2-client.mirror.125380.log.gz -- 2024-09-11T06:53:16.967 INFO:teuthology.orchestra.run.smithi112.stderr: /var/log/ceph/cluster2-client.mirror.74778.log 2024-09-11T06:53:16.967 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68763.log 2024-09-11T06:53:16.968 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.48942.log: /var/log/ceph/cluster2-client.mirror.74778.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58031.log 2024-09-11T06:53:16.968 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.68763.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71658.log 2024-09-11T06:53:16.968 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.58031.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62608.log 2024-09-11T06:53:16.968 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.71658.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.29091.log 2024-09-11T06:53:16.969 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.62608.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111542.log 2024-09-11T06:53:16.969 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.admin.29091.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.29091.log.gz 2024-09-11T06:53:16.969 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.51151.log.gz 2024-09-11T06:53:16.969 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60399.log 2024-09-11T06:53:16.970 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116468.log 2024-09-11T06:53:16.970 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.111542.log: /var/log/ceph/cluster2-client.mirror.60399.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70759.log 2024-09-11T06:53:16.970 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127279.log 2024-09-11T06:53:16.971 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.116468.log: /var/log/ceph/cluster2-client.mirror.70759.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72685.log 2024-09-11T06:53:16.971 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.55136.log.gz 2024-09-11T06:53:16.972 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.127279.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.57190.log.gz 2024-09-11T06:53:16.972 INFO:teuthology.orchestra.run.smithi112.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.127279.log.gz 2024-09-11T06:53:16.972 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33116.log 2024-09-11T06:53:16.973 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.72685.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125435.log 2024-09-11T06:53:16.973 INFO:teuthology.orchestra.run.smithi112.stderr: 91.2% -- replaced with /var/log/ceph/cluster1-client.mirror.116468.log.gz 2024-09-11T06:53:16.974 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.33116.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87463.log 2024-09-11T06:53:16.974 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.125435.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121393.log 2024-09-11T06:53:16.974 INFO:teuthology.orchestra.run.smithi112.stderr: 1.3% -- replaced with /var/log/ceph/cluster1-client.mirror.125435.log.gz 2024-09-11T06:53:16.974 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.87463.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.90161.log 2024-09-11T06:53:16.975 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.121393.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.121393.log.gz 2024-09-11T06:53:16.976 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38685.log 2024-09-11T06:53:16.976 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.90161.log: 84.3% -- replaced with /var/log/ceph/cluster2-client.mirror.90161.log.gz 2024-09-11T06:53:16.977 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.51802.log.gz 2024-09-11T06:53:16.978 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53763.log 2024-09-11T06:53:16.978 INFO:teuthology.orchestra.run.smithi112.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.52023.log.gz 2024-09-11T06:53:16.978 INFO:teuthology.orchestra.run.smithi112.stderr: 96.9% -- replaced with /var/log/ceph/cluster2-client.mirror.111542.log.gz 2024-09-11T06:53:16.979 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.38685.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51837.log.gz 2024-09-11T06:53:16.979 INFO:teuthology.orchestra.run.smithi112.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.52306.log.gz 2024-09-11T06:53:16.979 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121682.log 2024-09-11T06:53:16.979 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.53763.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56070.log 2024-09-11T06:53:16.980 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.121682.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124497.log 2024-09-11T06:53:16.982 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.56070.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.35263.log.gz 2024-09-11T06:53:16.982 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.124497.log: 90.8% -- replaced with /var/log/ceph/cluster2-client.mirror.121682.log.gz 2024-09-11T06:53:16.982 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47074.log 2024-09-11T06:53:16.982 INFO:teuthology.orchestra.run.smithi112.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.124497.log.gz 2024-09-11T06:53:16.982 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56349.log 2024-09-11T06:53:16.983 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64848.log 2024-09-11T06:53:16.983 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.47074.log: /var/log/ceph/cluster2-client.mirror.56349.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124224.log 2024-09-11T06:53:16.983 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.64848.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76832.log 2024-09-11T06:53:16.983 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.124224.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63914.log 2024-09-11T06:53:16.984 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.76832.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81569.log 2024-09-11T06:53:16.984 INFO:teuthology.orchestra.run.smithi112.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.124224.log.gz 2024-09-11T06:53:16.985 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.63914.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127773.log 2024-09-11T06:53:16.985 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.81569.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77169.log 2024-09-11T06:53:16.985 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.127773.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126244.log 2024-09-11T06:53:16.985 INFO:teuthology.orchestra.run.smithi112.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.127773.log.gz 2024-09-11T06:53:16.985 INFO:teuthology.orchestra.run.smithi112.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.38890.log.gz 2024-09-11T06:53:16.986 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34949.log 2024-09-11T06:53:16.986 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.77169.log: /var/log/ceph/cluster1-client.mirror.126244.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121824.log 2024-09-11T06:53:16.986 INFO:teuthology.orchestra.run.smithi112.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.126244.log.gz 2024-09-11T06:53:16.987 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.34949.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69267.log 2024-09-11T06:53:16.987 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.121824.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82038.log 2024-09-11T06:53:16.987 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.46702.log.gz 2024-09-11T06:53:16.988 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.69267.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71507.log 2024-09-11T06:53:16.988 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.82038.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116569.log 2024-09-11T06:53:16.989 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.71507.log: 90.8% -- replaced with /var/log/ceph/cluster2-client.mirror.121824.log.gz 2024-09-11T06:53:16.989 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125749.log 2024-09-11T06:53:16.989 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.116569.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128408.log 2024-09-11T06:53:16.990 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.125749.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.116569.log.gz 2024-09-11T06:53:16.990 INFO:teuthology.orchestra.run.smithi112.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.125749.log.gz 2024-09-11T06:53:16.990 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73003.log 2024-09-11T06:53:16.990 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.128408.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52678.log 2024-09-11T06:53:16.991 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.73003.log: 91.8% -- replaced with /var/log/ceph/cluster2-client.mirror.128408.log.gz 2024-09-11T06:53:16.992 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74964.log 2024-09-11T06:53:16.992 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.52678.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39913.log 2024-09-11T06:53:16.993 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.74964.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108300.log 2024-09-11T06:53:16.993 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.39913.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111243.log 2024-09-11T06:53:16.993 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.108300.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49128.log 2024-09-11T06:53:16.994 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.111243.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.111243.log.gz 2024-09-11T06:53:16.996 INFO:teuthology.orchestra.run.smithi112.stderr: 94.5% -- replaced with /var/log/ceph/cluster2-client.mirror.108300.log.gz 2024-09-11T06:53:16.996 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61430.log 2024-09-11T06:53:16.997 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.49128.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87050.log 2024-09-11T06:53:16.997 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.61430.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80741.log 2024-09-11T06:53:16.998 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.87050.log: /var/log/ceph/cluster2-client.mirror.80741.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31492.log 2024-09-11T06:53:16.998 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80055.log 2024-09-11T06:53:16.999 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.31492.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43999.log 2024-09-11T06:53:16.999 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.80055.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104175.log 2024-09-11T06:53:16.999 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.43999.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94086.log 2024-09-11T06:53:16.999 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.104175.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39762.log 2024-09-11T06:53:17.000 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.94086.log: 84.6% -- replaced with /var/log/ceph/cluster2-client.mirror.94086.log.gz 2024-09-11T06:53:17.000 INFO:teuthology.orchestra.run.smithi112.stderr: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.104175.log.gz 2024-09-11T06:53:17.001 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58814.log 2024-09-11T06:53:17.001 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.39762.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126986.log 2024-09-11T06:53:17.002 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.58814.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82895.log 2024-09-11T06:53:17.002 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54853.log 2024-09-11T06:53:17.002 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.126986.log: /var/log/ceph/cluster2-client.mirror.82895.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.126986.log.gz 2024-09-11T06:53:17.002 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.59713.log.gz 2024-09-11T06:53:17.003 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.59748.log.gz 2024-09-11T06:53:17.003 INFO:teuthology.orchestra.run.smithi112.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.56946.log.gz 2024-09-11T06:53:17.004 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79897.log 2024-09-11T06:53:17.004 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.54853.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52550.log 2024-09-11T06:53:17.005 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.79897.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.33957.log.gz 2024-09-11T06:53:17.005 INFO:teuthology.orchestra.run.smithi112.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.58403.log.gz 2024-09-11T06:53:17.006 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.131417.log 2024-09-11T06:53:17.006 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.58717.log.gz 2024-09-11T06:53:17.006 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.52550.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75150.log 2024-09-11T06:53:17.006 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.admin.131417.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.61240.log.gz 2024-09-11T06:53:17.006 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.131417.log.gz 2024-09-11T06:53:17.007 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36817.log 2024-09-11T06:53:17.007 INFO:teuthology.orchestra.run.smithi112.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.42312.log.gz/var/log/ceph/cluster2-client.mirror.75150.log: 2024-09-11T06:53:17.007 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62457.log 2024-09-11T06:53:17.008 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.36817.log: 95.3% -- replaced with /var/log/ceph/cluster2-client.mirror.94455.log.gz 2024-09-11T06:53:17.008 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130800.log 2024-09-11T06:53:17.008 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.62457.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35135.log 2024-09-11T06:53:17.008 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.51089.log.gz 2024-09-11T06:53:17.010 INFO:teuthology.orchestra.run.smithi112.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.54853.log.gz 2024-09-11T06:53:17.010 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.130800.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.55171.log.gz 2024-09-11T06:53:17.011 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86252.log 2024-09-11T06:53:17.011 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.47318.log.gz 2024-09-11T06:53:17.011 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.35135.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80232.log 2024-09-11T06:53:17.011 INFO:teuthology.orchestra.run.smithi112.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.130800.log.gz 2024-09-11T06:53:17.011 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.86252.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.77169.log.gz 2024-09-11T06:53:17.011 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.79897.log.gz 2024-09-11T06:53:17.011 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121640.log 2024-09-11T06:53:17.011 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.80232.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42689.log 2024-09-11T06:53:17.014 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.121640.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117889.log 2024-09-11T06:53:17.014 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107823.log 2024-09-11T06:53:17.014 INFO:teuthology.orchestra.run.smithi112.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.85263.log.gz 2024-09-11T06:53:17.014 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.117889.log: 89.2%gzip -- replaced with /var/log/ceph/cluster2-client.mirror.121640.log.gz -5 2024-09-11T06:53:17.014 INFO:teuthology.orchestra.run.smithi112.stderr: --verbose -- /var/log/ceph/cluster2-client.mirror.39727.log 2024-09-11T06:53:17.014 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.56070.log.gz 2024-09-11T06:53:17.014 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.61895.log.gz 2024-09-11T06:53:17.014 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.107823.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.107823.log.gz 2024-09-11T06:53:17.014 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129290.log 2024-09-11T06:53:17.015 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.42689.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54795.log 2024-09-11T06:53:17.015 INFO:teuthology.orchestra.run.smithi112.stderr: 92.0% -- replaced with /var/log/ceph/cluster1-client.mirror.117889.log.gz 2024-09-11T06:53:17.015 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.129290.log: /var/log/ceph/cluster2-client.mirror.39727.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45947.log 2024-09-11T06:53:17.015 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.54795.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116490.log 2024-09-11T06:53:17.015 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.129290.log.gz 2024-09-11T06:53:17.015 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.45947.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66251.log 2024-09-11T06:53:17.020 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.116490.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106580.log 2024-09-11T06:53:17.020 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.66251.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84670.log 2024-09-11T06:53:17.020 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.106580.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120282.log 2024-09-11T06:53:17.020 INFO:teuthology.orchestra.run.smithi112.stderr: 88.0%/var/log/ceph/cluster2-client.mirror.84670.log: -- replaced with /var/log/ceph/cluster2-client.mirror.106580.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39003.log 2024-09-11T06:53:17.020 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-11T06:53:17.020 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.118267.log 2024-09-11T06:53:17.020 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.120282.log: /var/log/ceph/cluster1-client.mirror.39003.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80432.log 2024-09-11T06:53:17.021 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.118267.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117777.log 2024-09-11T06:53:17.021 INFO:teuthology.orchestra.run.smithi112.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.120282.log.gz 2024-09-11T06:53:17.021 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.80432.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35042.log 2024-09-11T06:53:17.021 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84612.log 2024-09-11T06:53:17.021 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.117777.log: /var/log/ceph/cluster2-client.mirror.35042.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.117777.log.gz 2024-09-11T06:53:17.021 INFO:teuthology.orchestra.run.smithi112.stderr: 91.7% -- replaced with /var/log/ceph/cluster1-client.mirror.118267.log.gz 2024-09-11T06:53:17.021 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121662.log 2024-09-11T06:53:17.021 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89080.log 2024-09-11T06:53:17.022 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.84612.log: /var/log/ceph/cluster2-client.mirror.121662.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127211.log 2024-09-11T06:53:17.022 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.89080.log: 88.2%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58624.log 2024-09-11T06:53:17.022 INFO:teuthology.orchestra.run.smithi112.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.121662.log.gz 2024-09-11T06:53:17.022 INFO:teuthology.orchestra.run.smithi112.stderr: 84.7% -- replaced with /var/log/ceph/cluster2-client.mirror.89080.log.gz 2024-09-11T06:53:17.022 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.127211.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.127211.log.gz 2024-09-11T06:53:17.028 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64662.log 2024-09-11T06:53:17.028 INFO:teuthology.orchestra.run.smithi112.stderr: 96.7% -- replaced with /var/log/ceph/cluster2-client.mirror.116490.log.gz 2024-09-11T06:53:17.028 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122386.log 2024-09-11T06:53:17.028 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.58624.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65317.log 2024-09-11T06:53:17.028 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.64662.log: /var/log/ceph/cluster2-client.mirror.122386.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43007.log 2024-09-11T06:53:17.028 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.65317.log: 88.3%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124701.log 2024-09-11T06:53:17.029 INFO:teuthology.orchestra.run.smithi112.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.122386.log.gz 2024-09-11T06:53:17.029 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.43007.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87998.log 2024-09-11T06:53:17.029 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.124701.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110679.log 2024-09-11T06:53:17.029 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.87998.log: 88.4%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79506.log 2024-09-11T06:53:17.029 INFO:teuthology.orchestra.run.smithi112.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.124701.log.gz 2024-09-11T06:53:17.029 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.110679.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124306.log 2024-09-11T06:53:17.029 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.79506.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31903.log 2024-09-11T06:53:17.029 INFO:teuthology.orchestra.run.smithi112.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.110679.log.gz 2024-09-11T06:53:17.029 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.124306.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.124306.log.gz 2024-09-11T06:53:17.029 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.73619.log.gz 2024-09-11T06:53:17.029 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122474.log 2024-09-11T06:53:17.029 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.62457.log.gz 2024-09-11T06:53:17.029 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.31903.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37189.log 2024-09-11T06:53:17.030 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.53670.log.gz 2024-09-11T06:53:17.030 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.122474.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109116.log 2024-09-11T06:53:17.030 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.37189.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79223.log 2024-09-11T06:53:17.030 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.109116.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41816.log 2024-09-11T06:53:17.030 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.79223.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.109116.log.gz 2024-09-11T06:53:17.030 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45835.log 2024-09-11T06:53:17.030 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57845.log 2024-09-11T06:53:17.030 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.41816.log: /var/log/ceph/cluster2-client.mirror.45835.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127504.log 2024-09-11T06:53:17.030 INFO:teuthology.orchestra.run.smithi112.stderr: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.122474.log.gz 2024-09-11T06:53:17.033 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.57845.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53859.log 2024-09-11T06:53:17.033 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.127504.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.131365.log 2024-09-11T06:53:17.033 INFO:teuthology.orchestra.run.smithi112.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.127504.log.gz 2024-09-11T06:53:17.033 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.53859.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.53763.log.gz 2024-09-11T06:53:17.034 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66193.log 2024-09-11T06:53:17.034 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.admin.131365.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122315.log 2024-09-11T06:53:17.034 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.131365.log.gz 2024-09-11T06:53:17.034 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.66193.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65631.log 2024-09-11T06:53:17.034 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.52550.log.gz 2024-09-11T06:53:17.034 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.122315.log: 88.4% -- replaced with /var/log/ceph/cluster1-client.mirror.54215.log.gz 2024-09-11T06:53:17.034 INFO:teuthology.orchestra.run.smithi112.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.122315.log.gz 2024-09-11T06:53:17.034 INFO:teuthology.orchestra.run.smithi112.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.66902.log.gz 2024-09-11T06:53:17.034 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.74553.log.gz 2024-09-11T06:53:17.034 INFO:teuthology.orchestra.run.smithi112.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.75150.log.gz 2024-09-11T06:53:17.034 INFO:teuthology.orchestra.run.smithi112.stderr: 56.0% -- replaced with /var/log/ceph/cluster2-client.mirror.29654.log.gz 2024-09-11T06:53:17.034 INFO:teuthology.orchestra.run.smithi112.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.42067.log.gz 2024-09-11T06:53:17.034 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.42689.log.gz 2024-09-11T06:53:17.035 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.80204.log 2024-09-11T06:53:17.035 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.65631.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81232.log 2024-09-11T06:53:17.035 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.admin.80204.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120339.log 2024-09-11T06:53:17.035 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.81232.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94659.log 2024-09-11T06:53:17.035 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.120339.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62422.log 2024-09-11T06:53:17.037 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.94659.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78607.log 2024-09-11T06:53:17.038 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.62422.log: 92.0% -- replaced with /var/log/ceph/cluster2-client.mirror.120339.log.gz 2024-09-11T06:53:17.038 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61019.log 2024-09-11T06:53:17.038 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.78607.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42167.log 2024-09-11T06:53:17.038 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.61019.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50438.log 2024-09-11T06:53:17.038 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.42167.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40882.log 2024-09-11T06:53:17.038 INFO:teuthology.orchestra.run.smithi112.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.86941.log.gz 2024-09-11T06:53:17.038 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78793.log 2024-09-11T06:53:17.038 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.50438.log: /var/log/ceph/cluster2-client.mirror.40882.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45305.log 2024-09-11T06:53:17.038 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.78793.log: 84.6% -- replaced with /var/log/ceph/cluster2-client.mirror.88612.log.gz 2024-09-11T06:53:17.038 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128779.log 2024-09-11T06:53:17.039 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.45305.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88125.log 2024-09-11T06:53:17.039 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.128779.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.30238.log 2024-09-11T06:53:17.039 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.88125.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.128779.log.gz 2024-09-11T06:53:17.040 INFO:teuthology.orchestra.run.smithi112.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.82895.log.gz 2024-09-11T06:53:17.040 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.90795.log 2024-09-11T06:53:17.040 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.admin.30238.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39070.log 2024-09-11T06:53:17.040 INFO:teuthology.orchestra.run.smithi112.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.87050.log.gz 2024-09-11T06:53:17.041 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128121.log 2024-09-11T06:53:17.041 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.90795.log: 84.6% -- replaced with /var/log/ceph/cluster2-client.mirror.90795.log.gz 2024-09-11T06:53:17.041 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.39070.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32744.log 2024-09-11T06:53:17.042 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.128121.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72406.log 2024-09-11T06:53:17.042 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.32744.log: 95.8% -- replaced with /var/log/ceph/cluster2-client.mirror.95176.log.gz 2024-09-11T06:53:17.042 INFO:teuthology.orchestra.run.smithi112.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.128121.log.gz 2024-09-11T06:53:17.043 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55915.log 2024-09-11T06:53:17.043 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.72406.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63108.log 2024-09-11T06:53:17.043 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.55915.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42410.log 2024-09-11T06:53:17.043 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.63108.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92463.log 2024-09-11T06:53:17.045 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.42410.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48442.log 2024-09-11T06:53:17.045 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.92463.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69046.log 2024-09-11T06:53:17.045 INFO:teuthology.orchestra.run.smithi112.stderr: 84.4% -- replaced with /var/log/ceph/cluster2-client.mirror.92463.log.gz 2024-09-11T06:53:17.045 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.48442.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59186.log 2024-09-11T06:53:17.045 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129057.log 2024-09-11T06:53:17.045 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.69046.log: /var/log/ceph/cluster2-client.mirror.59186.log: gzip -5 --verbose -- /var/log/ceph/cluster2.audit.log 2024-09-11T06:53:17.045 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48570.log 2024-09-11T06:53:17.049 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2.audit.log: /var/log/ceph/cluster2-client.mirror.129057.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129890.log 2024-09-11T06:53:17.049 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.48570.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121202.log 2024-09-11T06:53:17.049 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.129890.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30686.log 2024-09-11T06:53:17.049 INFO:teuthology.orchestra.run.smithi112.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.129057.log.gz/var/log/ceph/cluster2-client.mirror.121202.log: 2024-09-11T06:53:17.049 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40103.log 2024-09-11T06:53:17.050 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.129890.log.gz 2024-09-11T06:53:17.050 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.30686.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.121202.log.gz 2024-09-11T06:53:17.050 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77673.log 2024-09-11T06:53:17.050 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.40103.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123234.log 2024-09-11T06:53:17.050 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.77673.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64325.log 2024-09-11T06:53:17.050 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.123234.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44968.log 2024-09-11T06:53:17.050 INFO:teuthology.orchestra.run.smithi112.stderr:gzip/var/log/ceph/cluster2-client.mirror.64325.log: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33922.log 2024-09-11T06:53:17.050 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.44968.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.72782.log.gz 2024-09-11T06:53:17.050 INFO:teuthology.orchestra.run.smithi112.stderr: 90.9% -- replaced with /var/log/ceph/cluster2-client.mirror.123234.log.gz 2024-09-11T06:53:17.050 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.36817.log.gz 2024-09-11T06:53:17.050 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88510.log 2024-09-11T06:53:17.054 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.33922.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127526.log 2024-09-11T06:53:17.054 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.88510.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124519.log 2024-09-11T06:53:17.054 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.127526.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58217.log 2024-09-11T06:53:17.055 INFO:teuthology.orchestra.run.smithi112.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.127526.log.gz 2024-09-11T06:53:17.055 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.124519.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127166.log 2024-09-11T06:53:17.055 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.58217.log: 88.2%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34856.log 2024-09-11T06:53:17.055 INFO:teuthology.orchestra.run.smithi112.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.124519.log.gz 2024-09-11T06:53:17.055 INFO:teuthology.orchestra.run.smithi112.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.86744.log.gz 2024-09-11T06:53:17.055 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.127166.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28586.log 2024-09-11T06:53:17.055 INFO:teuthology.orchestra.run.smithi112.stderr: 87.1% -- replaced with /var/log/ceph/cluster1-client.mirror.127166.log.gz 2024-09-11T06:53:17.055 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56570.log 2024-09-11T06:53:17.055 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.34856.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82219.log 2024-09-11T06:53:17.055 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.admin.28586.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.28586.log.gz 2024-09-11T06:53:17.055 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.56570.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59616.log 2024-09-11T06:53:17.055 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.82219.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45154.log 2024-09-11T06:53:17.055 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.59616.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125636.log 2024-09-11T06:53:17.055 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.45154.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73096.log 2024-09-11T06:53:17.055 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.125636.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.29862.log 2024-09-11T06:53:17.056 INFO:teuthology.orchestra.run.smithi112.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.125636.log.gz 2024-09-11T06:53:17.058 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.73096.log: 96.1% -- replaced with /var/log/ceph/cluster2-client.mirror.96879.log.gz 2024-09-11T06:53:17.058 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54352.log 2024-09-11T06:53:17.059 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.1.29862.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57132.log 2024-09-11T06:53:17.059 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.54352.log: gzip -5 --verbose -- /var/log/ceph/cluster2.log 2024-09-11T06:53:17.060 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.57132.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47225.log 2024-09-11T06:53:17.060 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45793.log 2024-09-11T06:53:17.060 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2.log: /var/log/ceph/cluster2-client.mirror.47225.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42724.log 2024-09-11T06:53:17.061 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.45793.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80900.log 2024-09-11T06:53:17.062 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.42724.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56535.log 2024-09-11T06:53:17.062 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71693.log 2024-09-11T06:53:17.062 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.80900.log: /var/log/ceph/cluster2-client.mirror.56535.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125218.log 2024-09-11T06:53:17.062 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.71693.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85704.log 2024-09-11T06:53:17.062 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66379.log 2024-09-11T06:53:17.064 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.125218.log: /var/log/ceph/cluster2-client.mirror.85704.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39028.log 2024-09-11T06:53:17.064 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.125218.log.gz 2024-09-11T06:53:17.064 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.66379.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48287.log 2024-09-11T06:53:17.064 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.39028.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75208.log 2024-09-11T06:53:17.064 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57655.log 2024-09-11T06:53:17.064 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.48287.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55822.log 2024-09-11T06:53:17.064 INFO:teuthology.orchestra.run.smithi112.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.56291.log.gz 2024-09-11T06:53:17.065 INFO:teuthology.orchestra.run.smithi112.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.56349.log.gz 2024-09-11T06:53:17.066 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.75208.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84294.log 2024-09-11T06:53:17.066 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.55822.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.56663.log.gz 2024-09-11T06:53:17.066 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.84294.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113634.log 2024-09-11T06:53:17.067 INFO:teuthology.orchestra.run.smithi112.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.56849.log.gz 2024-09-11T06:53:17.067 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29296.log 2024-09-11T06:53:17.067 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.57655.log: /var/log/ceph/cluster2-client.mirror.113634.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59806.log 2024-09-11T06:53:17.067 INFO:teuthology.orchestra.run.smithi112.stderr: 87.6% -- replaced with /var/log/ceph/cluster2.log.gz 2024-09-11T06:53:17.067 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.admin.29296.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130251.log 2024-09-11T06:53:17.067 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.59806.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72069.log 2024-09-11T06:53:17.067 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.130251.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63763.log 2024-09-11T06:53:17.067 INFO:teuthology.orchestra.run.smithi112.stderr: 83.5% -- replaced with /var/log/ceph/cluster2-client.mirror.130251.log.gz 2024-09-11T06:53:17.070 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.72069.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73844.log 2024-09-11T06:53:17.070 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.63763.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57938.log 2024-09-11T06:53:17.070 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.73844.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66344.log 2024-09-11T06:53:17.070 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.57938.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43158.log 2024-09-11T06:53:17.070 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.66344.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85775.log 2024-09-11T06:53:17.070 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.43158.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.29729.log 2024-09-11T06:53:17.070 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.85775.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.118986.log 2024-09-11T06:53:17.070 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.0.29729.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119904.log 2024-09-11T06:53:17.070 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.119795.log 2024-09-11T06:53:17.071 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.119904.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.119904.log.gz 2024-09-11T06:53:17.071 INFO:teuthology.orchestra.run.smithi112.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.38685.log.gz 2024-09-11T06:53:17.072 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41037.log 2024-09-11T06:53:17.072 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.119795.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128849.log 2024-09-11T06:53:17.072 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.39390.log.gz 2024-09-11T06:53:17.072 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.41037.log: 96.7% -- replaced with /var/log/ceph/cluster2-client.mirror.113634.log.gz 2024-09-11T06:53:17.072 INFO:teuthology.orchestra.run.smithi112.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.39727.log.gz 2024-09-11T06:53:17.072 INFO:teuthology.orchestra.run.smithi112.stderr: 91.8% -- replaced with /var/log/ceph/cluster1-client.mirror.119795.log.gz 2024-09-11T06:53:17.076 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.118986.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49783.log 2024-09-11T06:53:17.076 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.128849.log: 88.1%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64755.log 2024-09-11T06:53:17.076 INFO:teuthology.orchestra.run.smithi112.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.118986.log.gz 2024-09-11T06:53:17.077 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.49783.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.128849.log.gz 2024-09-11T06:53:17.077 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49934.log 2024-09-11T06:53:17.077 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.64755.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76925.log 2024-09-11T06:53:17.077 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.49934.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130544.log 2024-09-11T06:53:17.077 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.76925.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110370.log 2024-09-11T06:53:17.077 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103777.log 2024-09-11T06:53:17.077 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.130544.log: /var/log/ceph/cluster1-client.mirror.110370.log: 83.6% -- replaced with /var/log/ceph/cluster2-client.mirror.130544.log.gz 2024-09-11T06:53:17.077 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.131066.log 2024-09-11T06:53:17.077 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.103777.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73902.log 2024-09-11T06:53:17.077 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.admin.131066.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60457.log 2024-09-11T06:53:17.077 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.131066.log.gz 2024-09-11T06:53:17.077 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.73902.log: 91.9% -- replaced with /var/log/ceph/cluster1-client.mirror.110370.log.gz 2024-09-11T06:53:17.077 INFO:teuthology.orchestra.run.smithi112.stderr: 92.0% -- replaced with /var/log/ceph/cluster2-client.mirror.103777.log.gz 2024-09-11T06:53:17.078 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66437.log 2024-09-11T06:53:17.078 INFO:teuthology.orchestra.run.smithi112.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.72685.log.gz 2024-09-11T06:53:17.078 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.60457.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66158.log 2024-09-11T06:53:17.080 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.66437.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117725.log 2024-09-11T06:53:17.080 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.66158.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125546.log 2024-09-11T06:53:17.080 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120793.log 2024-09-11T06:53:17.080 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.117725.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.40196.log.gz 2024-09-11T06:53:17.080 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.125546.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87189.log 2024-09-11T06:53:17.080 INFO:teuthology.orchestra.run.smithi112.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.125546.log.gz 2024-09-11T06:53:17.080 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.120793.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45681.log 2024-09-11T06:53:17.086 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.87189.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.120793.log.gz 2024-09-11T06:53:17.086 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47787.log 2024-09-11T06:53:17.087 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.45681.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113149.log 2024-09-11T06:53:17.087 INFO:teuthology.orchestra.run.smithi112.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.43999.log.gz 2024-09-11T06:53:17.087 INFO:teuthology.orchestra.run.smithi112.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.43941.log.gz 2024-09-11T06:53:17.087 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.47787.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31620.log 2024-09-11T06:53:17.087 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.44278.log.gz 2024-09-11T06:53:17.087 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.113149.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125125.log 2024-09-11T06:53:17.087 INFO:teuthology.orchestra.run.smithi112.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.113149.log.gz 2024-09-11T06:53:17.087 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.31620.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74274.log 2024-09-11T06:53:17.087 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.125125.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113181.log 2024-09-11T06:53:17.087 INFO:teuthology.orchestra.run.smithi112.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.56477.log.gz 2024-09-11T06:53:17.087 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.74274.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43720.log 2024-09-11T06:53:17.088 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.113181.log: gzip 89.2% -5 --verbose -- replaced with /var/log/ceph/cluster2-client.mirror.125125.log.gz -- 2024-09-11T06:53:17.088 INFO:teuthology.orchestra.run.smithi112.stderr: /var/log/ceph/cluster2-client.mirror.58066.log 2024-09-11T06:53:17.088 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86205.log 2024-09-11T06:53:17.088 INFO:teuthology.orchestra.run.smithi112.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.113181.log.gz 2024-09-11T06:53:17.088 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.61019.log.gz 2024-09-11T06:53:17.088 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.58066.log: /var/log/ceph/cluster2-client.mirror.43720.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34015.log 2024-09-11T06:53:17.088 INFO:teuthology.orchestra.run.smithi112.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.31903.log.gz 2024-09-11T06:53:17.088 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123789.log 2024-09-11T06:53:17.088 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.41037.log.gz 2024-09-11T06:53:17.088 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.86205.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102280.log 2024-09-11T06:53:17.088 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.34015.log: /var/log/ceph/cluster2-client.mirror.123789.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.74274.log.gz 2024-09-11T06:53:17.088 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58124.log 2024-09-11T06:53:17.089 INFO:teuthology.orchestra.run.smithi112.stderr: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.88976.log.gz 2024-09-11T06:53:17.089 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.102280.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.131209.log 2024-09-11T06:53:17.089 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.102280.log.gz/var/log/ceph/cluster2-client.mirror.58124.log: 2024-09-11T06:53:17.089 INFO:teuthology.orchestra.run.smithi112.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.123789.log.gz 2024-09-11T06:53:17.089 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31717.log 2024-09-11T06:53:17.089 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55229.log 2024-09-11T06:53:17.089 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.admin.131209.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.131209.log.gz 2024-09-11T06:53:17.089 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.31717.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105683.log 2024-09-11T06:53:17.089 INFO:teuthology.orchestra.run.smithi112.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.34387.log.gz 2024-09-11T06:53:17.089 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.55229.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105977.log 2024-09-11T06:53:17.089 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.105683.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68856.log 2024-09-11T06:53:17.097 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.105977.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41723.log 2024-09-11T06:53:17.097 INFO:teuthology.orchestra.run.smithi112.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.35135.log.gz 2024-09-11T06:53:17.097 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.68856.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.105977.log.gz 2024-09-11T06:53:17.097 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.49783.log.gz 2024-09-11T06:53:17.097 INFO:teuthology.orchestra.run.smithi112.stderr: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.105683.log.gz 2024-09-11T06:53:17.098 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98144.log 2024-09-11T06:53:17.098 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.41723.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76049.log 2024-09-11T06:53:17.098 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.98144.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.54795.log.gz 2024-09-11T06:53:17.098 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62364.log 2024-09-11T06:53:17.098 INFO:teuthology.orchestra.run.smithi112.stderr: 84.6% -- replaced with /var/log/ceph/cluster2-client.mirror.98144.log.gz 2024-09-11T06:53:17.098 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.76049.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44840.log 2024-09-11T06:53:17.098 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.62364.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41630.log 2024-09-11T06:53:17.098 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.44840.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129797.log 2024-09-11T06:53:17.098 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.41630.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126716.log 2024-09-11T06:53:17.098 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.129797.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51372.log 2024-09-11T06:53:17.099 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.126716.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47225.log.gz 2024-09-11T06:53:17.099 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67678.log 2024-09-11T06:53:17.099 INFO:teuthology.orchestra.run.smithi112.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.126716.log.gz 2024-09-11T06:53:17.099 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.51372.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129952.log 2024-09-11T06:53:17.099 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.67678.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127414.log 2024-09-11T06:53:17.099 INFO:teuthology.orchestra.run.smithi112.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.65631.log.gz 2024-09-11T06:53:17.099 INFO:teuthology.orchestra.run.smithi112.stderr: 96.9%/var/log/ceph/cluster2-client.mirror.129952.log: -- replaced with /var/log/ceph/cluster2-client.mirror.117725.log.gz 2024-09-11T06:53:17.099 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71414.log 2024-09-11T06:53:17.099 INFO:teuthology.orchestra.run.smithi112.stderr: 83.6% -- replaced with /var/log/ceph/cluster2-client.mirror.129952.log.gz 2024-09-11T06:53:17.099 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.127414.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130112.log 2024-09-11T06:53:17.099 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.71414.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.127414.log.gz 2024-09-11T06:53:17.100 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97904.log 2024-09-11T06:53:17.100 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34763.log 2024-09-11T06:53:17.100 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.130112.log: /var/log/ceph/cluster2-client.mirror.97904.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32616.log 2024-09-11T06:53:17.100 INFO:teuthology.orchestra.run.smithi112.stderr: 84.6% -- replaced with /var/log/ceph/cluster2-client.mirror.97904.log.gz 2024-09-11T06:53:17.100 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.34763.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34519.log 2024-09-11T06:53:17.100 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.32616.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70073.log 2024-09-11T06:53:17.100 INFO:teuthology.orchestra.run.smithi112.stderr: 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.130112.log.gz 2024-09-11T06:53:17.100 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.34519.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106612.log 2024-09-11T06:53:17.100 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.70073.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79130.log 2024-09-11T06:53:17.100 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.106612.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41165.log 2024-09-11T06:53:17.100 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.79130.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72127.log 2024-09-11T06:53:17.100 INFO:teuthology.orchestra.run.smithi112.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.106612.log.gz 2024-09-11T06:53:17.101 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.41165.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49283.log 2024-09-11T06:53:17.101 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.72127.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28512.log 2024-09-11T06:53:17.101 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40754.log 2024-09-11T06:53:17.101 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.49283.log: /var/log/ceph/cluster2-client.admin.28512.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127436.log 2024-09-11T06:53:17.101 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.28512.log.gz 2024-09-11T06:53:17.101 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.40754.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127459.log 2024-09-11T06:53:17.101 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.127436.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126086.log 2024-09-11T06:53:17.101 INFO:teuthology.orchestra.run.smithi112.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.127436.log.gz 2024-09-11T06:53:17.101 INFO:teuthology.orchestra.run.smithi112.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.70759.log.gz 2024-09-11T06:53:17.101 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.127459.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.127459.log.gz 2024-09-11T06:53:17.114 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52492.log 2024-09-11T06:53:17.114 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.71507.log.gz 2024-09-11T06:53:17.114 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.126086.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113913.log 2024-09-11T06:53:17.114 INFO:teuthology.orchestra.run.smithi112.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.126086.log.gz 2024-09-11T06:53:17.114 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.52492.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117458.log 2024-09-11T06:53:17.115 INFO:teuthology.orchestra.run.smithi112.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.83864.log.gz 2024-09-11T06:53:17.115 INFO:teuthology.orchestra.run.smithi112.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.78793.log.gz 2024-09-11T06:53:17.115 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.113913.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80124.log 2024-09-11T06:53:17.115 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.113913.log.gz 2024-09-11T06:53:17.115 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.117458.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57411.log 2024-09-11T06:53:17.115 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.80124.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.117458.log.gz 2024-09-11T06:53:17.115 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102019.log 2024-09-11T06:53:17.115 INFO:teuthology.orchestra.run.smithi112.stderr:gzip/var/log/ceph/cluster2-client.mirror.57411.log: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55415.log 2024-09-11T06:53:17.115 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.102019.log: 1.3% -- replaced with /var/log/ceph/cluster1-client.mirror.102019.log.gz 2024-09-11T06:53:17.115 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128268.log 2024-09-11T06:53:17.116 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32426.log 2024-09-11T06:53:17.116 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.55415.log: /var/log/ceph/cluster1-client.mirror.128268.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89410.log 2024-09-11T06:53:17.116 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.32426.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129683.log 2024-09-11T06:53:17.116 INFO:teuthology.orchestra.run.smithi112.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.128268.log.gz 2024-09-11T06:53:17.116 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.89410.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109972.log 2024-09-11T06:53:17.116 INFO:teuthology.orchestra.run.smithi112.stderr: 84.7% -- replaced with /var/log/ceph/cluster2-client.mirror.89410.log.gz 2024-09-11T06:53:17.116 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.129683.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65968.log 2024-09-11T06:53:17.116 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.109972.log: 87.7%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72534.log 2024-09-11T06:53:17.116 INFO:teuthology.orchestra.run.smithi112.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.129683.log.gz 2024-09-11T06:53:17.116 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.65968.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73433.log 2024-09-11T06:53:17.116 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.72534.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67736.log 2024-09-11T06:53:17.116 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.73433.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116894.log 2024-09-11T06:53:17.117 INFO:teuthology.orchestra.run.smithi112.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.66623.log.gz 2024-09-11T06:53:17.117 INFO:teuthology.orchestra.run.smithi112.stderr:gzip/var/log/ceph/cluster2-client.mirror.67736.log: -5 --verbose -- /var/log/ceph/cluster2-client.admin.28274.log 2024-09-11T06:53:17.117 INFO:teuthology.orchestra.run.smithi112.stderr: 92.0% -- replaced with /var/log/ceph/cluster2-client.mirror.109972.log.gz 2024-09-11T06:53:17.117 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71193.log 2024-09-11T06:53:17.117 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.116894.log: /var/log/ceph/cluster2-client.admin.28274.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53484.log 2024-09-11T06:53:17.117 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.28274.log.gz 2024-09-11T06:53:17.117 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.71193.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64007.log 2024-09-11T06:53:17.117 INFO:teuthology.orchestra.run.smithi112.stderr: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.116894.log.gz 2024-09-11T06:53:17.117 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85444.log 2024-09-11T06:53:17.117 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.53484.log: /var/log/ceph/cluster2-client.mirror.64007.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79974.log 2024-09-11T06:53:17.118 INFO:teuthology.orchestra.run.smithi112.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.55229.log.gz 2024-09-11T06:53:17.118 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.85444.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45623.log 2024-09-11T06:53:17.118 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.79974.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118247.log 2024-09-11T06:53:17.118 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.45623.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38065.log 2024-09-11T06:53:17.118 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.118247.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.55822.log.gz 2024-09-11T06:53:17.118 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116005.log 2024-09-11T06:53:17.118 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.38065.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66565.log 2024-09-11T06:53:17.118 INFO:teuthology.orchestra.run.smithi112.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.60457.log.gz 2024-09-11T06:53:17.118 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.116005.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106966.log 2024-09-11T06:53:17.118 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.66565.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.116005.log.gz 2024-09-11T06:53:17.118 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87503.log 2024-09-11T06:53:17.118 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.106966.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80339.log 2024-09-11T06:53:17.118 INFO:teuthology.orchestra.run.smithi112.stderr: 91.9% -- replaced with /var/log/ceph/cluster2-client.mirror.118247.log.gz 2024-09-11T06:53:17.119 INFO:teuthology.orchestra.run.smithi112.stderr: 87.9%/var/log/ceph/cluster2-client.mirror.87503.log: -- replaced with /var/log/ceph/cluster2-client.mirror.106966.log.gz 2024-09-11T06:53:17.119 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119869.log 2024-09-11T06:53:17.119 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126604.log 2024-09-11T06:53:17.119 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.80339.log: /var/log/ceph/cluster2-client.mirror.119869.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42972.log 2024-09-11T06:53:17.119 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.68763.log.gz 2024-09-11T06:53:17.119 INFO:teuthology.orchestra.run.smithi112.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.119869.log.gz 2024-09-11T06:53:17.119 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.126604.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.126604.log.gz 2024-09-11T06:53:17.119 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108730.log 2024-09-11T06:53:17.119 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.42972.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117347.log 2024-09-11T06:53:17.119 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75619.log 2024-09-11T06:53:17.119 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.108730.log: /var/log/ceph/cluster2-client.mirror.117347.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76363.log 2024-09-11T06:53:17.119 INFO:teuthology.orchestra.run.smithi112.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.108730.log.gz 2024-09-11T06:53:17.119 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.75619.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114709.log 2024-09-11T06:53:17.120 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.76363.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103797.log 2024-09-11T06:53:17.120 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.114709.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41072.log 2024-09-11T06:53:17.120 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.103797.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128061.log 2024-09-11T06:53:17.120 INFO:teuthology.orchestra.run.smithi112.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.114709.log.gz 2024-09-11T06:53:17.120 INFO:teuthology.orchestra.run.smithi112.stderr: 92.9% -- replaced with /var/log/ceph/cluster2-client.mirror.46051.log.gz 2024-09-11T06:53:17.120 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.41072.log: 91.8% -- replaced with /var/log/ceph/cluster1-client.mirror.103797.log.gz 2024-09-11T06:53:17.120 INFO:teuthology.orchestra.run.smithi112.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.71658.log.gz 2024-09-11T06:53:17.120 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97513.log 2024-09-11T06:53:17.120 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.128061.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46946.log 2024-09-11T06:53:17.128 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.128061.log.gz 2024-09-11T06:53:17.128 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36197.log 2024-09-11T06:53:17.128 INFO:teuthology.orchestra.run.smithi112.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.75208.log.gz 2024-09-11T06:53:17.128 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.46946.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42503.log 2024-09-11T06:53:17.128 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.97513.log: /var/log/ceph/cluster2-client.mirror.36197.log: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.117347.log.gz 2024-09-11T06:53:17.128 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52364.log 2024-09-11T06:53:17.128 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.42503.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61333.log 2024-09-11T06:53:17.128 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77487.log 2024-09-11T06:53:17.128 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.52364.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.58217.log.gz 2024-09-11T06:53:17.128 INFO:teuthology.orchestra.run.smithi112.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.73433.log.gz 2024-09-11T06:53:17.128 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.61333.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86091.log 2024-09-11T06:53:17.129 INFO:teuthology.orchestra.run.smithi112.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.87189.log.gz 2024-09-11T06:53:17.129 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.77487.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92637.log 2024-09-11T06:53:17.129 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.86091.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.84294.log.gz 2024-09-11T06:53:17.129 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.92637.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37937.log 2024-09-11T06:53:17.129 INFO:teuthology.orchestra.run.smithi112.stderr: 84.5% -- replaced with /var/log/ceph/cluster2-client.mirror.92637.log.gz 2024-09-11T06:53:17.129 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62736.log 2024-09-11T06:53:17.129 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.37937.log: 93.3% -- replaced with /var/log/ceph/cluster2-client.mirror.82130.log.gz 2024-09-11T06:53:17.129 INFO:teuthology.orchestra.run.smithi112.stderr: 96.0% -- replaced with /var/log/ceph/cluster2-client.mirror.129398.log.gz 2024-09-11T06:53:17.129 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.69267.log.gz 2024-09-11T06:53:17.129 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106141.log 2024-09-11T06:53:17.129 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.62736.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85077.log 2024-09-11T06:53:17.130 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81476.log 2024-09-11T06:53:17.130 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.106141.log: /var/log/ceph/cluster2-client.mirror.85077.log: 84.7% -- replaced with /var/log/ceph/cluster2-client.mirror.88125.log.gz 2024-09-11T06:53:17.130 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53426.log 2024-09-11T06:53:17.130 INFO:teuthology.orchestra.run.smithi112.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.106141.log.gz 2024-09-11T06:53:17.130 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.81476.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46481.log 2024-09-11T06:53:17.130 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126649.log 2024-09-11T06:53:17.130 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.53426.log: /var/log/ceph/cluster2-client.mirror.46481.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115340.log 2024-09-11T06:53:17.130 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33678.log 2024-09-11T06:53:17.130 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.126649.log: /var/log/ceph/cluster2-client.mirror.115340.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.126649.log.gz 2024-09-11T06:53:17.130 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67492.log 2024-09-11T06:53:17.130 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose --/var/log/ceph/cluster2-client.mirror.33678.log: /var/log/ceph/cluster2-client.mirror.106452.log 2024-09-11T06:53:17.130 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.67492.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117048.log 2024-09-11T06:53:17.131 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51279.log 2024-09-11T06:53:17.131 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.106452.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.36197.log.gz 2024-09-11T06:53:17.131 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.117048.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32054.log 2024-09-11T06:53:17.131 INFO:teuthology.orchestra.run.smithi112.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.106452.log.gz 2024-09-11T06:53:17.131 INFO:teuthology.orchestra.run.smithi112.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.117048.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86272.log 2024-09-11T06:53:17.131 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-11T06:53:17.131 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.51279.log: /var/log/ceph/cluster2-client.mirror.32054.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83771.log 2024-09-11T06:53:17.131 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.86272.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79692.log 2024-09-11T06:53:17.131 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.83771.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116062.log 2024-09-11T06:53:17.131 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113945.log 2024-09-11T06:53:17.131 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.79692.log: /var/log/ceph/cluster2-client.mirror.116062.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.118653.log 2024-09-11T06:53:17.131 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.113945.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49035.log 2024-09-11T06:53:17.131 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.118653.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.113945.log.gz 2024-09-11T06:53:17.131 INFO:teuthology.orchestra.run.smithi112.stderr: 91.7% -- replaced with /var/log/ceph/cluster2-client.mirror.116062.log.gz 2024-09-11T06:53:17.141 INFO:teuthology.orchestra.run.smithi112.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.118653.log.gz 2024-09-11T06:53:17.141 INFO:teuthology.orchestra.run.smithi112.stderr: 96.7% -- replaced with /var/log/ceph/cluster2-client.mirror.115340.log.gz 2024-09-11T06:53:17.141 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129772.log 2024-09-11T06:53:17.141 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126739.log 2024-09-11T06:53:17.142 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.49035.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37003.log 2024-09-11T06:53:17.142 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.129772.log: /var/log/ceph/cluster1-client.mirror.126739.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87440.log 2024-09-11T06:53:17.142 INFO:teuthology.orchestra.run.smithi112.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.126739.log.gz 2024-09-11T06:53:17.142 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.37003.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130270.log 2024-09-11T06:53:17.142 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57097.log 2024-09-11T06:53:17.142 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.87440.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.129772.log.gz 2024-09-11T06:53:17.142 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.130270.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85954.log 2024-09-11T06:53:17.142 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.57097.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33585.log 2024-09-11T06:53:17.142 INFO:teuthology.orchestra.run.smithi112.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.130270.log.gz 2024-09-11T06:53:17.142 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.85954.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.30269.log 2024-09-11T06:53:17.143 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.33585.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76983.log 2024-09-11T06:53:17.143 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.admin.30269.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57469.log 2024-09-11T06:53:17.143 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130585.log 2024-09-11T06:53:17.143 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.76983.log: /var/log/ceph/cluster2-client.mirror.57469.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51186.log 2024-09-11T06:53:17.143 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35321.log 2024-09-11T06:53:17.143 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.130585.log: /var/log/ceph/cluster2-client.mirror.51186.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30651.log 2024-09-11T06:53:17.143 INFO:teuthology.orchestra.run.smithi112.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.130585.log.gz 2024-09-11T06:53:17.143 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36069.log 2024-09-11T06:53:17.143 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.35321.log: /var/log/ceph/cluster2-client.mirror.30651.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58907.log 2024-09-11T06:53:17.143 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.36069.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35883.log 2024-09-11T06:53:17.144 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.58907.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77355.log 2024-09-11T06:53:17.144 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86790.log 2024-09-11T06:53:17.144 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.35883.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125414.log 2024-09-11T06:53:17.144 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.77355.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37507.log 2024-09-11T06:53:17.144 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.86790.log: /var/log/ceph/cluster1-client.mirror.125414.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56384.log 2024-09-11T06:53:17.144 INFO:teuthology.orchestra.run.smithi112.stderr: 1.3% -- replaced with /var/log/ceph/cluster1-client.mirror.125414.log.gz 2024-09-11T06:53:17.144 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38030.log 2024-09-11T06:53:17.144 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.37507.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42445.log 2024-09-11T06:53:17.144 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.56384.log: /var/log/ceph/cluster2-client.mirror.38030.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73937.log 2024-09-11T06:53:17.144 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.29364.log 2024-09-11T06:53:17.145 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.42445.log: /var/log/ceph/cluster2-client.mirror.73937.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57376.log 2024-09-11T06:53:17.145 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120633.log 2024-09-11T06:53:17.145 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.29364.log: /var/log/ceph/cluster2-client.mirror.57376.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102720.log 2024-09-11T06:53:17.145 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.120633.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78386.log 2024-09-11T06:53:17.145 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.102720.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.120633.log.gz 2024-09-11T06:53:17.145 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124639.log 2024-09-11T06:53:17.145 INFO:teuthology.orchestra.run.smithi112.stderr: 88.2%/var/log/ceph/cluster2-client.mirror.78386.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55039.log 2024-09-11T06:53:17.145 INFO:teuthology.orchestra.run.smithi112.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.102720.log.gz 2024-09-11T06:53:17.145 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37972.log 2024-09-11T06:53:17.145 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.124639.log: /var/log/ceph/cluster2-client.mirror.55039.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80833.log 2024-09-11T06:53:17.145 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127868.log 2024-09-11T06:53:17.145 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.37972.log: /var/log/ceph/cluster2-client.mirror.80833.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39048.log 2024-09-11T06:53:17.146 INFO:teuthology.orchestra.run.smithi112.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.124639.log.gz 2024-09-11T06:53:17.146 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.127868.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68577.log 2024-09-11T06:53:17.146 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116356.log 2024-09-11T06:53:17.146 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.39048.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.127868.log.gz 2024-09-11T06:53:17.163 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.68577.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49411.log 2024-09-11T06:53:17.164 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.116356.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78324.log 2024-09-11T06:53:17.164 INFO:teuthology.orchestra.run.smithi112.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.116356.log.gz 2024-09-11T06:53:17.164 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.49411.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33267.log 2024-09-11T06:53:17.164 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.78324.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72441.log 2024-09-11T06:53:17.164 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74592.log 2024-09-11T06:53:17.164 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.33267.log: /var/log/ceph/cluster2-client.mirror.72441.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36573.log 2024-09-11T06:53:17.164 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125591.log 2024-09-11T06:53:17.164 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.74592.log: /var/log/ceph/cluster2-client.mirror.36573.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34484.log 2024-09-11T06:53:17.164 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.125591.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.29330.log 2024-09-11T06:53:17.165 INFO:teuthology.orchestra.run.smithi112.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.125591.log.gz 2024-09-11T06:53:17.165 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.34484.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127708.log 2024-09-11T06:53:17.165 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.29330.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40975.log 2024-09-11T06:53:17.165 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.127708.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62980.log 2024-09-11T06:53:17.165 INFO:teuthology.orchestra.run.smithi112.stderr: 54.0% -- replaced with /var/log/ceph/cluster1-client.mirror.127708.log.gz/var/log/ceph/cluster2-client.mirror.40975.log: 2024-09-11T06:53:17.165 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127754.log 2024-09-11T06:53:17.165 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.62980.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43906.log 2024-09-11T06:53:17.165 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.127754.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54573.log 2024-09-11T06:53:17.165 INFO:teuthology.orchestra.run.smithi112.stderr: 83.4% -- replaced with /var/log/ceph/cluster2-client.mirror.127754.log.gz 2024-09-11T06:53:17.165 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.43906.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97070.log 2024-09-11T06:53:17.165 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86853.log 2024-09-11T06:53:17.165 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.54573.log: /var/log/ceph/cluster2-client.mirror.97070.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65782.log 2024-09-11T06:53:17.166 INFO:teuthology.orchestra.run.smithi112.stderr: 84.5% -- replaced with /var/log/ceph/cluster2-client.mirror.97070.log.gz 2024-09-11T06:53:17.166 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36011.log 2024-09-11T06:53:17.166 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.86853.log: /var/log/ceph/cluster2-client.mirror.65782.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73995.log 2024-09-11T06:53:17.166 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70387.log 2024-09-11T06:53:17.166 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.36011.log: /var/log/ceph/cluster2-client.mirror.73995.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97490.log 2024-09-11T06:53:17.166 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.70387.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104583.log 2024-09-11T06:53:17.166 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.97490.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60775.log 2024-09-11T06:53:17.166 INFO:teuthology.orchestra.run.smithi112.stderr: 84.5% -- replaced with /var/log/ceph/cluster2-client.mirror.97490.log.gz 2024-09-11T06:53:17.166 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.104583.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87029.log 2024-09-11T06:53:17.167 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.60775.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39483.log 2024-09-11T06:53:17.167 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.87029.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.131042.log 2024-09-11T06:53:17.167 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86182.log 2024-09-11T06:53:17.167 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.39483.log: /var/log/ceph/cluster1-client.admin.131042.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75022.log 2024-09-11T06:53:17.167 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.131042.log.gz 2024-09-11T06:53:17.167 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.86182.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85864.log 2024-09-11T06:53:17.167 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48349.log 2024-09-11T06:53:17.167 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.75022.log: /var/log/ceph/cluster1-client.mirror.85864.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43565.log 2024-09-11T06:53:17.167 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.48349.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38220.log 2024-09-11T06:53:17.167 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.43565.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40138.log 2024-09-11T06:53:17.167 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.38220.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46795.log 2024-09-11T06:53:17.168 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.40138.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.29802.log 2024-09-11T06:53:17.168 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.46795.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112385.log 2024-09-11T06:53:17.168 INFO:teuthology.orchestra.run.smithi112.stderr: 96.5% -- replaced with /var/log/ceph/cluster2-client.mirror.104583.log.gz 2024-09-11T06:53:17.168 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.0.29802.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109947.log 2024-09-11T06:53:17.168 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.112385.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114895.log 2024-09-11T06:53:17.168 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.109947.log: 88.0%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36255.log 2024-09-11T06:53:17.168 INFO:teuthology.orchestra.run.smithi112.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.112385.log.gz 2024-09-11T06:53:17.168 INFO:teuthology.orchestra.run.smithi112.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.109947.log.gz 2024-09-11T06:53:17.168 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.114895.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.114895.log.gz 2024-09-11T06:53:17.168 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72817.log 2024-09-11T06:53:17.168 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.36255.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54329.log 2024-09-11T06:53:17.168 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.72817.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47601.log 2024-09-11T06:53:17.169 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96289.log 2024-09-11T06:53:17.169 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.54329.log: /var/log/ceph/cluster2-client.mirror.47601.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87390.log 2024-09-11T06:53:17.169 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.96289.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122364.log 2024-09-11T06:53:17.169 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.87390.log: 84.4% -- replaced with /var/log/ceph/cluster2-client.mirror.96289.log.gz 2024-09-11T06:53:17.169 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61767.log 2024-09-11T06:53:17.169 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106074.log 2024-09-11T06:53:17.169 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.122364.log: /var/log/ceph/cluster2-client.mirror.61767.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82150.log 2024-09-11T06:53:17.169 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74871.log 2024-09-11T06:53:17.169 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.106074.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.122364.log.gz 2024-09-11T06:53:17.169 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.82150.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116919.log 2024-09-11T06:53:17.169 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114331.log 2024-09-11T06:53:17.169 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.74871.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.106074.log.gz 2024-09-11T06:53:17.169 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.116919.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52085.log 2024-09-11T06:53:17.170 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44747.log 2024-09-11T06:53:17.170 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.114331.log: /var/log/ceph/cluster2-client.mirror.52085.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79820.log 2024-09-11T06:53:17.170 INFO:teuthology.orchestra.run.smithi112.stderr: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.114331.log.gz 2024-09-11T06:53:17.170 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47450.log 2024-09-11T06:53:17.170 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.44747.log: /var/log/ceph/cluster2-client.mirror.79820.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67060.log 2024-09-11T06:53:17.170 INFO:teuthology.orchestra.run.smithi112.stderr: 92.0% -- replaced with /var/log/ceph/cluster2-client.mirror.116919.log.gz 2024-09-11T06:53:17.170 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.47450.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126761.log 2024-09-11T06:53:17.170 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81852.log 2024-09-11T06:53:17.170 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.67060.log: /var/log/ceph/cluster1-client.mirror.126761.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58438.log 2024-09-11T06:53:17.170 INFO:teuthology.orchestra.run.smithi112.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.126761.log.gz 2024-09-11T06:53:17.170 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72592.log 2024-09-11T06:53:17.170 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.81852.log: /var/log/ceph/cluster2-client.mirror.58438.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53705.log 2024-09-11T06:53:17.171 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.72592.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123911.log 2024-09-11T06:53:17.171 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.53705.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128019.log 2024-09-11T06:53:17.171 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.123911.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58872.log 2024-09-11T06:53:17.171 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103391.log 2024-09-11T06:53:17.171 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.128019.log: /var/log/ceph/cluster2-client.mirror.58872.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.128019.log.gz 2024-09-11T06:53:17.171 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.27958.log 2024-09-11T06:53:17.171 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46853.log 2024-09-11T06:53:17.180 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.103391.log: /var/log/ceph/cluster2-client.admin.27958.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33453.log 2024-09-11T06:53:17.180 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.27958.log.gz 2024-09-11T06:53:17.180 INFO:teuthology.orchestra.run.smithi112.stderr: 90.8%/var/log/ceph/cluster2-client.mirror.46853.log: -- replaced with /var/log/ceph/cluster2-client.mirror.123911.log.gz 2024-09-11T06:53:17.180 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88635.log 2024-09-11T06:53:17.180 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77545.log 2024-09-11T06:53:17.180 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.33453.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.45681.log.gz 2024-09-11T06:53:17.180 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45623.log.gz/var/log/ceph/cluster2-client.mirror.88635.log: 2024-09-11T06:53:17.181 INFO:teuthology.orchestra.run.smithi112.stderr: 92.0% -- replaced with /var/log/ceph/cluster2-client.mirror.103391.log.gz 2024-09-11T06:53:17.181 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71135.log 2024-09-11T06:53:17.181 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.77545.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70294.log 2024-09-11T06:53:17.181 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.71135.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124539.log 2024-09-11T06:53:17.181 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40417.log 2024-09-11T06:53:17.181 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.70294.log: /var/log/ceph/cluster2-client.mirror.124539.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99013.log 2024-09-11T06:53:17.181 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57748.log 2024-09-11T06:53:17.181 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.40417.log: /var/log/ceph/cluster2-client.mirror.99013.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.29802.log 2024-09-11T06:53:17.181 INFO:teuthology.orchestra.run.smithi112.stderr: 84.4%gzip -- replaced with /var/log/ceph/cluster2-client.mirror.99013.log.gz 2024-09-11T06:53:17.181 INFO:teuthology.orchestra.run.smithi112.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104627.log 2024-09-11T06:53:17.181 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.57748.log: /var/log/ceph/cluster1-client.mirror.29802.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129464.log 2024-09-11T06:53:17.181 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82554.log 2024-09-11T06:53:17.181 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.104627.log: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.124539.log.gz 2024-09-11T06:53:17.182 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.129464.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56814.log 2024-09-11T06:53:17.182 INFO:teuthology.orchestra.run.smithi112.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.104627.log.gz 2024-09-11T06:53:17.182 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.82554.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69360.log 2024-09-11T06:53:17.182 INFO:teuthology.orchestra.run.smithi112.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.129464.log.gz 2024-09-11T06:53:17.182 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49000.log 2024-09-11T06:53:17.182 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.56814.log: /var/log/ceph/cluster2-client.mirror.69360.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35976.log 2024-09-11T06:53:17.182 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.49000.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124326.log 2024-09-11T06:53:17.182 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.35976.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76646.log 2024-09-11T06:53:17.182 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.124326.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36445.log 2024-09-11T06:53:17.182 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.76646.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45212.log 2024-09-11T06:53:17.182 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111898.log 2024-09-11T06:53:17.182 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.36445.log: /var/log/ceph/cluster2-client.mirror.45212.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115684.log 2024-09-11T06:53:17.182 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122222.log 2024-09-11T06:53:17.183 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.111898.log: /var/log/ceph/cluster2-client.mirror.115684.log: 90.8% -- replaced with /var/log/ceph/cluster2-client.mirror.124326.log.gz 2024-09-11T06:53:17.183 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30558.log 2024-09-11T06:53:17.183 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.122222.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.122222.log.gz 2024-09-11T06:53:17.183 INFO:teuthology.orchestra.run.smithi112.stderr: 92.0% -- replaced with /var/log/ceph/cluster2-client.mirror.115684.log.gz 2024-09-11T06:53:17.183 INFO:teuthology.orchestra.run.smithi112.stderr: 91.6% -- replaced with /var/log/ceph/cluster1-client.mirror.111898.log.gz 2024-09-11T06:53:17.183 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81104.log 2024-09-11T06:53:17.183 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58682.log 2024-09-11T06:53:17.183 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.30558.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32089.log 2024-09-11T06:53:17.183 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.81104.log: /var/log/ceph/cluster2-client.mirror.58682.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130318.log 2024-09-11T06:53:17.183 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55264.log 2024-09-11T06:53:17.183 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.32089.log: /var/log/ceph/cluster2-client.mirror.130318.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32182.log 2024-09-11T06:53:17.183 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.29501.log 2024-09-11T06:53:17.183 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.55264.log: /var/log/ceph/cluster2-client.mirror.32182.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.37972.log.gz 2024-09-11T06:53:17.184 INFO:teuthology.orchestra.run.smithi112.stderr: 87.9% 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.38065.log.gz -- replaced with /var/log/ceph/cluster2-client.mirror.38030.log.gz 2024-09-11T06:53:17.184 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-11T06:53:17.184 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79448.log 2024-09-11T06:53:17.184 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33395.log 2024-09-11T06:53:17.184 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.admin.29501.log: /var/log/ceph/cluster2-client.mirror.79448.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95361.log 2024-09-11T06:53:17.184 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.33395.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.43720.log.gz 2024-09-11T06:53:17.184 INFO:teuthology.orchestra.run.smithi112.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.33023.log.gz 2024-09-11T06:53:17.190 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87294.log 2024-09-11T06:53:17.190 INFO:teuthology.orchestra.run.smithi112.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.33116.log.gz 2024-09-11T06:53:17.190 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.95361.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78572.log 2024-09-11T06:53:17.190 INFO:teuthology.orchestra.run.smithi112.stderr: 84.6% -- replaced with /var/log/ceph/cluster2-client.mirror.95361.log.gz 2024-09-11T06:53:17.190 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71042.log 2024-09-11T06:53:17.190 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.87294.log: /var/log/ceph/cluster2-client.mirror.78572.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.43379.log.gz 2024-09-11T06:53:17.190 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35453.log 2024-09-11T06:53:17.190 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.52364.log.gz 2024-09-11T06:53:17.190 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113878.log 2024-09-11T06:53:17.191 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.71042.log: /var/log/ceph/cluster2-client.mirror.35453.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.54048.log 2024-09-11T06:53:17.191 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112848.log 2024-09-11T06:53:17.191 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.52492.log.gz 2024-09-11T06:53:17.191 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.113878.log: /var/log/ceph/cluster1-client.admin.54048.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64290.log 2024-09-11T06:53:17.191 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.112848.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.113878.log.gz 2024-09-11T06:53:17.191 INFO:teuthology.orchestra.run.smithi112.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.34015.log.gz 2024-09-11T06:53:17.191 INFO:teuthology.orchestra.run.smithi112.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.61333.log.gz 2024-09-11T06:53:17.191 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86565.log 2024-09-11T06:53:17.191 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.64290.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52178.log 2024-09-11T06:53:17.191 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.86565.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54946.log 2024-09-11T06:53:17.191 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.52178.log: 91.6% -- replaced with /var/log/ceph/cluster1-client.mirror.112848.log.gz 2024-09-11T06:53:17.191 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87367.log 2024-09-11T06:53:17.192 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50868.log 2024-09-11T06:53:17.192 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.54946.log: /var/log/ceph/cluster2-client.mirror.87367.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118954.log 2024-09-11T06:53:17.192 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114920.log 2024-09-11T06:53:17.192 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.50868.log: /var/log/ceph/cluster2-client.mirror.118954.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130680.log 2024-09-11T06:53:17.192 INFO:teuthology.orchestra.run.smithi112.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.118954.log.gz 2024-09-11T06:53:17.192 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.114920.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64604.log 2024-09-11T06:53:17.192 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.130680.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.130680.log.gz 2024-09-11T06:53:17.192 INFO:teuthology.orchestra.run.smithi112.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.31717.log.gz 2024-09-11T06:53:17.192 INFO:teuthology.orchestra.run.smithi112.stderr: 91.9% -- replaced with /var/log/ceph/cluster2-client.mirror.114920.log.gz 2024-09-11T06:53:17.192 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70817.log 2024-09-11T06:53:17.192 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.32182.log.gz 2024-09-11T06:53:17.196 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66983.log 2024-09-11T06:53:17.196 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.64604.log: gzip/var/log/ceph/cluster2-client.mirror.70817.log: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32275.log 2024-09-11T06:53:17.196 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.58066.log.gz 2024-09-11T06:53:17.196 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.66983.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87934.log 2024-09-11T06:53:17.196 INFO:teuthology.orchestra.run.smithi112.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.40975.log.gz 2024-09-11T06:53:17.196 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.32275.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68426.log 2024-09-11T06:53:17.196 INFO:teuthology.orchestra.run.smithi112.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.58124.log.gz 2024-09-11T06:53:17.196 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.87934.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.41072.log.gz 2024-09-11T06:53:17.196 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33174.log 2024-09-11T06:53:17.196 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.68426.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50120.log 2024-09-11T06:53:17.197 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.33174.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86475.log 2024-09-11T06:53:17.197 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.50120.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82519.log 2024-09-11T06:53:17.197 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.86475.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110778.log 2024-09-11T06:53:17.197 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46516.log 2024-09-11T06:53:17.197 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.82519.log: /var/log/ceph/cluster2-client.mirror.110778.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.34949.log.gz 2024-09-11T06:53:17.197 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.35042.log.gz 2024-09-11T06:53:17.197 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110325.log 2024-09-11T06:53:17.197 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.46516.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79072.log 2024-09-11T06:53:17.197 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.67678.log.gz 2024-09-11T06:53:17.197 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.110325.log: 88.1% 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.49035.log.gz 2024-09-11T06:53:17.197 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/cluster1-client.mirror.110325.log.gz 2024-09-11T06:53:17.197 INFO:teuthology.orchestra.run.smithi112.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.49000.log.gz 2024-09-11T06:53:17.198 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103720.log 2024-09-11T06:53:17.199 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65166.log 2024-09-11T06:53:17.199 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.79072.log: /var/log/ceph/cluster2-client.mirror.103720.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.103720.log.gz 2024-09-11T06:53:17.200 INFO:teuthology.orchestra.run.smithi112.stderr: 96.6% -- replaced with /var/log/ceph/cluster2-client.mirror.110778.log.gz 2024-09-11T06:53:17.200 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111853.log 2024-09-11T06:53:17.201 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87765.log 2024-09-11T06:53:17.201 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.65166.log: /var/log/ceph/cluster1-client.mirror.111853.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129526.log 2024-09-11T06:53:17.201 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.87765.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.111853.log.gz 2024-09-11T06:53:17.203 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81383.log 2024-09-11T06:53:17.203 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78700.log 2024-09-11T06:53:17.203 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.129526.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80490.log 2024-09-11T06:53:17.203 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.81383.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.129526.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69825.log 2024-09-11T06:53:17.203 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-11T06:53:17.203 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.78700.log: /var/log/ceph/cluster2-client.mirror.80490.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126221.log 2024-09-11T06:53:17.206 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.69825.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28690.log 2024-09-11T06:53:17.206 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.126221.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.82330.log 2024-09-11T06:53:17.206 INFO:teuthology.orchestra.run.smithi112.stderr: 88.2% 86.9% -- replaced with /var/log/ceph/cluster2-client.mirror.33453.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.126221.log.gz 2024-09-11T06:53:17.207 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-11T06:53:17.207 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.admin.28690.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39161.log 2024-09-11T06:53:17.207 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.28690.log.gz 2024-09-11T06:53:17.207 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61298.log 2024-09-11T06:53:17.207 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.admin.82330.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82802.log 2024-09-11T06:53:17.207 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.admin.39161.log: /var/log/ceph/cluster2-client.mirror.61298.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125816.log 2024-09-11T06:53:17.207 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53879.log 2024-09-11T06:53:17.207 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.82802.log: /var/log/ceph/cluster1-client.mirror.125816.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46419.log 2024-09-11T06:53:17.207 INFO:teuthology.orchestra.run.smithi112.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.125816.log.gz 2024-09-11T06:53:17.207 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.53879.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.74778.log.gz 2024-09-11T06:53:17.207 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42821.log 2024-09-11T06:53:17.211 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.46419.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77204.log 2024-09-11T06:53:17.212 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.42821.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35639.log 2024-09-11T06:53:17.212 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.77204.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70108.log 2024-09-11T06:53:17.212 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.35639.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73751.log 2024-09-11T06:53:17.212 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128346.log 2024-09-11T06:53:17.212 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.70108.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.76049.log.gz 2024-09-11T06:53:17.212 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.76142.log.gz 2024-09-11T06:53:17.212 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.73751.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123891.log 2024-09-11T06:53:17.212 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.51279.log.gz 2024-09-11T06:53:17.212 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.128346.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35511.log 2024-09-11T06:53:17.212 INFO:teuthology.orchestra.run.smithi112.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.128346.log.gz 2024-09-11T06:53:17.212 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.123891.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50589.log 2024-09-11T06:53:17.213 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.35511.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59151.log 2024-09-11T06:53:17.213 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.123891.log.gz 2024-09-11T06:53:17.213 INFO:teuthology.orchestra.run.smithi112.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.51372.log.gz 2024-09-11T06:53:17.213 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.50589.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109042.log 2024-09-11T06:53:17.213 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.59151.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82988.log 2024-09-11T06:53:17.213 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84515.log 2024-09-11T06:53:17.213 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.109042.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.77545.log.gz 2024-09-11T06:53:17.213 INFO:teuthology.orchestra.run.smithi112.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.77487.log.gz 2024-09-11T06:53:17.213 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.82988.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.71042.log.gz 2024-09-11T06:53:17.213 INFO:teuthology.orchestra.run.smithi112.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.70817.log.gz 2024-09-11T06:53:17.213 INFO:teuthology.orchestra.run.smithi112.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.79223.log.gz 2024-09-11T06:53:17.216 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.79448.log.gz 2024-09-11T06:53:17.216 INFO:teuthology.orchestra.run.smithi112.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.79506.log.gz 2024-09-11T06:53:17.216 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35077.log 2024-09-11T06:53:17.217 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.84515.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.46946.log.gz 2024-09-11T06:53:17.217 INFO:teuthology.orchestra.run.smithi112.stderr: 91.7% -- replaced with /var/log/ceph/cluster1-client.mirror.109042.log.gz 2024-09-11T06:53:17.217 INFO:teuthology.orchestra.run.smithi112.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.81852.log.gz 2024-09-11T06:53:17.217 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83267.log 2024-09-11T06:53:17.217 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.35077.log: gzip -5 --verbose 89.2% -- /var/log/ceph/cluster1-client.mirror.111520.log -- replaced with /var/log/ceph/cluster2-client.mirror.85077.log.gz 2024-09-11T06:53:17.217 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-11T06:53:17.217 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.83267.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37844.log 2024-09-11T06:53:17.217 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44782.log 2024-09-11T06:53:17.217 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.111520.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.80339.log.gz 2024-09-11T06:53:17.217 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.37844.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59430.log 2024-09-11T06:53:17.218 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.44782.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92068.log 2024-09-11T06:53:17.218 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.59430.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85911.log 2024-09-11T06:53:17.218 INFO:teuthology.orchestra.run.smithi112.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48570.log.gz 2024-09-11T06:53:17.218 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.92068.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61674.log 2024-09-11T06:53:17.218 INFO:teuthology.orchestra.run.smithi112.stderr: 91.7% -- replaced with /var/log/ceph/cluster1-client.mirror.111520.log.gz 2024-09-11T06:53:17.218 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.85911.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42032.log 2024-09-11T06:53:17.218 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.61674.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.29432.log 2024-09-11T06:53:17.220 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.42032.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125929.log 2024-09-11T06:53:17.220 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114398.log 2024-09-11T06:53:17.220 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.29432.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60647.log 2024-09-11T06:53:17.220 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.125929.log: /var/log/ceph/cluster2-client.mirror.114398.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.125929.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69546.log 2024-09-11T06:53:17.220 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-11T06:53:17.225 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.60647.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88260.log 2024-09-11T06:53:17.225 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.69546.log: 95.2% -- replaced with /var/log/ceph/cluster2-client.mirror.129797.log.gz 2024-09-11T06:53:17.225 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80397.log 2024-09-11T06:53:17.225 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65596.log 2024-09-11T06:53:17.226 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.88260.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.21612.log 2024-09-11T06:53:17.226 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.80397.log: /var/log/ceph/cluster2-client.mirror.65596.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77917.log 2024-09-11T06:53:17.226 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121131.log 2024-09-11T06:53:17.226 INFO:teuthology.orchestra.run.smithi112.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.49283.log.gz 2024-09-11T06:53:17.226 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.77917.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.29016.log 2024-09-11T06:53:17.226 INFO:teuthology.orchestra.run.smithi112.stderr: 88.4% -- replaced with /var/log/ceph/cluster1-client.admin.54048.log.gz 2024-09-11T06:53:17.226 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.68577.log.gz 2024-09-11T06:53:17.226 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.121131.log: /var/log/ceph/ceph.tmp-client.admin.21612.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119340.log 2024-09-11T06:53:17.226 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.21612.log.gz 2024-09-11T06:53:17.226 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.70108.log.gz 2024-09-11T06:53:17.226 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.admin.29016.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.29016.log.gz 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.121131.log.gz 2024-09-11T06:53:17.227 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-11T06:53:17.227 INFO:teuthology.orchestra.run.smithi112.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.64007.log.gz 2024-09-11T06:53:17.229 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116961.log 2024-09-11T06:53:17.229 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.119340.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111134.log 2024-09-11T06:53:17.229 INFO:teuthology.orchestra.run.smithi112.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.119340.log.gz 2024-09-11T06:53:17.229 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose --/var/log/ceph/cluster2-client.mirror.116961.log: /var/log/ceph/cluster2-client.mirror.53298.log 2024-09-11T06:53:17.229 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.62736.log.gz 2024-09-11T06:53:17.229 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.111134.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60213.log 2024-09-11T06:53:17.229 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.62922.log.gz 2024-09-11T06:53:17.229 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.53298.log: 96.9% -- replaced with /var/log/ceph/cluster2-client.mirror.114398.log.gz 2024-09-11T06:53:17.229 INFO:teuthology.orchestra.run.smithi112.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.64290.log.gz 2024-09-11T06:53:17.230 INFO:teuthology.orchestra.run.smithi112.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.64604.log.gz 2024-09-11T06:53:17.230 INFO:teuthology.orchestra.run.smithi112.stderr: 92.0% -- replaced with /var/log/ceph/cluster1-client.mirror.111134.log.gz 2024-09-11T06:53:17.230 INFO:teuthology.orchestra.run.smithi112.stderr: 84.6% -- replaced with /var/log/ceph/cluster2-client.mirror.87294.log.gz 2024-09-11T06:53:17.230 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126919.log 2024-09-11T06:53:17.231 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.60213.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48535.log 2024-09-11T06:53:17.231 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.126919.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116662.log 2024-09-11T06:53:17.232 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.48535.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.57845.log.gz 2024-09-11T06:53:17.232 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47353.log 2024-09-11T06:53:17.232 INFO:teuthology.orchestra.run.smithi112.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.126919.log.gz 2024-09-11T06:53:17.232 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.116662.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128169.log 2024-09-11T06:53:17.232 INFO:teuthology.orchestra.run.smithi112.stderr: 96.3% 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.116662.log.gz 2024-09-11T06:53:17.233 INFO:teuthology.orchestra.run.smithi112.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.116961.log.gz 2024-09-11T06:53:17.233 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.47353.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67018.log 2024-09-11T06:53:17.234 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.128169.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66658.log 2024-09-11T06:53:17.234 INFO:teuthology.orchestra.run.smithi112.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.66983.log.gz 2024-09-11T06:53:17.234 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.67018.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103819.log 2024-09-11T06:53:17.235 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.66658.log: 19.0% -- replaced with /var/log/ceph/cluster2-client.mirror.0.29769.log.gz 2024-09-11T06:53:17.235 INFO:teuthology.orchestra.run.smithi112.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.72406.log.gz 2024-09-11T06:53:17.235 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.103819.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122739.log 2024-09-11T06:53:17.235 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32368.log 2024-09-11T06:53:17.236 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.42410.log.gz 2024-09-11T06:53:17.236 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51337.log 2024-09-11T06:53:17.236 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.122739.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.84515.log.gz 2024-09-11T06:53:17.236 INFO:teuthology.orchestra.run.smithi112.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.73003.log.gz 2024-09-11T06:53:17.236 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.32368.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.122739.log.gz 2024-09-11T06:53:17.237 INFO:teuthology.orchestra.run.smithi112.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.73096.log.gz 2024-09-11T06:53:17.237 INFO:teuthology.orchestra.run.smithi112.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.86899.log.gz 2024-09-11T06:53:17.238 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35697.log 2024-09-11T06:53:17.238 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.51337.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105280.log 2024-09-11T06:53:17.239 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.35697.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75336.log 2024-09-11T06:53:17.240 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.105280.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.105280.log.gz 2024-09-11T06:53:17.240 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82612.log 2024-09-11T06:53:17.241 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.75336.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124568.log 2024-09-11T06:53:17.241 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125501.log 2024-09-11T06:53:17.242 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.82612.log: /var/log/ceph/cluster2-client.mirror.124568.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38627.log 2024-09-11T06:53:17.242 INFO:teuthology.orchestra.run.smithi112.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.124568.log.gz 2024-09-11T06:53:17.243 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.125501.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121946.log 2024-09-11T06:53:17.243 INFO:teuthology.orchestra.run.smithi112.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.125501.log.gz/var/log/ceph/cluster2-client.mirror.38627.log: 2024-09-11T06:53:17.243 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69418.log 2024-09-11T06:53:17.244 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.121946.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.121946.log.gz 2024-09-11T06:53:17.244 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64042.log 2024-09-11T06:53:17.245 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.69418.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65131.log 2024-09-11T06:53:17.245 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.64042.log: 95.8% -- replaced with /var/log/ceph/cluster2-client.mirror.97513.log.gz 2024-09-11T06:53:17.245 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79541.log 2024-09-11T06:53:17.246 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.65131.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36290.log 2024-09-11T06:53:17.246 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60926.log 2024-09-11T06:53:17.247 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.79541.log: /var/log/ceph/cluster2-client.mirror.36290.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109183.log 2024-09-11T06:53:17.247 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29279.log 2024-09-11T06:53:17.248 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.60926.log: /var/log/ceph/cluster1-client.mirror.109183.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.109183.log.gz 2024-09-11T06:53:17.248 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128081.log 2024-09-11T06:53:17.248 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.admin.29279.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45716.log 2024-09-11T06:53:17.249 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44875.log 2024-09-11T06:53:17.249 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.128081.log: /var/log/ceph/cluster2-client.mirror.45716.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74181.log 2024-09-11T06:53:17.249 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119305.log 2024-09-11T06:53:17.250 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.44875.log: /var/log/ceph/cluster2-client.mirror.74181.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130523.log 2024-09-11T06:53:17.250 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.119305.log: 91.8% -- replaced with /var/log/ceph/cluster2-client.mirror.128081.log.gz 2024-09-11T06:53:17.251 INFO:teuthology.orchestra.run.smithi112.stderr: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.119305.log.gz 2024-09-11T06:53:17.251 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31868.log 2024-09-11T06:53:17.251 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.130523.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127346.log 2024-09-11T06:53:17.252 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.31868.log: 91.8% -- replaced with /var/log/ceph/cluster2-client.mirror.130523.log.gz 2024-09-11T06:53:17.254 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103054.log 2024-09-11T06:53:17.255 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70631.log 2024-09-11T06:53:17.255 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.127346.log: /var/log/ceph/cluster2-client.mirror.103054.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112350.log 2024-09-11T06:53:17.255 INFO:teuthology.orchestra.run.smithi112.stderr: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.127346.log.gz/var/log/ceph/cluster2-client.mirror.70631.log: 2024-09-11T06:53:17.258 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36759.log 2024-09-11T06:53:17.259 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.112350.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.112350.log.gz 2024-09-11T06:53:17.259 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116448.log 2024-09-11T06:53:17.260 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.36759.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28516.log 2024-09-11T06:53:17.260 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.116448.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59992.log 2024-09-11T06:53:17.261 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.admin.28516.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.28516.log.gz 2024-09-11T06:53:17.261 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34705.log 2024-09-11T06:53:17.262 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.59992.log: 91.5% -- replaced with /var/log/ceph/cluster2-client.mirror.116448.log.gz 2024-09-11T06:53:17.262 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52736.log 2024-09-11T06:53:17.262 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.34705.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104961.log 2024-09-11T06:53:17.263 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.52736.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107023.log 2024-09-11T06:53:17.263 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123050.log 2024-09-11T06:53:17.265 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.104961.log: /var/log/ceph/cluster2-client.mirror.107023.log: 92.1% -- replaced with /var/log/ceph/cluster2-client.mirror.107023.log.gz 2024-09-11T06:53:17.265 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41444.log 2024-09-11T06:53:17.266 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.123050.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.123050.log.gz 2024-09-11T06:53:17.267 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31527.log 2024-09-11T06:53:17.267 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.41444.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111621.log 2024-09-11T06:53:17.268 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.31527.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29671.log 2024-09-11T06:53:17.269 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.111621.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.111621.log.gz 2024-09-11T06:53:17.270 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74030.log 2024-09-11T06:53:17.270 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.admin.29671.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29671.log.gz 2024-09-11T06:53:17.270 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120864.log 2024-09-11T06:53:17.271 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.74030.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46008.log 2024-09-11T06:53:17.272 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.120864.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.120864.log.gz 2024-09-11T06:53:17.272 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.46008.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.46516.log.gz 2024-09-11T06:53:17.273 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114734.log 2024-09-11T06:53:17.273 INFO:teuthology.orchestra.run.smithi112.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.56535.log.gz 2024-09-11T06:53:17.274 INFO:teuthology.orchestra.run.smithi112.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.32368.log.gz 2024-09-11T06:53:17.274 INFO:teuthology.orchestra.run.smithi112.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.73751.log.gz 2024-09-11T06:53:17.274 INFO:teuthology.orchestra.run.smithi112.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.35077.log.gz 2024-09-11T06:53:17.274 INFO:teuthology.orchestra.run.smithi112.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.35697.log.gz 2024-09-11T06:53:17.278 INFO:teuthology.orchestra.run.smithi112.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.50589.log.gz 2024-09-11T06:53:17.278 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.51337.log.gz 2024-09-11T06:53:17.282 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32240.log 2024-09-11T06:53:17.283 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.114734.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47353.log.gz 2024-09-11T06:53:17.284 INFO:teuthology.orchestra.run.smithi112.stderr: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.114734.log.gz 2024-09-11T06:53:17.285 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77262.log 2024-09-11T06:53:17.285 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.32240.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44185.log 2024-09-11T06:53:17.286 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.77262.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80010.log 2024-09-11T06:53:17.288 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.44185.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86024.log 2024-09-11T06:53:17.289 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.80010.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128293.log 2024-09-11T06:53:17.290 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.86024.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57562.log 2024-09-11T06:53:17.292 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.128293.log: 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.128293.log.gz 2024-09-11T06:53:17.292 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47543.log 2024-09-11T06:53:17.293 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.57562.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112039.log 2024-09-11T06:53:17.294 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.47543.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63949.log 2024-09-11T06:53:17.295 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.112039.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.112039.log.gz 2024-09-11T06:53:17.295 INFO:teuthology.orchestra.run.smithi112.stderr: 88.4% -- replaced with /var/log/ceph/cluster1-client.mirror.80010.log.gz 2024-09-11T06:53:17.295 INFO:teuthology.orchestra.run.smithi112.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.45835.log.gz 2024-09-11T06:53:17.296 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster1-client.admin.80204.log.gz 2024-09-11T06:53:17.296 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108686.log 2024-09-11T06:53:17.297 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.63949.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37321.log 2024-09-11T06:53:17.298 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.108686.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.83267.log.gz 2024-09-11T06:53:17.299 INFO:teuthology.orchestra.run.smithi112.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48535.log.gz 2024-09-11T06:53:17.300 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.52678.log.gz 2024-09-11T06:53:17.301 INFO:teuthology.orchestra.run.smithi112.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.52736.log.gz 2024-09-11T06:53:17.301 INFO:teuthology.orchestra.run.smithi112.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.60213.log.gz 2024-09-11T06:53:17.301 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.63763.log.gz 2024-09-11T06:53:17.301 INFO:teuthology.orchestra.run.smithi112.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.63949.log.gz 2024-09-11T06:53:17.301 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.65131.log.gz 2024-09-11T06:53:17.302 INFO:teuthology.orchestra.run.smithi112.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.66344.log.gz 2024-09-11T06:53:17.304 INFO:teuthology.orchestra.run.smithi112.stderr: 96.2% -- replaced with /var/log/ceph/cluster2-client.mirror.108686.log.gz 2024-09-11T06:53:17.304 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41781.log 2024-09-11T06:53:17.305 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.37321.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33209.log 2024-09-11T06:53:17.305 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.69046.log.gz 2024-09-11T06:53:17.306 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.41781.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37879.log 2024-09-11T06:53:17.307 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.33209.log: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.94659.log.gz 2024-09-11T06:53:17.308 INFO:teuthology.orchestra.run.smithi112.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.59806.log.gz 2024-09-11T06:53:17.308 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28966.log 2024-09-11T06:53:17.308 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43437.log 2024-09-11T06:53:17.313 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.37879.log: /var/log/ceph/cluster1-client.admin.28966.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28966.log.gz 2024-09-11T06:53:17.314 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128143.log 2024-09-11T06:53:17.314 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70166.log 2024-09-11T06:53:17.314 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.43437.log: /var/log/ceph/cluster2-client.mirror.128143.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74367.log 2024-09-11T06:53:17.314 INFO:teuthology.orchestra.run.smithi112.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.128143.log.gz 2024-09-11T06:53:17.314 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40568.log 2024-09-11T06:53:17.314 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.70166.log: /var/log/ceph/cluster2-client.mirror.74367.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52829.log 2024-09-11T06:53:17.314 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83360.log 2024-09-11T06:53:17.314 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.40568.log: /var/log/ceph/cluster2-client.mirror.52829.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60306.log 2024-09-11T06:53:17.314 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68147.log 2024-09-11T06:53:17.315 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.83360.log: /var/log/ceph/cluster2-client.mirror.60306.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85376.log 2024-09-11T06:53:17.315 INFO:teuthology.orchestra.run.smithi112.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.67018.log.gz 2024-09-11T06:53:17.315 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.68147.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81290.log 2024-09-11T06:53:17.315 INFO:teuthology.orchestra.run.smithi112.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.41781.log.gz 2024-09-11T06:53:17.316 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.85376.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65410.log 2024-09-11T06:53:17.316 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77859.log 2024-09-11T06:53:17.317 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.81290.log: /var/log/ceph/cluster2-client.mirror.65410.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67643.log 2024-09-11T06:53:17.317 INFO:teuthology.orchestra.run.smithi112.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.81139.log.gz 2024-09-11T06:53:17.317 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.77859.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86679.log 2024-09-11T06:53:17.318 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.67643.log: 92.8% -- replaced with /var/log/ceph/cluster1-client.mirror.86135.log.gz 2024-09-11T06:53:17.319 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47039.log 2024-09-11T06:53:17.320 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124590.log 2024-09-11T06:53:17.320 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.86679.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121151.log 2024-09-11T06:53:17.321 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.47039.log: /var/log/ceph/cluster2-client.mirror.124590.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.124590.log.gz 2024-09-11T06:53:17.321 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.46164.log 2024-09-11T06:53:17.323 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.121151.log: 90.8% -- replaced with /var/log/ceph/cluster2-client.mirror.121151.log.gz 2024-09-11T06:53:17.324 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126154.log 2024-09-11T06:53:17.325 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103012.log 2024-09-11T06:53:17.325 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.admin.46164.log: /var/log/ceph/cluster1-client.mirror.126154.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.126154.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42281.log 2024-09-11T06:53:17.325 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-11T06:53:17.325 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.103012.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44092.log 2024-09-11T06:53:17.327 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.admin.42281.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51523.log 2024-09-11T06:53:17.327 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.44092.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.25335.log 2024-09-11T06:53:17.327 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.51523.log: 92.0% -- replaced with /var/log/ceph/cluster2-client.mirror.103012.log.gz 2024-09-11T06:53:17.327 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36480.log 2024-09-11T06:53:17.329 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122859.log 2024-09-11T06:53:17.329 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.admin.25335.log: /var/log/ceph/cluster2-client.mirror.36480.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.25335.log.gz 2024-09-11T06:53:17.329 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86362.log 2024-09-11T06:53:17.330 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102170.log 2024-09-11T06:53:17.331 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.122859.log: /var/log/ceph/cluster1-client.mirror.86362.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129170.log 2024-09-11T06:53:17.331 INFO:teuthology.orchestra.run.smithi112.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.122859.log.gz/var/log/ceph/cluster1-client.mirror.102170.log: 2024-09-11T06:53:17.331 INFO:teuthology.orchestra.run.smithi112.stderr: 1.3% -- replaced with /var/log/ceph/cluster1-client.mirror.102170.log.gz 2024-09-11T06:53:17.331 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46888.log 2024-09-11T06:53:17.331 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119397.log 2024-09-11T06:53:17.332 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.129170.log: /var/log/ceph/cluster2-client.mirror.46888.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.129170.log.gz 2024-09-11T06:53:17.332 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124195.log 2024-09-11T06:53:17.333 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66100.log 2024-09-11T06:53:17.335 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.119397.log: /var/log/ceph/cluster2-client.mirror.124195.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.119397.log.gz 2024-09-11T06:53:17.335 INFO:teuthology.orchestra.run.smithi112.stderr: 90.8% -- replaced with /var/log/ceph/cluster2-client.mirror.124195.log.gz 2024-09-11T06:53:17.335 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113970.log 2024-09-11T06:53:17.335 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41258.log 2024-09-11T06:53:17.336 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.66100.log: /var/log/ceph/cluster2-client.mirror.113970.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37751.log 2024-09-11T06:53:17.336 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.41258.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122759.log 2024-09-11T06:53:17.338 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.37751.log: 91.8% -- replaced with /var/log/ceph/cluster2-client.mirror.113970.log.gz 2024-09-11T06:53:17.338 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85170.log 2024-09-11T06:53:17.338 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123163.log 2024-09-11T06:53:17.340 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.122759.log: /var/log/ceph/cluster2-client.mirror.85170.log: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.122759.log.gz 2024-09-11T06:53:17.340 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54888.log 2024-09-11T06:53:17.342 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.123163.log: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.123163.log.gz 2024-09-11T06:53:17.343 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112442.log 2024-09-11T06:53:17.343 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86225.log 2024-09-11T06:53:17.344 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.54888.log: /var/log/ceph/cluster2-client.mirror.112442.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71007.log 2024-09-11T06:53:17.344 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.86225.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118541.log 2024-09-11T06:53:17.346 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.71007.log: 92.0% -- replaced with /var/log/ceph/cluster2-client.mirror.112442.log.gz 2024-09-11T06:53:17.346 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46384.log 2024-09-11T06:53:17.346 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28916.log 2024-09-11T06:53:17.346 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.118541.log: /var/log/ceph/cluster2-client.mirror.46384.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.118541.log.gz 2024-09-11T06:53:17.346 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.131495.log 2024-09-11T06:53:17.347 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62046.log 2024-09-11T06:53:17.348 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.admin.28916.log: /var/log/ceph/cluster2-client.admin.131495.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28916.log.gz 2024-09-11T06:53:17.348 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.131495.log.gz 2024-09-11T06:53:17.348 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114376.log 2024-09-11T06:53:17.349 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127821.log 2024-09-11T06:53:17.349 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.62046.log: /var/log/ceph/cluster1-client.mirror.114376.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118155.log 2024-09-11T06:53:17.350 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.127821.log: 91.4% -- replaced with /var/log/ceph/cluster1-client.mirror.114376.log.gz 2024-09-11T06:53:17.350 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.46419.log.gz 2024-09-11T06:53:17.351 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.46888.log.gz 2024-09-11T06:53:17.351 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.46384.log.gz 2024-09-11T06:53:17.351 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.57132.log.gz 2024-09-11T06:53:17.357 INFO:teuthology.orchestra.run.smithi112.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.58814.log.gz 2024-09-11T06:53:17.357 INFO:teuthology.orchestra.run.smithi112.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.32240.log.gz 2024-09-11T06:53:17.359 INFO:teuthology.orchestra.run.smithi112.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.41630.log.gz 2024-09-11T06:53:17.359 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.73902.log.gz 2024-09-11T06:53:17.360 INFO:teuthology.orchestra.run.smithi112.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.67643.log.gz 2024-09-11T06:53:17.360 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111022.log 2024-09-11T06:53:17.360 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.118155.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.118155.log.gz 2024-09-11T06:53:17.361 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82073.log 2024-09-11T06:53:17.361 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.111022.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.111022.log.gz 2024-09-11T06:53:17.362 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42914.log 2024-09-11T06:53:17.362 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.82073.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50531.log 2024-09-11T06:53:17.363 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.42914.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54260.log 2024-09-11T06:53:17.364 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.50531.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40382.log 2024-09-11T06:53:17.364 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.54260.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107344.log 2024-09-11T06:53:17.365 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.40382.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85821.log 2024-09-11T06:53:17.369 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.107344.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.107344.log.gz 2024-09-11T06:53:17.369 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60682.log 2024-09-11T06:53:17.370 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.85821.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69325.log 2024-09-11T06:53:17.370 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.60682.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87662.log 2024-09-11T06:53:17.371 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.69325.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126109.log 2024-09-11T06:53:17.373 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.87662.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44127.log 2024-09-11T06:53:17.373 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.43437.log.gz 2024-09-11T06:53:17.373 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.126109.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.126109.log.gz 2024-09-11T06:53:17.374 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.50531.log.gz 2024-09-11T06:53:17.374 INFO:teuthology.orchestra.run.smithi112.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.43158.log.gz 2024-09-11T06:53:17.374 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58965.log 2024-09-11T06:53:17.375 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.44127.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.77262.log.gz 2024-09-11T06:53:17.375 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.77859.log.gz 2024-09-11T06:53:17.375 INFO:teuthology.orchestra.run.smithi112.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.82073.log.gz 2024-09-11T06:53:17.376 INFO:teuthology.orchestra.run.smithi112.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.45793.log.gz 2024-09-11T06:53:17.376 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.79974.log.gz 2024-09-11T06:53:17.376 INFO:teuthology.orchestra.run.smithi112.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.68147.log.gz 2024-09-11T06:53:17.377 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.70073.log.gz 2024-09-11T06:53:17.377 INFO:teuthology.orchestra.run.smithi112.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.70166.log.gz 2024-09-11T06:53:17.378 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.53426.log.gz 2024-09-11T06:53:17.378 INFO:teuthology.orchestra.run.smithi112.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.63914.log.gz 2024-09-11T06:53:17.378 INFO:teuthology.orchestra.run.smithi112.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.64755.log.gz 2024-09-11T06:53:17.378 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.41723.log.gz 2024-09-11T06:53:17.378 INFO:teuthology.orchestra.run.smithi112.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.42914.log.gz 2024-09-11T06:53:17.379 INFO:teuthology.orchestra.run.smithi112.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.84670.log.gz 2024-09-11T06:53:17.380 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50155.log 2024-09-11T06:53:17.380 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.58965.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66003.log 2024-09-11T06:53:17.381 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.50155.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42596.log 2024-09-11T06:53:17.382 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.66003.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37658.log 2024-09-11T06:53:17.383 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.42596.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126874.log 2024-09-11T06:53:17.384 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.37658.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87542.log 2024-09-11T06:53:17.384 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.126874.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.126874.log.gz 2024-09-11T06:53:17.385 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106166.log 2024-09-11T06:53:17.385 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.87542.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.68019.log.gz 2024-09-11T06:53:17.386 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120203.log 2024-09-11T06:53:17.387 INFO:teuthology.orchestra.run.smithi112.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.66658.log.gz 2024-09-11T06:53:17.390 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.106166.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.37844.log.gz 2024-09-11T06:53:17.390 INFO:teuthology.orchestra.run.smithi112.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.37879.log.gz 2024-09-11T06:53:17.391 INFO:teuthology.orchestra.run.smithi112.stderr: 91.9% -- replaced with /var/log/ceph/cluster2-client.mirror.106166.log.gz 2024-09-11T06:53:17.392 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112771.log 2024-09-11T06:53:17.399 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.120203.log: 96.5% 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.58031.log.gz 2024-09-11T06:53:17.399 INFO:teuthology.orchestra.run.smithi112.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.120203.log.gz 2024-09-11T06:53:17.399 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60554.log 2024-09-11T06:53:17.400 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.112771.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.112771.log.gz 2024-09-11T06:53:17.401 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127076.log 2024-09-11T06:53:17.401 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.60554.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120725.log 2024-09-11T06:53:17.402 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.127076.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.127076.log.gz 2024-09-11T06:53:17.402 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125338.log 2024-09-11T06:53:17.404 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.120725.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.37131.log.gz 2024-09-11T06:53:17.404 INFO:teuthology.orchestra.run.smithi112.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.37321.log.gz 2024-09-11T06:53:17.404 INFO:teuthology.orchestra.run.smithi112.stderr: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.103819.log.gz 2024-09-11T06:53:17.405 INFO:teuthology.orchestra.run.smithi112.stderr: 91.6% -- replaced with /var/log/ceph/cluster2-client.mirror.120725.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68798.log 2024-09-11T06:53:17.405 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-11T06:53:17.406 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.125338.log: 93.1% -- replaced with /var/log/ceph/cluster2-client.mirror.86252.log.gz 2024-09-11T06:53:17.406 INFO:teuthology.orchestra.run.smithi112.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.125338.log.gz 2024-09-11T06:53:17.406 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63015.log 2024-09-11T06:53:17.407 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52213.log 2024-09-11T06:53:17.407 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.68798.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.44782.log.gz 2024-09-11T06:53:17.407 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.44840.log.gz 2024-09-11T06:53:17.408 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.63015.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.44968.log.gz 2024-09-11T06:53:17.408 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.39762.log.gz 2024-09-11T06:53:17.408 INFO:teuthology.orchestra.run.smithi112.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.35511.log.gz 2024-09-11T06:53:17.408 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58496.log 2024-09-11T06:53:17.408 INFO:teuthology.orchestra.run.smithi112.stderr: 93.2% -- replaced with /var/log/ceph/cluster1-client.mirror.86495.log.gz 2024-09-11T06:53:17.408 INFO:teuthology.orchestra.run.smithi112.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.44185.log.gz 2024-09-11T06:53:17.408 INFO:teuthology.orchestra.run.smithi112.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.44127.log.gz 2024-09-11T06:53:17.409 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.52213.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123305.log 2024-09-11T06:53:17.409 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.58496.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28508.log 2024-09-11T06:53:17.410 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.123305.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28991.log 2024-09-11T06:53:17.410 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.admin.28508.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.28508.log.gz 2024-09-11T06:53:17.411 INFO:teuthology.orchestra.run.smithi112.stderr: 90.8% -- replaced with /var/log/ceph/cluster2-client.mirror.123305.log.gz 2024-09-11T06:53:17.411 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118919.log 2024-09-11T06:53:17.412 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.admin.28991.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107922.log 2024-09-11T06:53:17.412 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28991.log.gz 2024-09-11T06:53:17.413 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28716.log 2024-09-11T06:53:17.413 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.118919.log: /var/log/ceph/cluster2-client.mirror.107922.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.40006.log.gz 2024-09-11T06:53:17.413 INFO:teuthology.orchestra.run.smithi112.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.118919.log.gz 2024-09-11T06:53:17.413 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122244.log 2024-09-11T06:53:17.414 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.admin.28716.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.28716.log.gz 2024-09-11T06:53:17.414 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83957.log 2024-09-11T06:53:17.414 INFO:teuthology.orchestra.run.smithi112.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.107922.log.gz 2024-09-11T06:53:17.414 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49221.log 2024-09-11T06:53:17.415 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.122244.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.34050.log.gz 2024-09-11T06:53:17.416 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.83957.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28594.log 2024-09-11T06:53:17.416 INFO:teuthology.orchestra.run.smithi112.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.58496.log.gz 2024-09-11T06:53:17.416 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.58624.log.gz 2024-09-11T06:53:17.416 INFO:teuthology.orchestra.run.smithi112.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.122244.log.gz 2024-09-11T06:53:17.416 INFO:teuthology.orchestra.run.smithi112.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.58965.log.gz 2024-09-11T06:53:17.417 INFO:teuthology.orchestra.run.smithi112.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.61430.log.gz 2024-09-11T06:53:17.417 INFO:teuthology.orchestra.run.smithi112.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.31620.log.gz 2024-09-11T06:53:17.417 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.49221.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.32054.log.gz 2024-09-11T06:53:17.417 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53577.log 2024-09-11T06:53:17.418 INFO:teuthology.orchestra.run.smithi112.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.41165.log.gz 2024-09-11T06:53:17.418 INFO:teuthology.orchestra.run.smithi112.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.37937.log.gz 2024-09-11T06:53:17.418 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.admin.28594.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.28594.log.gz 2024-09-11T06:53:17.418 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.50155.log.gz 2024-09-11T06:53:17.418 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38936.log 2024-09-11T06:53:17.419 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.53577.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48101.log 2024-09-11T06:53:17.419 INFO:teuthology.orchestra.run.smithi112.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.65968.log.gz 2024-09-11T06:53:17.419 INFO:teuthology.orchestra.run.smithi112.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.66003.log.gz 2024-09-11T06:53:17.419 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.38936.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32895.log 2024-09-11T06:53:17.420 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.67736.log.gz 2024-09-11T06:53:17.420 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.32895.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83736.log 2024-09-11T06:53:17.420 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.45947.log.gz 2024-09-11T06:53:17.420 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.48101.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45026.log 2024-09-11T06:53:17.420 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.83736.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31678.log 2024-09-11T06:53:17.421 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.45026.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73375.log 2024-09-11T06:53:17.421 INFO:teuthology.orchestra.run.smithi112.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.82519.log.gz 2024-09-11T06:53:17.421 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.31678.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44499.log 2024-09-11T06:53:17.422 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.73375.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108664.log 2024-09-11T06:53:17.422 INFO:teuthology.orchestra.run.smithi112.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.82988.log.gz 2024-09-11T06:53:17.423 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.44499.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.82038.log.gz 2024-09-11T06:53:17.423 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73189.log 2024-09-11T06:53:17.424 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.108664.log: 91.2% -- replaced with /var/log/ceph/cluster1-client.mirror.108664.log.gz 2024-09-11T06:53:17.425 INFO:teuthology.orchestra.run.smithi112.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.66565.log.gz 2024-09-11T06:53:17.425 INFO:teuthology.orchestra.run.smithi112.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.83957.log.gz 2024-09-11T06:53:17.425 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55543.log 2024-09-11T06:53:17.426 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.73189.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106657.log 2024-09-11T06:53:17.426 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.55543.log: 92.9% -- replaced with /var/log/ceph/cluster2-client.mirror.53859.log.gz 2024-09-11T06:53:17.427 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.71193.log.gz 2024-09-11T06:53:17.428 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104098.log 2024-09-11T06:53:17.428 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.48942.log.gz 2024-09-11T06:53:17.429 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.106657.log: 91.9% -- replaced with /var/log/ceph/cluster1-client.mirror.106657.log.gz 2024-09-11T06:53:17.430 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81794.log 2024-09-11T06:53:17.430 INFO:teuthology.orchestra.run.smithi112.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.33678.log.gz 2024-09-11T06:53:17.430 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.104098.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.62364.log.gz 2024-09-11T06:53:17.430 INFO:teuthology.orchestra.run.smithi112.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.104098.log.gz 2024-09-11T06:53:17.430 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126581.log 2024-09-11T06:53:17.431 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.81794.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65817.log 2024-09-11T06:53:17.431 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.126581.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.126581.log.gz 2024-09-11T06:53:17.431 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62232.log 2024-09-11T06:53:17.432 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.65817.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115140.log 2024-09-11T06:53:17.433 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.62232.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108997.log 2024-09-11T06:53:17.433 INFO:teuthology.orchestra.run.smithi112.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.53484.log.gz 2024-09-11T06:53:17.433 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.115140.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.60554.log.gz 2024-09-11T06:53:17.435 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.65317.log.gz 2024-09-11T06:53:17.435 INFO:teuthology.orchestra.run.smithi112.stderr: 91.7% -- replaced with /var/log/ceph/cluster1-client.mirror.115140.log.gz 2024-09-11T06:53:17.435 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.66193.log.gz 2024-09-11T06:53:17.435 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83395.log 2024-09-11T06:53:17.435 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.108997.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.63015.log.gz 2024-09-11T06:53:17.436 INFO:teuthology.orchestra.run.smithi112.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.108997.log.gz 2024-09-11T06:53:17.437 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84856.log 2024-09-11T06:53:17.437 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.83395.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.72162.log.gz 2024-09-11T06:53:17.437 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.90818.log 2024-09-11T06:53:17.437 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.84856.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102342.log 2024-09-11T06:53:17.437 INFO:teuthology.orchestra.run.smithi112.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.104961.log.gz 2024-09-11T06:53:17.438 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.90818.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82740.log 2024-09-11T06:53:17.438 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.102342.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.77673.log.gz 2024-09-11T06:53:17.439 INFO:teuthology.orchestra.run.smithi112.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.102342.log.gz 2024-09-11T06:53:17.439 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.77917.log.gz 2024-09-11T06:53:17.440 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.82740.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.72534.log.gz 2024-09-11T06:53:17.440 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81325.log 2024-09-11T06:53:17.440 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84236.log 2024-09-11T06:53:17.440 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.81325.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74123.log 2024-09-11T06:53:17.441 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.84236.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47260.log 2024-09-11T06:53:17.441 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.74123.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60027.log 2024-09-11T06:53:17.442 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.47260.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32837.log 2024-09-11T06:53:17.443 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.60027.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66286.log 2024-09-11T06:53:17.443 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.32837.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38158.log 2024-09-11T06:53:17.444 INFO:teuthology.orchestra.run.smithi112.stderr: 95.4% -- replaced with /var/log/ceph/cluster2-client.mirror.130318.log.gz 2024-09-11T06:53:17.444 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.66286.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80035.log 2024-09-11T06:53:17.444 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.38158.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80078.log 2024-09-11T06:53:17.445 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.80035.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30783.log 2024-09-11T06:53:17.445 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.80078.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84015.log 2024-09-11T06:53:17.446 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.30783.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54702.log 2024-09-11T06:53:17.446 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.84015.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89960.log 2024-09-11T06:53:17.446 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.54702.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108258.log 2024-09-11T06:53:17.446 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.89960.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62887.log 2024-09-11T06:53:17.448 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.108258.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47132.log.gz 2024-09-11T06:53:17.448 INFO:teuthology.orchestra.run.smithi112.stderr: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.108258.log.gz 2024-09-11T06:53:17.449 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117503.log 2024-09-11T06:53:17.449 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.62887.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32988.log 2024-09-11T06:53:17.450 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.117503.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.83771.log.gz 2024-09-11T06:53:17.450 INFO:teuthology.orchestra.run.smithi112.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.83736.log.gz 2024-09-11T06:53:17.450 INFO:teuthology.orchestra.run.smithi112.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.42972.log.gz 2024-09-11T06:53:17.451 INFO:teuthology.orchestra.run.smithi112.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.80741.log.gz 2024-09-11T06:53:17.451 INFO:teuthology.orchestra.run.smithi112.stderr: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.117503.log.gz 2024-09-11T06:53:17.451 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45398.log 2024-09-11T06:53:17.452 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.32988.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.84856.log.gz 2024-09-11T06:53:17.452 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36945.log 2024-09-11T06:53:17.452 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.45398.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111089.log 2024-09-11T06:53:17.453 INFO:teuthology.orchestra.run.smithi112.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.36348.log.gz 2024-09-11T06:53:17.453 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.36945.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.86790.log.gz 2024-09-11T06:53:17.453 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- 88.0% /var/log/ceph/cluster2-client.mirror.33488.log 2024-09-11T06:53:17.453 INFO:teuthology.orchestra.run.smithi112.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.36573.log.gz 2024-09-11T06:53:17.454 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.111089.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.111089.log.gz 2024-09-11T06:53:17.454 INFO:teuthology.orchestra.run.smithi112.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.32709.log.gz 2024-09-11T06:53:17.455 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.33488.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.33209.log.gz 2024-09-11T06:53:17.456 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45305.log.gz 2024-09-11T06:53:17.456 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128241.log 2024-09-11T06:53:17.457 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56756.log 2024-09-11T06:53:17.457 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.128241.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.44499.log.gz 2024-09-11T06:53:17.457 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.52213.log.gz 2024-09-11T06:53:17.457 INFO:teuthology.orchestra.run.smithi112.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.128241.log.gz 2024-09-11T06:53:17.458 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.33585.log.gz 2024-09-11T06:53:17.458 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108278.log 2024-09-11T06:53:17.459 INFO:teuthology.orchestra.run.smithi112.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.35321.log.gz 2024-09-11T06:53:17.459 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63542.log 2024-09-11T06:53:17.459 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.56756.log: /var/log/ceph/cluster1-client.mirror.108278.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.76363.log.gz 2024-09-11T06:53:17.459 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.38936.log.gz 2024-09-11T06:53:17.459 INFO:teuthology.orchestra.run.smithi112.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.39003.log.gz 2024-09-11T06:53:17.460 INFO:teuthology.orchestra.run.smithi112.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.56756.log.gz 2024-09-11T06:53:17.460 INFO:teuthology.orchestra.run.smithi112.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.57097.log.gz 2024-09-11T06:53:17.460 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.56384.log.gz 2024-09-11T06:53:17.460 INFO:teuthology.orchestra.run.smithi112.stderr: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.108278.log.gz 2024-09-11T06:53:17.462 INFO:teuthology.orchestra.run.smithi112.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.33488.log.gz 2024-09-11T06:53:17.463 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.31678.log.gz 2024-09-11T06:53:17.464 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.73937.log.gz 2024-09-11T06:53:17.465 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster1-client.admin.82330.log.gz 2024-09-11T06:53:17.466 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.63542.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.73189.log.gz 2024-09-11T06:53:17.467 INFO:teuthology.orchestra.run.smithi112.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.34484.log.gz 2024-09-11T06:53:17.467 INFO:teuthology.orchestra.run.smithi112.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.49411.log.gz 2024-09-11T06:53:17.467 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81945.log 2024-09-11T06:53:17.468 INFO:teuthology.orchestra.run.smithi112.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.54573.log.gz 2024-09-11T06:53:17.468 INFO:teuthology.orchestra.run.smithi112.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.57411.log.gz 2024-09-11T06:53:17.468 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.51186.log.gz 2024-09-11T06:53:17.468 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125524.log 2024-09-11T06:53:17.468 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47450.log.gz 2024-09-11T06:53:17.468 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.81945.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109628.log 2024-09-11T06:53:17.469 INFO:teuthology.orchestra.run.smithi112.stderr: 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.46008.log.gz 2024-09-11T06:53:17.469 INFO:teuthology.orchestra.run.smithi112.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47601.log.gz 2024-09-11T06:53:17.469 INFO:teuthology.orchestra.run.smithi112.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.65782.log.gz 2024-09-11T06:53:17.470 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38348.log 2024-09-11T06:53:17.470 INFO:teuthology.orchestra.run.smithi112.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.78324.log.gz 2024-09-11T06:53:17.470 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.79820.log.gz 2024-09-11T06:53:17.470 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.125524.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.81794.log.gz 2024-09-11T06:53:17.471 INFO:teuthology.orchestra.run.smithi112.stderr:gzip 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.125524.log.gz -5 2024-09-11T06:53:17.471 INFO:teuthology.orchestra.run.smithi112.stderr: --verbose -- /var/log/ceph/cluster2-client.mirror.123740.log 2024-09-11T06:53:17.471 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33771.log 2024-09-11T06:53:17.472 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.38348.log: /var/log/ceph/cluster2-client.mirror.123740.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.123740.log.gz 2024-09-11T06:53:17.472 INFO:teuthology.orchestra.run.smithi112.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.79130.log.gz 2024-09-11T06:53:17.472 INFO:teuthology.orchestra.run.smithi112.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.81945.log.gz 2024-09-11T06:53:17.472 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85977.log 2024-09-11T06:53:17.473 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.33771.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28941.log 2024-09-11T06:53:17.473 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.85977.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.83395.log.gz 2024-09-11T06:53:17.473 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.66379.log.gz 2024-09-11T06:53:17.474 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77111.log 2024-09-11T06:53:17.474 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.admin.28941.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28941.log.gz 2024-09-11T06:53:17.475 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106186.log 2024-09-11T06:53:17.475 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.77111.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43813.log 2024-09-11T06:53:17.477 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.106186.log: /var/log/ceph/cluster2-client.mirror.109628.log: 96.9% -- replaced with /var/log/ceph/cluster2-client.mirror.46097.log.gz 2024-09-11T06:53:17.478 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.55543.log.gz 2024-09-11T06:53:17.478 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.55787.log.gz 2024-09-11T06:53:17.478 INFO:teuthology.orchestra.run.smithi112.stderr: 91.7% -- replaced with /var/log/ceph/cluster1-client.mirror.106186.log.gz 2024-09-11T06:53:17.479 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.60775.log.gz 2024-09-11T06:53:17.479 INFO:teuthology.orchestra.run.smithi112.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.62232.log.gz 2024-09-11T06:53:17.479 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.43813.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89703.log 2024-09-11T06:53:17.480 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.62887.log.gz 2024-09-11T06:53:17.480 INFO:teuthology.orchestra.run.smithi112.stderr:gzip 88.8% -5 -- replaced with /var/log/ceph/cluster2-client.mirror.67060.log.gz --verbose 2024-09-11T06:53:17.481 INFO:teuthology.orchestra.run.smithi112.stderr: -- /var/log/ceph/cluster2-client.mirror.96312.log 2024-09-11T06:53:17.481 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107900.log 2024-09-11T06:53:17.481 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.42445.log.gz 2024-09-11T06:53:17.481 INFO:teuthology.orchestra.run.smithi112.stderr: 95.7%/var/log/ceph/cluster2-client.mirror.89703.log: /var/log/ceph/cluster2-client.mirror.96312.log: -- replaced with /var/log/ceph/cluster2-client.mirror.109628.log.gz 2024-09-11T06:53:17.481 INFO:teuthology.orchestra.run.smithi112.stderr: 84.5% -- replaced with /var/log/ceph/cluster2-client.mirror.89703.log.gz 2024-09-11T06:53:17.482 INFO:teuthology.orchestra.run.smithi112.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.82740.log.gz 2024-09-11T06:53:17.483 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.25487.log 2024-09-11T06:53:17.484 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.107900.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.87029.log.gz 2024-09-11T06:53:17.484 INFO:teuthology.orchestra.run.smithi112.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.36255.log.gz 2024-09-11T06:53:17.484 INFO:teuthology.orchestra.run.smithi112.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.36445.log.gz 2024-09-11T06:53:17.485 INFO:teuthology.orchestra.run.smithi112.stderr: 92.0% -- replaced with /var/log/ceph/cluster1-client.mirror.107900.log.gz 2024-09-11T06:53:17.486 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87139.log 2024-09-11T06:53:17.486 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.62980.log.gz 2024-09-11T06:53:17.486 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.admin.25487.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.25487.log.gz 2024-09-11T06:53:17.487 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39189.log 2024-09-11T06:53:17.487 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.87139.log: 54.9% -- replaced with /var/log/ceph/cluster1-client.mirror.29330.log.gz 2024-09-11T06:53:17.487 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75301.log 2024-09-11T06:53:17.487 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.39189.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71565.log 2024-09-11T06:53:17.490 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.75301.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.30651.log.gz 2024-09-11T06:53:17.491 INFO:teuthology.orchestra.run.smithi112.stderr: 88.7% -- replaced with /var/log/ceph/cluster1-client.admin.54442.log.gz 2024-09-11T06:53:17.493 INFO:teuthology.orchestra.run.smithi112.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.65817.log.gz 2024-09-11T06:53:17.493 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36666.log 2024-09-11T06:53:17.494 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.71565.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126559.log 2024-09-11T06:53:17.495 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.36666.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42191.log 2024-09-11T06:53:17.496 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.126559.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.126559.log.gz 2024-09-11T06:53:17.496 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.29116.log 2024-09-11T06:53:17.497 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.42191.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127616.log 2024-09-11T06:53:17.498 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.admin.29116.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.29116.log.gz 2024-09-11T06:53:17.498 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49318.log 2024-09-11T06:53:17.499 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.127616.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.127616.log.gz 2024-09-11T06:53:17.502 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49841.log 2024-09-11T06:53:17.503 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.49318.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42631.log 2024-09-11T06:53:17.504 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.49841.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61988.log 2024-09-11T06:53:17.505 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.42631.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113535.log 2024-09-11T06:53:17.506 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.61988.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115704.log 2024-09-11T06:53:17.506 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.113535.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.113535.log.gz 2024-09-11T06:53:17.507 INFO:teuthology.orchestra.run.smithi112.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.32837.log.gz 2024-09-11T06:53:17.507 INFO:teuthology.orchestra.run.smithi112.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.32895.log.gz 2024-09-11T06:53:17.508 INFO:teuthology.orchestra.run.smithi112.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.32988.log.gz 2024-09-11T06:53:17.509 INFO:teuthology.orchestra.run.smithi112.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.33267.log.gz 2024-09-11T06:53:17.509 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77076.log 2024-09-11T06:53:17.511 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.115704.log: 91.8% -- replaced with /var/log/ceph/cluster1-client.mirror.115704.log.gz 2024-09-11T06:53:17.511 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85340.log 2024-09-11T06:53:17.512 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.77076.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34670.log 2024-09-11T06:53:17.513 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.85340.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41223.log 2024-09-11T06:53:17.514 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.34670.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72034.log 2024-09-11T06:53:17.515 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.41223.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55977.log 2024-09-11T06:53:17.515 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.72034.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120842.log 2024-09-11T06:53:17.516 INFO:teuthology.orchestra.run.smithi112.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.38220.log.gz 2024-09-11T06:53:17.521 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.55977.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83488.log 2024-09-11T06:53:17.522 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.120842.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.120842.log.gz 2024-09-11T06:53:17.522 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87117.log 2024-09-11T06:53:17.522 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.83488.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72627.log 2024-09-11T06:53:17.522 INFO:teuthology.orchestra.run.smithi112.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.43813.log.gz 2024-09-11T06:53:17.523 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.87117.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52271.log 2024-09-11T06:53:17.523 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.72627.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76270.log 2024-09-11T06:53:17.523 INFO:teuthology.orchestra.run.smithi112.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.43906.log.gz 2024-09-11T06:53:17.524 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.52271.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.52085.log.gz 2024-09-11T06:53:17.524 INFO:teuthology.orchestra.run.smithi112.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.43565.log.gz 2024-09-11T06:53:17.525 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.47074.log.gz 2024-09-11T06:53:17.527 INFO:teuthology.orchestra.run.smithi112.stderr: 84.8% -- replaced with /var/log/ceph/cluster2-client.mirror.87440.log.gz 2024-09-11T06:53:17.527 INFO:teuthology.orchestra.run.smithi112.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.31248.log.gz 2024-09-11T06:53:17.528 INFO:teuthology.orchestra.run.smithi112.stderr: 88.7% -- replaced with /var/log/ceph/cluster1-client.mirror.85376.log.gz 2024-09-11T06:53:17.528 INFO:teuthology.orchestra.run.smithi112.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.40417.log.gz 2024-09-11T06:53:17.529 INFO:teuthology.orchestra.run.smithi112.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.34670.log.gz 2024-09-11T06:53:17.530 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65069.log 2024-09-11T06:53:17.532 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.76270.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.37189.log.gz 2024-09-11T06:53:17.532 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49841.log.gz 2024-09-11T06:53:17.532 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50120.log.gz 2024-09-11T06:53:17.533 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120745.log 2024-09-11T06:53:17.533 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.57376.log.gz 2024-09-11T06:53:17.533 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.57748.log.gz 2024-09-11T06:53:17.534 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.65069.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38313.log 2024-09-11T06:53:17.535 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.120745.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.76646.log.gz 2024-09-11T06:53:17.535 INFO:teuthology.orchestra.run.smithi112.stderr: 88.8% 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.77111.log.gz -- replaced with /var/log/ceph/cluster2-client.mirror.77076.log.gz 2024-09-11T06:53:17.535 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-11T06:53:17.535 INFO:teuthology.orchestra.run.smithi112.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.71565.log.gz 2024-09-11T06:53:17.535 INFO:teuthology.orchestra.run.smithi112.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.81887.log.gz 2024-09-11T06:53:17.535 INFO:teuthology.orchestra.run.smithi112.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.83488.log.gz 2024-09-11T06:53:17.535 INFO:teuthology.orchestra.run.smithi112.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.55915.log.gz 2024-09-11T06:53:17.536 INFO:teuthology.orchestra.run.smithi112.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.55977.log.gz 2024-09-11T06:53:17.536 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.68519.log.gz 2024-09-11T06:53:17.536 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.65069.log.gz 2024-09-11T06:53:17.537 INFO:teuthology.orchestra.run.smithi112.stderr: 91.6% -- replaced with /var/log/ceph/cluster1-client.mirror.120745.log.gz 2024-09-11T06:53:17.537 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.90572.log 2024-09-11T06:53:17.538 INFO:teuthology.orchestra.run.smithi112.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.72592.log.gz 2024-09-11T06:53:17.538 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.72627.log.gz 2024-09-11T06:53:17.539 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.38313.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46223.log 2024-09-11T06:53:17.540 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.90572.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.74871.log.gz 2024-09-11T06:53:17.541 INFO:teuthology.orchestra.run.smithi112.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.75301.log.gz 2024-09-11T06:53:17.542 INFO:teuthology.orchestra.run.smithi112.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.35976.log.gz 2024-09-11T06:53:17.543 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.72817.log.gz 2024-09-11T06:53:17.544 INFO:teuthology.orchestra.run.smithi112.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.87139.log.gz 2024-09-11T06:53:17.544 INFO:teuthology.orchestra.run.smithi112.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.81232.log.gz 2024-09-11T06:53:17.544 INFO:teuthology.orchestra.run.smithi112.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.87117.log.gz 2024-09-11T06:53:17.544 INFO:teuthology.orchestra.run.smithi112.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.82802.log.gz 2024-09-11T06:53:17.549 INFO:teuthology.orchestra.run.smithi112.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.36290.log.gz 2024-09-11T06:53:17.550 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.36666.log.gz 2024-09-11T06:53:17.552 INFO:teuthology.orchestra.run.smithi112.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.81104.log.gz 2024-09-11T06:53:17.554 INFO:teuthology.orchestra.run.smithi112.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.40138.log.gz 2024-09-11T06:53:17.557 INFO:teuthology.orchestra.run.smithi112.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.49221.log.gz 2024-09-11T06:53:17.558 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53798.log 2024-09-11T06:53:17.559 INFO:teuthology.orchestra.run.smithi112.stderr: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.92068.log.gz 2024-09-11T06:53:17.559 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.46223.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.76270.log.gz 2024-09-11T06:53:17.559 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42213.log 2024-09-11T06:53:17.559 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.53798.log: 93.3% -- replaced with /var/log/ceph/cluster2-client.mirror.86612.log.gz 2024-09-11T06:53:17.560 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124082.log 2024-09-11T06:53:17.561 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.42213.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53971.log 2024-09-11T06:53:17.562 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.124082.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.124082.log.gz 2024-09-11T06:53:17.562 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49504.log 2024-09-11T06:53:17.563 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.53971.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124033.log 2024-09-11T06:53:17.564 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.49504.log: 95.5% -- replaced with /var/log/ceph/cluster2-client.mirror.90818.log.gz 2024-09-11T06:53:17.564 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109494.log 2024-09-11T06:53:17.565 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.127751.log 2024-09-11T06:53:17.566 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.124033.log: 88.3%/var/log/ceph/cluster2-client.mirror.109494.log: -- replaced with /var/log/ceph/cluster2-client.mirror.124033.log.gz 2024-09-11T06:53:17.566 INFO:teuthology.orchestra.run.smithi112.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.109494.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86452.log 2024-09-11T06:53:17.566 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-11T06:53:17.566 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.0.127751.log: 20.3%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89726.log 2024-09-11T06:53:17.566 INFO:teuthology.orchestra.run.smithi112.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.0.127751.log.gz 2024-09-11T06:53:17.567 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.86452.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129621.log 2024-09-11T06:53:17.567 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.89726.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44689.log 2024-09-11T06:53:17.568 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.129621.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.38348.log.gz 2024-09-11T06:53:17.569 INFO:teuthology.orchestra.run.smithi112.stderr: 91.8% -- replaced with /var/log/ceph/cluster2-client.mirror.129621.log.gz 2024-09-11T06:53:17.570 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63670.log 2024-09-11T06:53:17.570 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.44689.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39297.log 2024-09-11T06:53:17.572 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.63670.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70201.log 2024-09-11T06:53:17.573 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.39297.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49376.log 2024-09-11T06:53:17.573 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.70201.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102454.log 2024-09-11T06:53:17.574 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.49376.log: gzip 88.3% -5 -- replaced with /var/log/ceph/cluster2-client.mirror.36069.log.gz 2024-09-11T06:53:17.574 INFO:teuthology.orchestra.run.smithi112.stderr: --verbose -- /var/log/ceph/cluster2-client.mirror.124104.log 2024-09-11T06:53:17.576 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.102454.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.36945.log.gz 2024-09-11T06:53:17.576 INFO:teuthology.orchestra.run.smithi112.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.37003.log.gz 2024-09-11T06:53:17.576 INFO:teuthology.orchestra.run.smithi112.stderr: 91.7% -- replaced with /var/log/ceph/cluster1-client.mirror.102454.log.gz 2024-09-11T06:53:17.577 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45870.log 2024-09-11T06:53:17.577 INFO:teuthology.orchestra.run.smithi112.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.30558.log.gz 2024-09-11T06:53:17.578 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.124104.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.124104.log.gz 2024-09-11T06:53:17.578 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125005.log 2024-09-11T06:53:17.578 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.52271.log.gz 2024-09-11T06:53:17.579 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.45870.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.46853.log.gz 2024-09-11T06:53:17.579 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48814.log 2024-09-11T06:53:17.581 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.125005.log: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.125005.log.gz 2024-09-11T06:53:17.582 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57318.log 2024-09-11T06:53:17.582 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.48814.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129559.log 2024-09-11T06:53:17.584 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.57318.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58159.log 2024-09-11T06:53:17.584 INFO:teuthology.orchestra.run.smithi112.stderr: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.128169.log.gz 2024-09-11T06:53:17.584 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.129559.log: gzip 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.129559.log.gz -5 2024-09-11T06:53:17.584 INFO:teuthology.orchestra.run.smithi112.stderr: --verbose -- /var/log/ceph/cluster2-client.mirror.61616.log 2024-09-11T06:53:17.585 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.58159.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80808.log 2024-09-11T06:53:17.586 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.61616.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76421.log 2024-09-11T06:53:17.586 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.80808.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40789.log 2024-09-11T06:53:17.586 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53519.log 2024-09-11T06:53:17.587 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.76421.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59058.log 2024-09-11T06:53:17.588 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.40789.log: /var/log/ceph/cluster2-client.mirror.53519.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41955.log 2024-09-11T06:53:17.589 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.59058.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28438.log 2024-09-11T06:53:17.589 INFO:teuthology.orchestra.run.smithi112.stderr: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.32616.log.gz 2024-09-11T06:53:17.589 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71848.log 2024-09-11T06:53:17.589 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.41955.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.56814.log.gz 2024-09-11T06:53:17.589 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.admin.28438.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.28438.log.gz 2024-09-11T06:53:17.589 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68205.log 2024-09-11T06:53:17.590 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64569.log 2024-09-11T06:53:17.590 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.68205.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.32089.log.gz 2024-09-11T06:53:17.590 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.71848.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.32275.log.gz 2024-09-11T06:53:17.590 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124914.log 2024-09-11T06:53:17.591 INFO:teuthology.orchestra.run.smithi112.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.40882.log.gz 2024-09-11T06:53:17.591 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.64569.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.34856.log.gz 2024-09-11T06:53:17.591 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41316.log 2024-09-11T06:53:17.592 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85569.log 2024-09-11T06:53:17.592 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.124914.log: /var/log/ceph/cluster2-client.mirror.41316.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.124914.log.gz 2024-09-11T06:53:17.592 INFO:teuthology.orchestra.run.smithi112.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.35639.log.gz 2024-09-11T06:53:17.592 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126941.log 2024-09-11T06:53:17.593 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.85569.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86702.log 2024-09-11T06:53:17.594 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86878.log 2024-09-11T06:53:17.594 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.126941.log: /var/log/ceph/cluster2-client.mirror.86702.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.126941.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68891.log 2024-09-11T06:53:17.595 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-11T06:53:17.597 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.86878.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62267.log 2024-09-11T06:53:17.597 INFO:teuthology.orchestra.run.smithi112.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.80432.log.gz 2024-09-11T06:53:17.597 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.68891.log: 95.3% -- replaced with /var/log/ceph/cluster2-client.mirror.127821.log.gz 2024-09-11T06:53:17.597 INFO:teuthology.orchestra.run.smithi112.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.89960.log.gz 2024-09-11T06:53:17.597 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86962.log 2024-09-11T06:53:17.597 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.90184.log 2024-09-11T06:53:17.597 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.62267.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68953.log 2024-09-11T06:53:17.597 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.86962.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.33771.log.gz 2024-09-11T06:53:17.597 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.90184.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49376.log.gz 2024-09-11T06:53:17.599 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125771.log 2024-09-11T06:53:17.599 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.68953.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115726.log 2024-09-11T06:53:17.600 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.50438.log.gz 2024-09-11T06:53:17.600 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.125771.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126289.log 2024-09-11T06:53:17.600 INFO:teuthology.orchestra.run.smithi112.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.125771.log.gz 2024-09-11T06:53:17.600 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.115726.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30841.log 2024-09-11T06:53:17.600 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.50868.log.gz 2024-09-11T06:53:17.600 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.126289.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.126289.log.gz 2024-09-11T06:53:17.600 INFO:teuthology.orchestra.run.smithi112.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.77204.log.gz 2024-09-11T06:53:17.600 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61581.log 2024-09-11T06:53:17.600 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52957.log 2024-09-11T06:53:17.601 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.30841.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130758.log 2024-09-11T06:53:17.601 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48252.log 2024-09-11T06:53:17.601 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.61581.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.79072.log.gz 2024-09-11T06:53:17.602 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.52957.log: 84.9% -- replaced with /var/log/ceph/cluster2-client.mirror.87367.log.gz 2024-09-11T06:53:17.602 INFO:teuthology.orchestra.run.smithi112.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.71135.log.gz 2024-09-11T06:53:17.603 INFO:teuthology.orchestra.run.smithi112.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.55264.log.gz 2024-09-11T06:53:17.603 INFO:teuthology.orchestra.run.smithi112.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.62267.log.gz 2024-09-11T06:53:17.603 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.130758.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.68205.log.gz 2024-09-11T06:53:17.603 INFO:teuthology.orchestra.run.smithi112.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.130758.log.gz 2024-09-11T06:53:17.603 INFO:teuthology.orchestra.run.smithi112.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.69825.log.gz 2024-09-11T06:53:17.604 INFO:teuthology.orchestra.run.smithi112.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.103054.log.gz 2024-09-11T06:53:17.604 INFO:teuthology.orchestra.run.smithi112.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.54946.log.gz 2024-09-11T06:53:17.604 INFO:teuthology.orchestra.run.smithi112.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.115726.log.gz 2024-09-11T06:53:17.606 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130461.log 2024-09-11T06:53:17.607 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.48252.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123940.log 2024-09-11T06:53:17.607 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.59058.log.gz 2024-09-11T06:53:17.607 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81701.log 2024-09-11T06:53:17.607 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.130461.log: 86.9%/var/log/ceph/cluster2-client.mirror.123940.log: -- replaced with /var/log/ceph/cluster1-client.mirror.130461.log.gz 2024-09-11T06:53:17.609 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.29807.log 2024-09-11T06:53:17.609 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.53798.log.gz 2024-09-11T06:53:17.609 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.81701.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116939.log 2024-09-11T06:53:17.609 INFO:teuthology.orchestra.run.smithi112.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.123940.log.gz 2024-09-11T06:53:17.609 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109064.log 2024-09-11T06:53:17.609 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.1.29807.log: gzip/var/log/ceph/cluster1-client.mirror.116939.log: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88148.log 2024-09-11T06:53:17.609 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.109064.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.63108.log.gz 2024-09-11T06:53:17.609 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.88148.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51930.log 2024-09-11T06:53:17.609 INFO:teuthology.orchestra.run.smithi112.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.62422.log.gz 2024-09-11T06:53:17.610 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106338.log 2024-09-11T06:53:17.610 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.51930.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.63670.log.gz 2024-09-11T06:53:17.610 INFO:teuthology.orchestra.run.smithi112.stderr: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.116939.log.gz 2024-09-11T06:53:17.611 INFO:teuthology.orchestra.run.smithi112.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.59616.log.gz 2024-09-11T06:53:17.611 INFO:teuthology.orchestra.run.smithi112.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.64569.log.gz 2024-09-11T06:53:17.611 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.65166.log.gz 2024-09-11T06:53:17.612 INFO:teuthology.orchestra.run.smithi112.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.68953.log.gz 2024-09-11T06:53:17.612 INFO:teuthology.orchestra.run.smithi112.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.71848.log.gz 2024-09-11T06:53:17.612 INFO:teuthology.orchestra.run.smithi112.stderr: 54.2% -- replaced with /var/log/ceph/cluster1-client.mirror.29432.log.gz 2024-09-11T06:53:17.613 INFO:teuthology.orchestra.run.smithi112.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.72069.log.gz 2024-09-11T06:53:17.613 INFO:teuthology.orchestra.run.smithi112.stderr: 80.2% -- replaced with /var/log/ceph/cluster2-client.admin.29518.log.gz 2024-09-11T06:53:17.615 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123192.log 2024-09-11T06:53:17.615 INFO:teuthology.orchestra.run.smithi112.stderr: 17.9% -- replaced with /var/log/ceph/cluster2-client.mirror.1.29807.log.gz 2024-09-11T06:53:17.615 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.106338.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86432.log 2024-09-11T06:53:17.615 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.123192.log: 96.9% -- replaced with /var/log/ceph/cluster2-client.mirror.109064.log.gz 2024-09-11T06:53:17.615 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.78700.log.gz 2024-09-11T06:53:17.616 INFO:teuthology.orchestra.run.smithi112.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.41955.log.gz 2024-09-11T06:53:17.616 INFO:teuthology.orchestra.run.smithi112.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.123192.log.gz 2024-09-11T06:53:17.617 INFO:teuthology.orchestra.run.smithi112.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.86702.log.gz 2024-09-11T06:53:17.618 INFO:teuthology.orchestra.run.smithi112.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.86853.log.gz 2024-09-11T06:53:17.619 INFO:teuthology.orchestra.run.smithi112.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.42821.log.gz 2024-09-11T06:53:17.620 INFO:teuthology.orchestra.run.smithi112.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.81383.log.gz 2024-09-11T06:53:17.621 INFO:teuthology.orchestra.run.smithi112.stderr: 96.5% -- replaced with /var/log/ceph/cluster2-client.mirror.106338.log.gz 2024-09-11T06:53:17.621 INFO:teuthology.orchestra.run.smithi112.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.86878.log.gz 2024-09-11T06:53:17.621 INFO:teuthology.orchestra.run.smithi112.stderr: 84.7% -- replaced with /var/log/ceph/cluster2-client.mirror.88510.log.gz 2024-09-11T06:53:17.621 INFO:teuthology.orchestra.run.smithi112.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.86962.log.gz 2024-09-11T06:53:17.626 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130210.log 2024-09-11T06:53:17.627 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.86432.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78045.log 2024-09-11T06:53:17.627 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.130210.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87610.log 2024-09-11T06:53:17.627 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3%/var/log/ceph/cluster2-client.mirror.78045.log: -- replaced with /var/log/ceph/cluster2-client.mirror.130210.log.gz 2024-09-11T06:53:17.627 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64883.log 2024-09-11T06:53:17.630 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.87610.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127369.log 2024-09-11T06:53:17.630 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5/var/log/ceph/cluster2-client.mirror.64883.log: --verbose -- /var/log/ceph/cluster2-client.mirror.130482.log 2024-09-11T06:53:17.630 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.127369.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30173.log 2024-09-11T06:53:17.630 INFO:teuthology.orchestra.run.smithi112.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.127369.log.gz/var/log/ceph/cluster2-client.mirror.130482.log: 2024-09-11T06:53:17.630 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124985.log 2024-09-11T06:53:17.630 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.30173.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.29603.log 2024-09-11T06:53:17.630 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.130482.log.gz 2024-09-11T06:53:17.630 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.124985.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99465.log 2024-09-11T06:53:17.631 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.29603.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58779.log 2024-09-11T06:53:17.631 INFO:teuthology.orchestra.run.smithi112.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.124985.log.gz 2024-09-11T06:53:17.631 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.99465.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116037.log 2024-09-11T06:53:17.631 INFO:teuthology.orchestra.run.smithi112.stderr: 1.3% -- replaced with /var/log/ceph/cluster2-client.mirror.99465.log.gz 2024-09-11T06:53:17.633 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.58779.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.35453.log.gz 2024-09-11T06:53:17.633 INFO:teuthology.orchestra.run.smithi112.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.44689.log.gz 2024-09-11T06:53:17.633 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49876.log 2024-09-11T06:53:17.633 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.51930.log.gz 2024-09-11T06:53:17.633 INFO:teuthology.orchestra.run.smithi112.stderr: 79.9% -- replaced with /var/log/ceph/cluster1-client.admin.29279.log.gz 2024-09-11T06:53:17.633 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.116037.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35546.log 2024-09-11T06:53:17.633 INFO:teuthology.orchestra.run.smithi112.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.33922.log.gz 2024-09-11T06:53:17.633 INFO:teuthology.orchestra.run.smithi112.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.116037.log.gz 2024-09-11T06:53:17.633 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.49876.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.58779.log.gz 2024-09-11T06:53:17.633 INFO:teuthology.orchestra.run.smithi112.stderr: 88.8%gzip -- replaced with /var/log/ceph/cluster2-client.mirror.60926.log.gz -5 2024-09-11T06:53:17.633 INFO:teuthology.orchestra.run.smithi112.stderr: --verbose -- /var/log/ceph/cluster1-client.mirror.126514.log 2024-09-11T06:53:17.633 INFO:teuthology.orchestra.run.smithi112.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.61581.log.gz 2024-09-11T06:53:17.634 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79320.log 2024-09-11T06:53:17.634 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.35546.log: /var/log/ceph/cluster1-client.mirror.126514.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.126514.log.gz 2024-09-11T06:53:17.635 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117483.log 2024-09-11T06:53:17.635 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.79320.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63391.log 2024-09-11T06:53:17.635 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.117483.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115298.log 2024-09-11T06:53:17.635 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.63391.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126379.log 2024-09-11T06:53:17.636 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.115298.log: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.117483.log.gz 2024-09-11T06:53:17.636 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54240.log 2024-09-11T06:53:17.636 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.126379.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74685.log 2024-09-11T06:53:17.637 INFO:teuthology.orchestra.run.smithi112.stderr: 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.115298.log.gz 2024-09-11T06:53:17.637 INFO:teuthology.orchestra.run.smithi112.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.126379.log.gz 2024-09-11T06:53:17.637 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.54240.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42340.log 2024-09-11T06:53:17.638 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.74685.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121109.log 2024-09-11T06:53:17.638 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88839.log 2024-09-11T06:53:17.638 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.42340.log: /var/log/ceph/cluster2-client.mirror.121109.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53240.log 2024-09-11T06:53:17.638 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94430.log 2024-09-11T06:53:17.638 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.88839.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.121109.log.gz 2024-09-11T06:53:17.639 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.53240.log: 84.7% -- replaced with /var/log/ceph/cluster2-client.mirror.88839.log.gz 2024-09-11T06:53:17.641 INFO:teuthology.orchestra.run.smithi112.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.30841.log.gz 2024-09-11T06:53:17.641 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106109.log 2024-09-11T06:53:17.641 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.94430.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116104.log 2024-09-11T06:53:17.641 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.106109.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125974.log 2024-09-11T06:53:17.641 INFO:teuthology.orchestra.run.smithi112.stderr: 84.7% -- replaced with /var/log/ceph/cluster2-client.mirror.94430.log.gz 2024-09-11T06:53:17.643 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124963.log 2024-09-11T06:53:17.643 INFO:teuthology.orchestra.run.smithi112.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.106109.log.gz 2024-09-11T06:53:17.643 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.116104.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.31492.log.gz 2024-09-11T06:53:17.643 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.125974.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.125974.log.gz 2024-09-11T06:53:17.643 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95384.log 2024-09-11T06:53:17.643 INFO:teuthology.orchestra.run.smithi112.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.41316.log.gz 2024-09-11T06:53:17.643 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.124963.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123092.log 2024-09-11T06:53:17.643 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.95384.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.82299.log 2024-09-11T06:53:17.643 INFO:teuthology.orchestra.run.smithi112.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.124963.log.gz 2024-09-11T06:53:17.644 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.123092.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38123.log 2024-09-11T06:53:17.644 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.74030.log.gz 2024-09-11T06:53:17.644 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.admin.82299.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.74181.log.gz 2024-09-11T06:53:17.644 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59244.log 2024-09-11T06:53:17.645 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.38123.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.57562.log.gz 2024-09-11T06:53:17.646 INFO:teuthology.orchestra.run.smithi112.stderr: 90.9% -- replaced with /var/log/ceph/cluster2-client.mirror.123092.log.gz 2024-09-11T06:53:17.646 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.65596.log.gz 2024-09-11T06:53:17.646 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.49876.log.gz 2024-09-11T06:53:17.646 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72220.log 2024-09-11T06:53:17.646 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.59244.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120668.log 2024-09-11T06:53:17.647 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.72220.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124377.log 2024-09-11T06:53:17.647 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.120668.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.120668.log.gz 2024-09-11T06:53:17.647 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50252.log 2024-09-11T06:53:17.647 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.124377.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72499.log 2024-09-11T06:53:17.648 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.50252.log: 96.0% 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.124377.log.gz 2024-09-11T06:53:17.648 INFO:teuthology.orchestra.run.smithi112.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.116104.log.gz 2024-09-11T06:53:17.649 INFO:teuthology.orchestra.run.smithi112.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.79320.log.gz 2024-09-11T06:53:17.650 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.79541.log.gz 2024-09-11T06:53:17.651 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126896.log 2024-09-11T06:53:17.651 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.72499.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128655.log 2024-09-11T06:53:17.653 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73716.log 2024-09-11T06:53:17.653 INFO:teuthology.orchestra.run.smithi112.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.81701.log.gz 2024-09-11T06:53:17.653 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.126896.log: /var/log/ceph/cluster1-client.mirror.128655.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47822.log 2024-09-11T06:53:17.653 INFO:teuthology.orchestra.run.smithi112.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.128655.log.gz 2024-09-11T06:53:17.653 INFO:teuthology.orchestra.run.smithi112.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.126896.log.gz 2024-09-11T06:53:17.653 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.73716.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81604.log 2024-09-11T06:53:17.653 INFO:teuthology.orchestra.run.smithi112.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.60647.log.gz 2024-09-11T06:53:17.653 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.47822.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73061.log 2024-09-11T06:53:17.655 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76704.log 2024-09-11T06:53:17.655 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.81604.log: /var/log/ceph/cluster2-client.mirror.73061.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127684.log 2024-09-11T06:53:17.655 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.76704.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.70631.log.gz 2024-09-11T06:53:17.655 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61953.log 2024-09-11T06:53:17.655 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124934.log 2024-09-11T06:53:17.655 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.127684.log: /var/log/ceph/cluster2-client.mirror.61953.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55450.log 2024-09-11T06:53:17.656 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.124934.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.52957.log.gz 2024-09-11T06:53:17.656 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45061.log 2024-09-11T06:53:17.657 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.53298.log.gz 2024-09-11T06:53:17.657 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.53240.log.gz 2024-09-11T06:53:17.657 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.55450.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.127684.log.gz 2024-09-11T06:53:17.657 INFO:teuthology.orchestra.run.smithi112.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.59992.log.gz 2024-09-11T06:53:17.658 INFO:teuthology.orchestra.run.smithi112.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.30238.log.gz 2024-09-11T06:53:17.658 INFO:teuthology.orchestra.run.smithi112.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.60399.log.gz 2024-09-11T06:53:17.658 INFO:teuthology.orchestra.run.smithi112.stderr: 90.8% -- replaced with /var/log/ceph/cluster2-client.mirror.124934.log.gz 2024-09-11T06:53:17.658 INFO:teuthology.orchestra.run.smithi112.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.63391.log.gz 2024-09-11T06:53:17.658 INFO:teuthology.orchestra.run.smithi112.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.66158.log.gz 2024-09-11T06:53:17.658 INFO:teuthology.orchestra.run.smithi112.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.72499.log.gz 2024-09-11T06:53:17.658 INFO:teuthology.orchestra.run.smithi112.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.42032.log.gz 2024-09-11T06:53:17.659 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121273.log 2024-09-11T06:53:17.661 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43848.log 2024-09-11T06:53:17.661 INFO:teuthology.orchestra.run.smithi112.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.81290.log.gz 2024-09-11T06:53:17.661 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.45061.log: /var/log/ceph/cluster2-client.mirror.121273.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43530.log 2024-09-11T06:53:17.661 INFO:teuthology.orchestra.run.smithi112.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.121273.log.gz 2024-09-11T06:53:17.661 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67172.log 2024-09-11T06:53:17.661 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.43848.log: /var/log/ceph/cluster2-client.mirror.43530.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124448.log 2024-09-11T06:53:17.661 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.67172.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.119417.log 2024-09-11T06:53:17.661 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.74495.log.gz 2024-09-11T06:53:17.662 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.124448.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.124448.log.gz 2024-09-11T06:53:17.662 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72313.log 2024-09-11T06:53:17.662 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.119417.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125906.log 2024-09-11T06:53:17.663 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.72313.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74309.log 2024-09-11T06:53:17.663 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129372.log 2024-09-11T06:53:17.663 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.125906.log: /var/log/ceph/cluster2-client.mirror.74309.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.125906.log.gz 2024-09-11T06:53:17.664 INFO:teuthology.orchestra.run.smithi112.stderr: 89.0% -- replaced with /var/log/ceph/cluster1-client.admin.82299.log.gz 2024-09-11T06:53:17.664 INFO:teuthology.orchestra.run.smithi112.stderr: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.119417.log.gz 2024-09-11T06:53:17.669 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47729.log 2024-09-11T06:53:17.670 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.129372.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.129372.log.gz 2024-09-11T06:53:17.671 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32930.log 2024-09-11T06:53:17.671 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61205.log 2024-09-11T06:53:17.671 INFO:teuthology.orchestra.run.smithi112.stderr: 96.4% -- replaced with /var/log/ceph/cluster2-client.mirror.54329.log.gz 2024-09-11T06:53:17.672 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.47729.log: /var/log/ceph/cluster2-client.mirror.32930.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.29475.log 2024-09-11T06:53:17.672 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.61205.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51523.log.gz 2024-09-11T06:53:17.672 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41688.log 2024-09-11T06:53:17.673 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.admin.29475.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45247.log 2024-09-11T06:53:17.674 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.29475.log.gz 2024-09-11T06:53:17.674 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.41688.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88283.log 2024-09-11T06:53:17.674 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128717.log 2024-09-11T06:53:17.674 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.45247.log: /var/log/ceph/cluster2-client.mirror.88283.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121804.log 2024-09-11T06:53:17.674 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.128717.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.85704.log.gz 2024-09-11T06:53:17.676 INFO:teuthology.orchestra.run.smithi112.stderr: 91.8% -- replaced with /var/log/ceph/cluster2-client.mirror.128717.log.gz 2024-09-11T06:53:17.676 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73282.log 2024-09-11T06:53:17.677 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31155.log 2024-09-11T06:53:17.678 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.121804.log: /var/log/ceph/cluster2-client.mirror.73282.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60120.log 2024-09-11T06:53:17.678 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.31155.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104197.log 2024-09-11T06:53:17.678 INFO:teuthology.orchestra.run.smithi112.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.121804.log.gz 2024-09-11T06:53:17.679 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.60120.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76518.log 2024-09-11T06:53:17.680 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.104197.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98575.log 2024-09-11T06:53:17.680 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.76518.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87635.log 2024-09-11T06:53:17.681 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45758.log 2024-09-11T06:53:17.681 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.98575.log: /var/log/ceph/cluster2-client.mirror.87635.log: 84.5% -- replaced with /var/log/ceph/cluster2-client.mirror.98575.log.gz 2024-09-11T06:53:17.681 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32519.log 2024-09-11T06:53:17.682 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126469.log 2024-09-11T06:53:17.682 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.32519.log: /var/log/ceph/cluster2-client.mirror.45758.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75991.log 2024-09-11T06:53:17.683 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.126469.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41409.log 2024-09-11T06:53:17.683 INFO:teuthology.orchestra.run.smithi112.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.126469.log.gz 2024-09-11T06:53:17.684 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.75991.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106545.log 2024-09-11T06:53:17.684 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.41409.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.131443.log 2024-09-11T06:53:17.685 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113567.log 2024-09-11T06:53:17.685 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.106545.log: /var/log/ceph/cluster2-client.admin.131443.log: 96.4% -- replaced with /var/log/ceph/cluster2-client.mirror.104197.log.gz 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.106545.log.gz 2024-09-11T06:53:17.686 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-11T06:53:17.686 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.131443.log.gz 2024-09-11T06:53:17.686 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129350.log 2024-09-11T06:53:17.687 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56628.log 2024-09-11T06:53:17.688 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.113567.log: /var/log/ceph/cluster2-client.mirror.129350.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.113567.log.gz 2024-09-11T06:53:17.688 INFO:teuthology.orchestra.run.smithi112.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.129350.log.gz 2024-09-11T06:53:17.689 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123285.log 2024-09-11T06:53:17.689 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.56628.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75057.log 2024-09-11T06:53:17.689 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.123285.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107402.log 2024-09-11T06:53:17.689 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.29415.log 2024-09-11T06:53:17.689 INFO:teuthology.orchestra.run.smithi112.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.123285.log.gz 2024-09-11T06:53:17.690 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.75057.log: 93.2% -- replaced with /var/log/ceph/cluster1-client.mirror.82150.log.gz 2024-09-11T06:53:17.690 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.107402.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114828.log 2024-09-11T06:53:17.691 INFO:teuthology.orchestra.run.smithi112.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.107402.log.gz 2024-09-11T06:53:17.692 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.29066.log 2024-09-11T06:53:17.692 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.29415.log: /var/log/ceph/cluster2-client.mirror.114828.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41990.log 2024-09-11T06:53:17.692 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.admin.29066.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.114828.log.gz 2024-09-11T06:53:17.693 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65875.log 2024-09-11T06:53:17.693 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.29066.log.gz 2024-09-11T06:53:17.693 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102676.log 2024-09-11T06:53:17.693 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.41990.log: /var/log/ceph/cluster2-client.mirror.65875.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109151.log 2024-09-11T06:53:17.693 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.102676.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30969.log 2024-09-11T06:53:17.693 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.109151.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.109151.log.gz 2024-09-11T06:53:17.695 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.48252.log.gz 2024-09-11T06:53:17.695 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48442.log.gz 2024-09-11T06:53:17.696 INFO:teuthology.orchestra.run.smithi112.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.33395.log.gz 2024-09-11T06:53:17.696 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.48287.log.gz 2024-09-11T06:53:17.697 INFO:teuthology.orchestra.run.smithi112.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.35546.log.gz 2024-09-11T06:53:17.697 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45061.log.gz 2024-09-11T06:53:17.698 INFO:teuthology.orchestra.run.smithi112.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.38123.log.gz 2024-09-11T06:53:17.698 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110293.log 2024-09-11T06:53:17.698 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38592.log 2024-09-11T06:53:17.699 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.30969.log: /var/log/ceph/cluster2-client.mirror.110293.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128323.log 2024-09-11T06:53:17.699 INFO:teuthology.orchestra.run.smithi112.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.110293.log.gz 2024-09-11T06:53:17.700 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.38592.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123121.log 2024-09-11T06:53:17.700 INFO:teuthology.orchestra.run.smithi112.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.39913.log.gz 2024-09-11T06:53:17.701 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.43848.log.gz 2024-09-11T06:53:17.701 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.128323.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.128323.log.gz 2024-09-11T06:53:17.702 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129017.log 2024-09-11T06:53:17.702 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.56570.log.gz 2024-09-11T06:53:17.702 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.56628.log.gz 2024-09-11T06:53:17.702 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.61205.log.gz 2024-09-11T06:53:17.702 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127549.log 2024-09-11T06:53:17.703 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.123121.log: /var/log/ceph/cluster2-client.mirror.129017.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.30969.log.gz 2024-09-11T06:53:17.704 INFO:teuthology.orchestra.run.smithi112.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.40568.log.gz 2024-09-11T06:53:17.704 INFO:teuthology.orchestra.run.smithi112.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.40754.log.gz 2024-09-11T06:53:17.704 INFO:teuthology.orchestra.run.smithi112.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.123121.log.gz 2024-09-11T06:53:17.704 INFO:teuthology.orchestra.run.smithi112.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.41258.log.gz 2024-09-11T06:53:17.704 INFO:teuthology.orchestra.run.smithi112.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.73716.log.gz 2024-09-11T06:53:17.705 INFO:teuthology.orchestra.run.smithi112.stderr: 91.8% -- replaced with /var/log/ceph/cluster2-client.mirror.129017.log.gz 2024-09-11T06:53:17.705 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster1-client.admin.42281.log.gz 2024-09-11T06:53:17.705 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.50252.log.gz 2024-09-11T06:53:17.705 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.54888.log.gz 2024-09-11T06:53:17.706 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.127549.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.127549.log.gz 2024-09-11T06:53:17.707 INFO:teuthology.orchestra.run.smithi112.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.57655.log.gz 2024-09-11T06:53:17.708 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.57938.log.gz 2024-09-11T06:53:17.708 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29313.log 2024-09-11T06:53:17.709 INFO:teuthology.orchestra.run.smithi112.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.43530.log.gz 2024-09-11T06:53:17.709 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47729.log.gz -- replaced with /var/log/ceph/cluster2-client.mirror.47787.log.gz 2024-09-11T06:53:17.709 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-11T06:53:17.709 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.57469.log.gz 2024-09-11T06:53:17.710 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.76925.log.gz 2024-09-11T06:53:17.710 INFO:teuthology.orchestra.run.smithi112.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.71414.log.gz 2024-09-11T06:53:17.710 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79165.log 2024-09-11T06:53:17.710 INFO:teuthology.orchestra.run.smithi112.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.81759.log.gz 2024-09-11T06:53:17.711 INFO:teuthology.orchestra.run.smithi112.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.83360.log.gz 2024-09-11T06:53:17.711 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48101.log.gz 2024-09-11T06:53:17.711 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60740.log 2024-09-11T06:53:17.711 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.79165.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.48384.log.gz 2024-09-11T06:53:17.711 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.admin.29313.log: 88.5% 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.55415.log.gz -- replaced with /var/log/ceph/cluster2-client.mirror.55450.log.gz 2024-09-11T06:53:17.711 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-11T06:53:17.711 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.60120.log.gz 2024-09-11T06:53:17.712 INFO:teuthology.orchestra.run.smithi112.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.72127.log.gz 2024-09-11T06:53:17.712 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.74592.log.gz 2024-09-11T06:53:17.712 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.75057.log.gz 2024-09-11T06:53:17.714 INFO:teuthology.orchestra.run.smithi112.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.75619.log.gz 2024-09-11T06:53:17.714 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124843.log 2024-09-11T06:53:17.715 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.60740.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34422.log 2024-09-11T06:53:17.715 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66716.log 2024-09-11T06:53:17.716 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.124843.log: /var/log/ceph/cluster2-client.mirror.34422.log: 88.2%gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28300.log 2024-09-11T06:53:17.717 INFO:teuthology.orchestra.run.smithi112.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.124843.log.gz 2024-09-11T06:53:17.717 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.66716.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86385.log 2024-09-11T06:53:17.718 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102434.log 2024-09-11T06:53:17.718 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.admin.28300.log: /var/log/ceph/cluster2-client.mirror.86385.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.28300.log.gz 2024-09-11T06:53:17.718 INFO:teuthology.orchestra.run.smithi112.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.42596.log.gz 2024-09-11T06:53:17.718 INFO:teuthology.orchestra.run.smithi112.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.81569.log.gz 2024-09-11T06:53:17.718 INFO:teuthology.orchestra.run.smithi112.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.73375.log.gz 2024-09-11T06:53:17.719 INFO:teuthology.orchestra.run.smithi112.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.81476.log.gz 2024-09-11T06:53:17.719 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116627.log 2024-09-11T06:53:17.720 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.102434.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126491.log 2024-09-11T06:53:17.720 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.116627.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.116627.log.gz 2024-09-11T06:53:17.721 INFO:teuthology.orchestra.run.smithi112.stderr: 92.0% -- replaced with /var/log/ceph/cluster2-client.mirror.102434.log.gz 2024-09-11T06:53:17.725 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.80173.log 2024-09-11T06:53:17.725 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.126491.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105305.log 2024-09-11T06:53:17.726 INFO:teuthology.orchestra.run.smithi112.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.126491.log.gz 2024-09-11T06:53:17.726 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122717.log 2024-09-11T06:53:17.726 INFO:teuthology.orchestra.run.smithi112.stderr: 84.7% -- replaced with /var/log/ceph/cluster2-client.mirror.87662.log.gz 2024-09-11T06:53:17.726 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.105305.log: /var/log/ceph/cluster1-client.admin.80173.log: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.105305.log.gz 2024-09-11T06:53:17.726 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109450.log 2024-09-11T06:53:17.727 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.122717.log: 89.2%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103411.log 2024-09-11T06:53:17.727 INFO:teuthology.orchestra.run.smithi112.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.122717.log.gz 2024-09-11T06:53:17.727 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.109450.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123334.log 2024-09-11T06:53:17.728 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.103411.log: 91.7% -- replaced with /var/log/ceph/cluster1-client.mirror.103411.log.gz 2024-09-11T06:53:17.729 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115206.log 2024-09-11T06:53:17.729 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.123334.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.123334.log.gz 2024-09-11T06:53:17.730 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36852.log 2024-09-11T06:53:17.730 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.115206.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.115206.log.gz 2024-09-11T06:53:17.731 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91315.log 2024-09-11T06:53:17.731 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.36852.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91338.log 2024-09-11T06:53:17.732 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.91315.log: 96.5% -- replaced with /var/log/ceph/cluster2-client.mirror.109450.log.gz 2024-09-11T06:53:17.732 INFO:teuthology.orchestra.run.smithi112.stderr: 84.4% -- replaced with /var/log/ceph/cluster2-client.mirror.91315.log.gz 2024-09-11T06:53:17.735 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128805.log 2024-09-11T06:53:17.735 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.91338.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.90390.log 2024-09-11T06:53:17.740 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.128805.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.36480.log.gz 2024-09-11T06:53:17.740 INFO:teuthology.orchestra.run.smithi112.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.30269.log.gz 2024-09-11T06:53:17.741 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47822.log.gz 2024-09-11T06:53:17.741 INFO:teuthology.orchestra.run.smithi112.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.36852.log.gz 2024-09-11T06:53:17.746 INFO:teuthology.orchestra.run.smithi112.stderr: 69.3% -- replaced with /var/log/ceph/cluster1-client.admin.29313.log.gz 2024-09-11T06:53:17.747 INFO:teuthology.orchestra.run.smithi112.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.79165.log.gz 2024-09-11T06:53:17.748 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118289.log 2024-09-11T06:53:17.749 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.90390.log: 84.7% -- replaced with /var/log/ceph/cluster2-client.mirror.90390.log.gz 2024-09-11T06:53:17.749 INFO:teuthology.orchestra.run.smithi112.stderr: 88.7% -- replaced with /var/log/ceph/cluster1-client.mirror.85775.log.gz 2024-09-11T06:53:17.753 INFO:teuthology.orchestra.run.smithi112.stderr: 95.5% -- replaced with /var/log/ceph/cluster2-client.mirror.90572.log.gz 93.2% -- replaced with /var/log/ceph/cluster1-client.mirror.80900.log.gz 2024-09-11T06:53:17.753 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-11T06:53:17.753 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125861.log 2024-09-11T06:53:17.755 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.118289.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.118289.log.gz 2024-09-11T06:53:17.755 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128757.log 2024-09-11T06:53:17.756 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.125861.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.125861.log.gz 2024-09-11T06:53:17.756 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29255.log 2024-09-11T06:53:17.757 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.128757.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.128757.log.gz 2024-09-11T06:53:17.758 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69453.log 2024-09-11T06:53:17.758 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.admin.29255.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29255.log.gz 2024-09-11T06:53:17.758 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123072.log 2024-09-11T06:53:17.759 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.69453.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54608.log 2024-09-11T06:53:17.760 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.123072.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.123072.log.gz 2024-09-11T06:53:17.760 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36724.log 2024-09-11T06:53:17.760 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.54608.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62550.log 2024-09-11T06:53:17.761 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.36724.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70259.log 2024-09-11T06:53:17.762 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.62550.log: 96.5% -- replaced with /var/log/ceph/cluster2-client.mirror.39070.log.gz 2024-09-11T06:53:17.762 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122668.log 2024-09-11T06:53:17.764 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.70259.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122597.log 2024-09-11T06:53:17.765 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.122668.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.122668.log.gz 2024-09-11T06:53:17.765 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86920.log 2024-09-11T06:53:17.766 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.122597.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.122597.log.gz 2024-09-11T06:53:17.767 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129143.log 2024-09-11T06:53:17.767 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.86920.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28359.log 2024-09-11T06:53:17.768 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.129143.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.129143.log.gz 2024-09-11T06:53:17.770 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126266.log 2024-09-11T06:53:17.770 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.admin.28359.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.28359.log.gz 2024-09-11T06:53:17.771 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84329.log 2024-09-11T06:53:17.771 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.126266.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.126266.log.gz 2024-09-11T06:53:17.772 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81030.log 2024-09-11T06:53:17.772 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.84329.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55074.log 2024-09-11T06:53:17.773 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.81030.log: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.38592.log.gz 2024-09-11T06:53:17.773 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.46481.log.gz 2024-09-11T06:53:17.773 INFO:teuthology.orchestra.run.smithi112.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.58438.log.gz 2024-09-11T06:53:17.774 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122830.log 2024-09-11T06:53:17.775 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.55074.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124153.log 2024-09-11T06:53:17.778 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.122830.log: 92.8% -- replaced with /var/log/ceph/cluster1-client.mirror.85911.log.gz 2024-09-11T06:53:17.778 INFO:teuthology.orchestra.run.smithi112.stderr: 90.8% -- replaced with /var/log/ceph/cluster2-client.mirror.122830.log.gz 2024-09-11T06:53:17.778 INFO:teuthology.orchestra.run.smithi112.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.40382.log.gz 2024-09-11T06:53:17.778 INFO:teuthology.orchestra.run.smithi112.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.73995.log.gz 2024-09-11T06:53:17.779 INFO:teuthology.orchestra.run.smithi112.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.34422.log.gz 2024-09-11T06:53:17.779 INFO:teuthology.orchestra.run.smithi112.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.34763.log.gz 2024-09-11T06:53:17.780 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86811.log 2024-09-11T06:53:17.780 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.67492.log.gz 2024-09-11T06:53:17.780 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.124153.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.124153.log.gz 2024-09-11T06:53:17.780 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55880.log 2024-09-11T06:53:17.781 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.86811.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46028.log 2024-09-11T06:53:17.782 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.55880.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86655.log 2024-09-11T06:53:17.782 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.46028.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111500.log 2024-09-11T06:53:17.783 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.86655.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34891.log 2024-09-11T06:53:17.785 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.111500.log: 91.8% -- replaced with /var/log/ceph/cluster2-client.mirror.111500.log.gz 2024-09-11T06:53:17.786 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85135.log 2024-09-11T06:53:17.786 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.34891.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43193.log 2024-09-11T06:53:17.787 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.85135.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101928.log 2024-09-11T06:53:17.787 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.43193.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107265.log 2024-09-11T06:53:17.788 INFO:teuthology.orchestra.run.smithi112.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.37507.log.gz 2024-09-11T06:53:17.788 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.37658.log.gz 2024-09-11T06:53:17.788 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.101928.log: 1.3% -- replaced with /var/log/ceph/cluster2-client.mirror.101928.log.gz 2024-09-11T06:53:17.788 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.29449.log 2024-09-11T06:53:17.789 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.107265.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.54608.log.gz 2024-09-11T06:53:17.791 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.54702.log.gz 2024-09-11T06:53:17.791 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47880.log.gz 2024-09-11T06:53:17.791 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.47260.log.gz 2024-09-11T06:53:17.791 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.71007.log.gz -- replaced with /var/log/ceph/cluster2-client.mirror.45870.log.gz 2024-09-11T06:53:17.791 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-11T06:53:17.792 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49128.log.gz 2024-09-11T06:53:17.792 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.79727.log.gz -- replaced with /var/log/ceph/cluster2-client.mirror.79692.log.gz 2024-09-11T06:53:17.792 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-11T06:53:17.792 INFO:teuthology.orchestra.run.smithi112.stderr: 92.9% -- replaced with /var/log/ceph/cluster2-client.mirror.39028.log.gz 2024-09-11T06:53:17.793 INFO:teuthology.orchestra.run.smithi112.stderr: 88.9% -- replaced with /var/log/ceph/cluster1-client.admin.80173.log.gz 2024-09-11T06:53:17.793 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48349.log.gz 2024-09-11T06:53:17.793 INFO:teuthology.orchestra.run.smithi112.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.60740.log.gz 2024-09-11T06:53:17.793 INFO:teuthology.orchestra.run.smithi112.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.61988.log.gz 2024-09-11T06:53:17.794 INFO:teuthology.orchestra.run.smithi112.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.62046.log.gz 2024-09-11T06:53:17.794 INFO:teuthology.orchestra.run.smithi112.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.68426.log.gz 2024-09-11T06:53:17.795 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.70294.log.gz 2024-09-11T06:53:17.795 INFO:teuthology.orchestra.run.smithi112.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.70259.log.gz 2024-09-11T06:53:17.795 INFO:teuthology.orchestra.run.smithi112.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.70387.log.gz 2024-09-11T06:53:17.795 INFO:teuthology.orchestra.run.smithi112.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.60027.log.gz 2024-09-11T06:53:17.796 INFO:teuthology.orchestra.run.smithi112.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.66251.log.gz 2024-09-11T06:53:17.796 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.63542.log.gz 2024-09-11T06:53:17.796 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.68798.log.gz 2024-09-11T06:53:17.796 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.66286.log.gz 2024-09-11T06:53:17.796 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.69453.log.gz 2024-09-11T06:53:17.796 INFO:teuthology.orchestra.run.smithi112.stderr: 2.3% -- replaced with /var/log/ceph/cluster1-client.mirror.29364.log.gz 2024-09-11T06:53:17.798 INFO:teuthology.orchestra.run.smithi112.stderr: 95.8% -- replaced with /var/log/ceph/cluster2-client.mirror.107265.log.gz 80.1% -- replaced with /var/log/ceph/cluster2-client.admin.29501.log.gz 2024-09-11T06:53:17.799 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-11T06:53:17.800 INFO:teuthology.orchestra.run.smithi112.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.35883.log.gz 2024-09-11T06:53:17.801 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44406.log 2024-09-11T06:53:17.801 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.admin.29449.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.29449.log.gz 2024-09-11T06:53:17.801 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.29569.log 2024-09-11T06:53:17.802 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.44406.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93155.log 2024-09-11T06:53:17.802 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.29569.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58589.log 2024-09-11T06:53:17.804 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.93155.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.81325.log.gz 2024-09-11T06:53:17.804 INFO:teuthology.orchestra.run.smithi112.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.82554.log.gz 2024-09-11T06:53:17.804 INFO:teuthology.orchestra.run.smithi112.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.84236.log.gz 2024-09-11T06:53:17.805 INFO:teuthology.orchestra.run.smithi112.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.84612.log.gz 2024-09-11T06:53:17.826 INFO:teuthology.orchestra.run.smithi112.stderr: 96.0% -- replaced with /var/log/ceph/cluster2-client.mirror.95384.log.gz 2024-09-11T06:53:17.826 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86405.log 2024-09-11T06:53:17.834 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.58589.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67771.log 2024-09-11T06:53:17.835 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.86405.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86522.log 2024-09-11T06:53:17.836 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.67771.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.119031.log 2024-09-11T06:53:17.836 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.86522.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48008.log 2024-09-11T06:53:17.837 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.119031.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.66716.log.gz 2024-09-11T06:53:17.839 INFO:teuthology.orchestra.run.smithi112.stderr: 91.8% -- replaced with /var/log/ceph/cluster1-client.mirror.119031.log.gz 2024-09-11T06:53:17.839 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62701.log 2024-09-11T06:53:17.840 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.48008.log: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.96312.log.gz 2024-09-11T06:53:17.840 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35604.log 2024-09-11T06:53:17.840 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.62701.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41595.log 2024-09-11T06:53:17.841 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59934.log 2024-09-11T06:53:17.841 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.35604.log: /var/log/ceph/cluster2-client.mirror.41595.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31027.log 2024-09-11T06:53:17.842 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.59934.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81418.log 2024-09-11T06:53:17.842 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.31027.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124610.log 2024-09-11T06:53:17.843 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124468.log 2024-09-11T06:53:17.843 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.81418.log: /var/log/ceph/cluster2-client.mirror.124610.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125681.log 2024-09-11T06:53:17.845 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.124468.log: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.124610.log.gz 2024-09-11T06:53:17.846 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77824.log 2024-09-11T06:53:17.846 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.125681.log: 90.9% -- replaced with /var/log/ceph/cluster2-client.mirror.124468.log.gz 2024-09-11T06:53:17.846 INFO:teuthology.orchestra.run.smithi112.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.125681.log.gz 2024-09-11T06:53:17.847 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43065.log 2024-09-11T06:53:17.847 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129911.log 2024-09-11T06:53:17.848 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.77824.log: /var/log/ceph/cluster2-client.mirror.43065.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66065.log 2024-09-11T06:53:17.848 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.129911.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.129911.log.gz 2024-09-11T06:53:17.848 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39634.log 2024-09-11T06:53:17.849 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.66065.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75522.log 2024-09-11T06:53:17.850 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100707.log 2024-09-11T06:53:17.850 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.39634.log: /var/log/ceph/cluster2-client.mirror.75522.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43658.log 2024-09-11T06:53:17.852 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45433.log.gz 2024-09-11T06:53:17.852 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.100707.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.100707.log.gz 2024-09-11T06:53:17.852 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128738.log 2024-09-11T06:53:17.852 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.38158.log.gz 2024-09-11T06:53:17.852 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.43658.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.40103.log.gz 2024-09-11T06:53:17.853 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49748.log 2024-09-11T06:53:17.853 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.44406.log.gz 2024-09-11T06:53:17.853 INFO:teuthology.orchestra.run.smithi112.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.61298.log.gz 2024-09-11T06:53:17.854 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.128738.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.58682.log.gz 2024-09-11T06:53:17.854 INFO:teuthology.orchestra.run.smithi112.stderr: 83.5% -- replaced with /var/log/ceph/cluster2-client.mirror.128738.log.gz 2024-09-11T06:53:17.855 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42099.log 2024-09-11T06:53:17.855 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114356.log 2024-09-11T06:53:17.855 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.49748.log: /var/log/ceph/cluster1-client.mirror.42099.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68112.log 2024-09-11T06:53:17.856 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78944.log 2024-09-11T06:53:17.859 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.114356.log: /var/log/ceph/cluster2-client.mirror.68112.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51616.log 2024-09-11T06:53:17.859 INFO:teuthology.orchestra.run.smithi112.stderr: 91.6% -- replaced with /var/log/ceph/cluster2-client.mirror.114356.log.gz 2024-09-11T06:53:17.859 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.78944.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104939.log 2024-09-11T06:53:17.859 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127594.log 2024-09-11T06:53:17.861 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.51616.log: /var/log/ceph/cluster1-client.mirror.104939.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.41223.log.gz 2024-09-11T06:53:17.861 INFO:teuthology.orchestra.run.smithi112.stderr: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.104939.log.gz 2024-09-11T06:53:17.861 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.49318.log.gz 2024-09-11T06:53:17.861 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108765.log 2024-09-11T06:53:17.862 INFO:teuthology.orchestra.run.smithi112.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.55039.log.gz 2024-09-11T06:53:17.862 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.127594.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.127594.log.gz 2024-09-11T06:53:17.863 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52922.log 2024-09-11T06:53:17.863 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46667.log 2024-09-11T06:53:17.864 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.108765.log: /var/log/ceph/cluster2-client.mirror.52922.log: 88.3%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120314.log 2024-09-11T06:53:17.864 INFO:teuthology.orchestra.run.smithi112.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.108765.log.gz 2024-09-11T06:53:17.865 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.46667.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69174.log 2024-09-11T06:53:17.865 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.120314.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104541.log 2024-09-11T06:53:17.865 INFO:teuthology.orchestra.run.smithi112.stderr: 88.1%/var/log/ceph/cluster2-client.mirror.69174.log: -- replaced with /var/log/ceph/cluster1-client.mirror.120314.log.gz 2024-09-11T06:53:17.865 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129195.log 2024-09-11T06:53:17.866 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.104541.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80583.log 2024-09-11T06:53:17.868 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.129195.log: 92.0% -- replaced with /var/log/ceph/cluster2-client.mirror.104541.log.gz 2024-09-11T06:53:17.868 INFO:teuthology.orchestra.run.smithi112.stderr: 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.129195.log.gz 2024-09-11T06:53:17.869 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39855.log 2024-09-11T06:53:17.869 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.80583.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125884.log 2024-09-11T06:53:17.869 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85305.log 2024-09-11T06:53:17.869 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.39855.log: /var/log/ceph/cluster1-client.mirror.125884.log: gzip -5 86.7% --verbose -- -- replaced with /var/log/ceph/cluster1-client.mirror.125884.log.gz 2024-09-11T06:53:17.869 INFO:teuthology.orchestra.run.smithi112.stderr: /var/log/ceph/cluster2-client.mirror.76739.log 2024-09-11T06:53:17.870 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.85305.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64941.log 2024-09-11T06:53:17.870 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.76739.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49186.log 2024-09-11T06:53:17.871 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57225.log 2024-09-11T06:53:17.871 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.64941.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.43658.log.gz 2024-09-11T06:53:17.871 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.76983.log.gz 2024-09-11T06:53:17.873 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.49186.log: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.102676.log.gz 2024-09-11T06:53:17.873 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29231.log 2024-09-11T06:53:17.873 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.78572.log.gz 2024-09-11T06:53:17.873 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.57225.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80876.log 2024-09-11T06:53:17.873 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33360.log 2024-09-11T06:53:17.874 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.admin.29231.log: /var/log/ceph/cluster2-client.mirror.80876.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62081.log 2024-09-11T06:53:17.875 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29231.log.gz 2024-09-11T06:53:17.875 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.33360.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34798.log 2024-09-11T06:53:17.875 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.62081.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60589.log 2024-09-11T06:53:17.875 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105725.log 2024-09-11T06:53:17.875 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.78607.log.gz 2024-09-11T06:53:17.875 INFO:teuthology.orchestra.run.smithi112.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.85305.log.gz 2024-09-11T06:53:17.875 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.34798.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.61767.log.gz 2024-09-11T06:53:17.876 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129331.log 2024-09-11T06:53:17.876 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.60589.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.70201.log.gz 2024-09-11T06:53:17.876 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.105725.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.52922.log.gz 2024-09-11T06:53:17.876 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.53519.log.gz 2024-09-11T06:53:17.879 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121293.log 2024-09-11T06:53:17.879 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.129331.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68240.log 2024-09-11T06:53:17.879 INFO:teuthology.orchestra.run.smithi112.stderr: 83.6% -- replaced with /var/log/ceph/cluster2-client.mirror.129331.log.gz 2024-09-11T06:53:17.879 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.121293.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41502.log 2024-09-11T06:53:17.881 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.68240.log: 90.9% -- replaced with /var/log/ceph/cluster2-client.mirror.121293.log.gz 2024-09-11T06:53:17.881 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39448.log 2024-09-11T06:53:17.882 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.41502.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39669.log 2024-09-11T06:53:17.882 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129836.log 2024-09-11T06:53:17.883 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.39448.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104484.log 2024-09-11T06:53:17.884 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.39669.log: /var/log/ceph/cluster1-client.mirror.129836.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50624.log 2024-09-11T06:53:17.884 INFO:teuthology.orchestra.run.smithi112.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.129836.log.gz 2024-09-11T06:53:17.884 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57039.log 2024-09-11T06:53:17.885 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.104484.log: 87.9% 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.64941.log.gz 2024-09-11T06:53:17.885 INFO:teuthology.orchestra.run.smithi112.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.104484.log.gz 2024-09-11T06:53:17.885 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.50624.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111057.log 2024-09-11T06:53:17.885 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.57039.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129741.log 2024-09-11T06:53:17.886 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.111057.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.111057.log.gz 2024-09-11T06:53:17.888 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104063.log 2024-09-11T06:53:17.888 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.129741.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56256.log 2024-09-11T06:53:17.888 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.104063.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.129741.log.gz 2024-09-11T06:53:17.888 INFO:teuthology.orchestra.run.smithi112.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.104063.log.gz 2024-09-11T06:53:17.888 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58310.log 2024-09-11T06:53:17.888 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.56256.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125726.log 2024-09-11T06:53:17.889 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.58310.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121895.log 2024-09-11T06:53:17.889 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.125726.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76328.log 2024-09-11T06:53:17.889 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.121895.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.125726.log.gz 2024-09-11T06:53:17.889 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107880.log 2024-09-11T06:53:17.890 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.76328.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124355.log 2024-09-11T06:53:17.891 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.107880.log: 90.8% -- replaced with /var/log/ceph/cluster2-client.mirror.121895.log.gz 2024-09-11T06:53:17.892 INFO:teuthology.orchestra.run.smithi112.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.72034.log.gz 2024-09-11T06:53:17.892 INFO:teuthology.orchestra.run.smithi112.stderr: 95.9% -- replaced with /var/log/ceph/cluster2-client.mirror.90184.log.gz 2024-09-11T06:53:17.893 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86987.log 2024-09-11T06:53:17.895 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.124355.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.124355.log.gz 2024-09-11T06:53:17.895 INFO:teuthology.orchestra.run.smithi112.stderr: 92.2%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43100.log 2024-09-11T06:53:17.896 INFO:teuthology.orchestra.run.smithi112.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.107880.log.gz 2024-09-11T06:53:17.896 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46326.log 2024-09-11T06:53:17.896 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.86987.log: /var/log/ceph/cluster2-client.mirror.43100.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83678.log 2024-09-11T06:53:17.897 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.46326.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89291.log 2024-09-11T06:53:17.897 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.83678.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106998.log 2024-09-11T06:53:17.898 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.89291.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127661.log 2024-09-11T06:53:17.898 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.106998.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56105.log 2024-09-11T06:53:17.898 INFO:teuthology.orchestra.run.smithi112.stderr: 88.2%/var/log/ceph/cluster1-client.mirror.127661.log: -- replaced with /var/log/ceph/cluster1-client.mirror.106998.log.gz 2024-09-11T06:53:17.898 INFO:teuthology.orchestra.run.smithi112.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.127661.log.gz 2024-09-11T06:53:17.899 INFO:teuthology.orchestra.run.smithi112.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.75522.log.gz 2024-09-11T06:53:17.900 INFO:teuthology.orchestra.run.smithi112.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.76421.log.gz 2024-09-11T06:53:17.900 INFO:teuthology.orchestra.run.smithi112.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.59934.log.gz 2024-09-11T06:53:17.901 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86542.log 2024-09-11T06:53:17.901 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.56105.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36104.log 2024-09-11T06:53:17.902 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.60589.log.gz 2024-09-11T06:53:17.902 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83546.log 2024-09-11T06:53:17.902 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.86542.log: /var/log/ceph/cluster2-client.mirror.36104.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115592.log 2024-09-11T06:53:17.903 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122646.log 2024-09-11T06:53:17.903 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.83546.log: /var/log/ceph/cluster2-client.mirror.115592.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57597.log 2024-09-11T06:53:17.903 INFO:teuthology.orchestra.run.smithi112.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.115592.log.gz 2024-09-11T06:53:17.904 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.122646.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.122646.log.gz 2024-09-11T06:53:17.905 INFO:teuthology.orchestra.run.smithi112.stderr: 84.7% -- replaced with /var/log/ceph/cluster2-client.mirror.87765.log.gz 2024-09-11T06:53:17.906 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82358.log 2024-09-11T06:53:17.907 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.57597.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.30686.log.gz 2024-09-11T06:53:17.907 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.30783.log.gz 2024-09-11T06:53:17.908 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112007.log 2024-09-11T06:53:17.908 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.82358.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31120.log 2024-09-11T06:53:17.908 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.112007.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.112007.log.gz 2024-09-11T06:53:17.909 INFO:teuthology.orchestra.run.smithi112.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.59151.log.gz 2024-09-11T06:53:17.910 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55601.log 2024-09-11T06:53:17.910 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.31120.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62794.log 2024-09-11T06:53:17.911 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.55601.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121711.log 2024-09-11T06:53:17.912 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.62794.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70852.log 2024-09-11T06:53:17.912 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.121711.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122788.log 2024-09-11T06:53:17.912 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.70852.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48194.log 2024-09-11T06:53:17.912 INFO:teuthology.orchestra.run.smithi112.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.121711.log.gz 2024-09-11T06:53:17.913 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.122788.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127481.log 2024-09-11T06:53:17.913 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.48194.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51244.log 2024-09-11T06:53:17.913 INFO:teuthology.orchestra.run.smithi112.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.122788.log.gz 2024-09-11T06:53:17.914 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.127481.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.127481.log.gz 2024-09-11T06:53:17.917 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37286.log 2024-09-11T06:53:17.918 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.51244.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118675.log 2024-09-11T06:53:17.918 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.37286.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28841.log 2024-09-11T06:53:17.918 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.118675.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36538.log 2024-09-11T06:53:17.919 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.admin.28841.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.28841.log.gz 2024-09-11T06:53:17.920 INFO:teuthology.orchestra.run.smithi112.stderr: 92.9% -- replaced with /var/log/ceph/cluster2-client.mirror.85977.log.gz 2024-09-11T06:53:17.922 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48907.log 2024-09-11T06:53:17.922 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.36538.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67550.log 2024-09-11T06:53:17.922 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.48907.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82837.log 2024-09-11T06:53:17.923 INFO:teuthology.orchestra.run.smithi112.stderr: 96.1% -- replaced with /var/log/ceph/cluster2-client.mirror.118675.log.gz 2024-09-11T06:53:17.924 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.67550.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89937.log 2024-09-11T06:53:17.924 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.82837.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56442.log 2024-09-11T06:53:17.924 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.89937.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51558.log 2024-09-11T06:53:17.924 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.56442.log: 84.5% -- replaced with /var/log/ceph/cluster2-client.mirror.89937.log.gz 2024-09-11T06:53:17.926 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109992.log 2024-09-11T06:53:17.926 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.51558.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75487.log 2024-09-11T06:53:17.927 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.109992.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.68112.log.gz 2024-09-11T06:53:17.928 INFO:teuthology.orchestra.run.smithi112.stderr: 91.9% -- replaced with /var/log/ceph/cluster1-client.mirror.109992.log.gz 2024-09-11T06:53:17.928 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88533.log 2024-09-11T06:53:17.929 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.75487.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126806.log 2024-09-11T06:53:17.929 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.88533.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129310.log 2024-09-11T06:53:17.930 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.126806.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61054.log 2024-09-11T06:53:17.930 INFO:teuthology.orchestra.run.smithi112.stderr: 86.9%/var/log/ceph/cluster2-client.mirror.129310.log: -- replaced with /var/log/ceph/cluster1-client.mirror.126806.log.gz 2024-09-11T06:53:17.931 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30934.log 2024-09-11T06:53:17.931 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.61054.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51709.log 2024-09-11T06:53:17.933 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.30934.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38499.log 2024-09-11T06:53:17.933 INFO:teuthology.orchestra.run.smithi112.stderr: 91.8% 95.3% -- replaced with /var/log/ceph/cluster2-client.mirror.91338.log.gz 2024-09-11T06:53:17.933 INFO:teuthology.orchestra.run.smithi112.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.129310.log.gz/var/log/ceph/cluster2-client.mirror.51709.log: 2024-09-11T06:53:17.933 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47508.log 2024-09-11T06:53:17.933 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.38499.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117305.log 2024-09-11T06:53:17.934 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41909.log 2024-09-11T06:53:17.935 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.47508.log: /var/log/ceph/cluster2-client.mirror.117305.log: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.117305.log.gz 2024-09-11T06:53:17.935 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33081.log 2024-09-11T06:53:17.936 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.41909.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71472.log 2024-09-11T06:53:17.936 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.33081.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87008.log 2024-09-11T06:53:17.937 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.71472.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119011.log 2024-09-11T06:53:17.937 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.87008.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53015.log 2024-09-11T06:53:17.937 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119961.log 2024-09-11T06:53:17.938 INFO:teuthology.orchestra.run.smithi006.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-mon.a.log.gz 2024-09-11T06:53:17.938 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.119011.log: /var/log/ceph/cluster2-client.mirror.53015.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41537.log 2024-09-11T06:53:17.939 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.119961.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67585.log 2024-09-11T06:53:17.940 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.41537.log: 91.1% -- replaced with /var/log/ceph/cluster2-client.mirror.119961.log.gz 2024-09-11T06:53:17.940 INFO:teuthology.orchestra.run.smithi112.stderr: 92.1% -- replaced with /var/log/ceph/cluster2-client.mirror.119011.log.gz 2024-09-11T06:53:17.940 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125704.log 2024-09-11T06:53:17.940 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.67585.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48663.log 2024-09-11T06:53:17.942 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.125704.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122688.log 2024-09-11T06:53:17.942 INFO:teuthology.orchestra.run.smithi112.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.125704.log.gz 2024-09-11T06:53:17.942 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.48663.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.29757.log 2024-09-11T06:53:17.942 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129225.log 2024-09-11T06:53:17.942 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.122688.log: /var/log/ceph/cluster1-client.mirror.1.29757.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123598.log 2024-09-11T06:53:17.942 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.129225.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.129225.log.gz 2024-09-11T06:53:17.943 INFO:teuthology.orchestra.run.smithi112.stderr: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.122688.log.gz 2024-09-11T06:53:17.944 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39093.log 2024-09-11T06:53:17.944 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.123598.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43623.log 2024-09-11T06:53:17.946 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.39093.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.123598.log.gz 2024-09-11T06:53:17.946 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126964.log 2024-09-11T06:53:17.946 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64697.log 2024-09-11T06:53:17.946 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.43623.log: /var/log/ceph/cluster1-client.mirror.126964.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53902.log 2024-09-11T06:53:17.946 INFO:teuthology.orchestra.run.smithi112.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.126964.log.gz/var/log/ceph/cluster2-client.mirror.64697.log: 2024-09-11T06:53:17.946 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.25359.log 2024-09-11T06:53:17.946 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61488.log 2024-09-11T06:53:17.948 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.53902.log: /var/log/ceph/cluster2-client.admin.25359.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40231.log 2024-09-11T06:53:17.948 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.25359.log.gz 2024-09-11T06:53:17.948 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.61488.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88953.log 2024-09-11T06:53:17.948 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.40231.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54146.log 2024-09-11T06:53:17.948 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.88953.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121364.log 2024-09-11T06:53:17.949 INFO:teuthology.orchestra.run.smithi112.stderr: 84.5% -- replaced with /var/log/ceph/cluster2-client.mirror.88953.log.gz 2024-09-11T06:53:17.949 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.54146.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45212.log.gz 2024-09-11T06:53:17.949 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.121364.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45154.log.gz 2024-09-11T06:53:17.949 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82197.log 2024-09-11T06:53:17.950 INFO:teuthology.orchestra.run.smithi112.stderr: 90.8% -- replaced with /var/log/ceph/cluster2-client.mirror.121364.log.gz 2024-09-11T06:53:17.950 INFO:teuthology.orchestra.run.smithi112.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.38627.log.gz 2024-09-11T06:53:17.951 INFO:teuthology.orchestra.run.smithi112.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.39669.log.gz 2024-09-11T06:53:17.951 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.82197.log: 88.3% 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46667.log.gz -- replaced with /var/log/ceph/cluster2-client.mirror.52178.log.gz 2024-09-11T06:53:17.951 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-11T06:53:17.951 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.58159.log.gz 2024-09-11T06:53:17.952 INFO:teuthology.orchestra.run.smithi112.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.31120.log.gz 2024-09-11T06:53:17.952 INFO:teuthology.orchestra.run.smithi112.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.49186.log.gz 2024-09-11T06:53:17.952 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.38313.log.gz 2024-09-11T06:53:17.953 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.57225.log.gz 2024-09-11T06:53:17.953 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116534.log 2024-09-11T06:53:17.953 INFO:teuthology.orchestra.run.smithi112.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.57597.log.gz 2024-09-11T06:53:17.954 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71228.log 2024-09-11T06:53:17.954 INFO:teuthology.orchestra.run.smithi112.stderr: 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.42167.log.gz 2024-09-11T06:53:17.954 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.116534.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.116534.log.gz 2024-09-11T06:53:17.954 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93132.log 2024-09-11T06:53:17.955 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.71228.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117013.log 2024-09-11T06:53:17.955 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.93132.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57004.log 2024-09-11T06:53:17.955 INFO:teuthology.orchestra.run.smithi112.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.89726.log.gz 2024-09-11T06:53:17.955 INFO:teuthology.orchestra.run.smithi112.stderr: 84.4% -- replaced with /var/log/ceph/cluster2-client.mirror.93132.log.gz 2024-09-11T06:53:17.955 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.117013.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.117013.log.gz 2024-09-11T06:53:17.957 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59558.log 2024-09-11T06:53:17.957 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.57004.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125267.log 2024-09-11T06:53:17.957 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130019.log 2024-09-11T06:53:17.957 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.59558.log: /var/log/ceph/cluster2-client.mirror.125267.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62643.log 2024-09-11T06:53:17.957 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.130019.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119817.log 2024-09-11T06:53:17.958 INFO:teuthology.orchestra.run.smithi112.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.125267.log.gz 2024-09-11T06:53:17.958 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.62643.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130503.log 2024-09-11T06:53:17.959 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.119817.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69604.log 2024-09-11T06:53:17.960 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.130503.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75956.log 2024-09-11T06:53:17.960 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5%/var/log/ceph/cluster2-client.mirror.69604.log: -- replaced with /var/log/ceph/cluster2-client.mirror.130503.log.gz 2024-09-11T06:53:17.960 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47508.log.gz 2024-09-11T06:53:17.966 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.71472.log.gz 2024-09-11T06:53:17.966 INFO:teuthology.orchestra.run.smithi112.stderr: 96.6% -- replaced with /var/log/ceph/cluster2-client.mirror.119817.log.gz 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.55601.log.gz 2024-09-11T06:53:17.966 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-11T06:53:17.966 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73526.log 2024-09-11T06:53:17.966 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44592.log 2024-09-11T06:53:17.967 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.75956.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117325.log 2024-09-11T06:53:17.967 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.73526.log: /var/log/ceph/cluster2-client.mirror.44592.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129437.log 2024-09-11T06:53:17.968 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.117325.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126694.log 2024-09-11T06:53:17.968 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.129437.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129079.log 2024-09-11T06:53:17.971 INFO:teuthology.orchestra.run.smithi112.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.129437.log.gz/var/log/ceph/cluster1-client.mirror.126694.log: 2024-09-11T06:53:17.971 INFO:teuthology.orchestra.run.smithi112.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.126694.log.gz 2024-09-11T06:53:17.971 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60085.log 2024-09-11T06:53:17.971 INFO:teuthology.orchestra.run.smithi112.stderr: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.117325.log.gz 2024-09-11T06:53:17.971 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82105.log 2024-09-11T06:53:17.971 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.129079.log: /var/log/ceph/cluster2-client.mirror.60085.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78103.log 2024-09-11T06:53:17.971 INFO:teuthology.orchestra.run.smithi112.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.129079.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51465.log 2024-09-11T06:53:17.971 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-11T06:53:17.971 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.82105.log: /var/log/ceph/cluster2-client.mirror.78103.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122575.log 2024-09-11T06:53:17.971 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.51465.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66937.log 2024-09-11T06:53:17.972 INFO:teuthology.orchestra.run.smithi112.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.62081.log.gz 2024-09-11T06:53:17.972 INFO:teuthology.orchestra.run.smithi112.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.61953.log.gz 2024-09-11T06:53:17.973 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.122575.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.122575.log.gz 2024-09-11T06:53:17.973 INFO:teuthology.orchestra.run.smithi112.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.68240.log.gz 2024-09-11T06:53:17.973 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28437.log 2024-09-11T06:53:17.973 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.66937.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102193.log 2024-09-11T06:53:17.973 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.admin.28437.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78758.log 2024-09-11T06:53:17.973 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.28437.log.gz 2024-09-11T06:53:17.974 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.102193.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109606.log 2024-09-11T06:53:17.975 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.78758.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.54146.log.gz 2024-09-11T06:53:17.975 INFO:teuthology.orchestra.run.smithi112.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.59186.log.gz 2024-09-11T06:53:17.975 INFO:teuthology.orchestra.run.smithi112.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.59430.log.gz 2024-09-11T06:53:17.977 INFO:teuthology.orchestra.run.smithi112.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.69604.log.gz 2024-09-11T06:53:17.977 INFO:teuthology.orchestra.run.smithi112.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.66937.log.gz 2024-09-11T06:53:17.978 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.71693.log.gz 2024-09-11T06:53:17.978 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.72441.log.gz 2024-09-11T06:53:17.978 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.75487.log.gz 2024-09-11T06:53:17.979 INFO:teuthology.orchestra.run.smithi112.stderr: 96.7% -- replaced with /var/log/ceph/cluster2-client.mirror.102193.log.gz 2024-09-11T06:53:17.980 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69922.log 2024-09-11T06:53:17.980 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.109606.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117391.log 2024-09-11T06:53:17.983 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.69922.log: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.109606.log.gz 2024-09-11T06:53:17.983 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33643.log 2024-09-11T06:53:17.983 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.117391.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130168.log 2024-09-11T06:53:17.983 INFO:teuthology.orchestra.run.smithi112.stderr: 88.0%/var/log/ceph/cluster2-client.mirror.33643.log: -- replaced with /var/log/ceph/cluster2-client.mirror.117391.log.gz 2024-09-11T06:53:17.984 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86723.log 2024-09-11T06:53:17.984 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.130168.log: gzip 86.9% -5 -- replaced with /var/log/ceph/cluster1-client.mirror.130168.log.gz --verbose 2024-09-11T06:53:17.984 INFO:teuthology.orchestra.run.smithi112.stderr: -- /var/log/ceph/cluster2-client.mirror.106679.log 2024-09-11T06:53:17.984 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.86723.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121415.log 2024-09-11T06:53:17.984 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.106679.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75584.log 2024-09-11T06:53:17.985 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.121415.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.121415.log.gz 2024-09-11T06:53:17.985 INFO:teuthology.orchestra.run.smithi112.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.73282.log.gz 2024-09-11T06:53:17.987 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50345.log 2024-09-11T06:53:17.988 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.75584.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65224.log 2024-09-11T06:53:17.989 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.50345.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37038.log 2024-09-11T06:53:17.989 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.65224.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46609.log 2024-09-11T06:53:17.990 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.37038.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113248.log 2024-09-11T06:53:17.990 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.46609.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51651.log 2024-09-11T06:53:17.991 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129993.log 2024-09-11T06:53:17.993 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.113248.log: /var/log/ceph/cluster2-client.mirror.51651.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.36104.log.gz 2024-09-11T06:53:17.995 INFO:teuthology.orchestra.run.smithi112.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.59558.log.gz 2024-09-11T06:53:17.995 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79355.log 2024-09-11T06:53:17.995 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.129993.log: 95.5% -- replaced with /var/log/ceph/cluster2-client.mirror.113248.log.gz 2024-09-11T06:53:17.995 INFO:teuthology.orchestra.run.smithi112.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.129993.log.gz 2024-09-11T06:53:17.995 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69511.log 2024-09-11T06:53:17.996 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.79355.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56163.log 2024-09-11T06:53:17.996 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124426.log 2024-09-11T06:53:17.997 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.69511.log: /var/log/ceph/cluster2-client.mirror.56163.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91782.log 2024-09-11T06:53:17.997 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78514.log 2024-09-11T06:53:17.998 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.124426.log: /var/log/ceph/cluster2-client.mirror.91782.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.124426.log.gz 2024-09-11T06:53:18.001 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70352.log 2024-09-11T06:53:18.006 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.78514.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130898.log 2024-09-11T06:53:18.007 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.70352.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59655.log 2024-09-11T06:53:18.007 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.130898.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.130898.log.gz 2024-09-11T06:53:18.007 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61860.log 2024-09-11T06:53:18.008 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.59655.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127920.log 2024-09-11T06:53:18.009 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.61860.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109424.log 2024-09-11T06:53:18.011 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.127920.log: 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.127920.log.gz 2024-09-11T06:53:18.011 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119439.log 2024-09-11T06:53:18.013 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.109424.log: 92.0% -- replaced with /var/log/ceph/cluster1-client.mirror.109424.log.gz 2024-09-11T06:53:18.015 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32461.log 2024-09-11T06:53:18.018 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.119439.log: 95.5% -- replaced with /var/log/ceph/cluster2-client.mirror.119439.log.gz 2024-09-11T06:53:18.019 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.131261.log 2024-09-11T06:53:18.019 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.71228.log.gz 2024-09-11T06:53:18.019 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.32461.log: 84.7% -- replaced with /var/log/ceph/cluster2-client.mirror.87503.log.gz 2024-09-11T06:53:18.023 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130820.log 2024-09-11T06:53:18.023 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.admin.131261.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.131261.log.gz 2024-09-11T06:53:18.023 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43251.log 2024-09-11T06:53:18.023 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.130820.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.56105.log.gz 2024-09-11T06:53:18.025 INFO:teuthology.orchestra.run.smithi112.stderr: 91.9% -- replaced with /var/log/ceph/cluster2-client.mirror.130820.log.gz 2024-09-11T06:53:18.025 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80660.log 2024-09-11T06:53:18.026 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.43251.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74088.log 2024-09-11T06:53:18.027 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.80660.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72720.log 2024-09-11T06:53:18.027 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.74088.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124124.log 2024-09-11T06:53:18.028 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.72720.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91479.log 2024-09-11T06:53:18.030 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.124124.log: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.124124.log.gz 2024-09-11T06:53:18.030 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89433.log 2024-09-11T06:53:18.031 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.91479.log: 84.4% -- replaced with /var/log/ceph/cluster2-client.mirror.91479.log.gz 2024-09-11T06:53:18.032 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46120.log 2024-09-11T06:53:18.041 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.89433.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45026.log.gz 2024-09-11T06:53:18.046 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77766.log 2024-09-11T06:53:18.049 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.46120.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.57004.log.gz 2024-09-11T06:53:18.050 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.40789.log.gz 2024-09-11T06:53:18.050 INFO:teuthology.orchestra.run.smithi112.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.32461.log.gz 2024-09-11T06:53:18.051 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.41409.log.gz 2024-09-11T06:53:18.051 INFO:teuthology.orchestra.run.smithi112.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.51465.log.gz 2024-09-11T06:53:18.052 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.73844.log.gz 2024-09-11T06:53:18.052 INFO:teuthology.orchestra.run.smithi112.stderr: 95.9% -- replaced with /var/log/ceph/cluster2-client.mirror.89291.log.gz 2024-09-11T06:53:18.053 INFO:teuthology.orchestra.run.smithi112.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.34705.log.gz 2024-09-11T06:53:18.053 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121178.log 2024-09-11T06:53:18.054 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.77766.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31961.log 2024-09-11T06:53:18.054 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.121178.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.37751.log.gz 2024-09-11T06:53:18.055 INFO:teuthology.orchestra.run.smithi112.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.55074.log.gz 2024-09-11T06:53:18.056 INFO:teuthology.orchestra.run.smithi112.stderr: 92.7% -- replaced with /var/log/ceph/cluster1-client.mirror.42191.log.gz 2024-09-11T06:53:18.057 INFO:teuthology.orchestra.run.smithi112.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.121178.log.gz 2024-09-11T06:53:18.057 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38255.log 2024-09-11T06:53:18.058 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.31961.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50961.log 2024-09-11T06:53:18.059 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.38255.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.43251.log.gz 2024-09-11T06:53:18.059 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116391.log 2024-09-11T06:53:18.059 INFO:teuthology.orchestra.run.smithi112.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.76832.log.gz 2024-09-11T06:53:18.059 INFO:teuthology.orchestra.run.smithi112.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.77766.log.gz 2024-09-11T06:53:18.060 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.50961.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.79355.log.gz 2024-09-11T06:53:18.063 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster1-client.mirror.82105.log.gz 2024-09-11T06:53:18.063 INFO:teuthology.orchestra.run.smithi112.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.69922.log.gz 2024-09-11T06:53:18.063 INFO:teuthology.orchestra.run.smithi112.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.85170.log.gz 2024-09-11T06:53:18.063 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.52829.log.gz 2024-09-11T06:53:18.063 INFO:teuthology.orchestra.run.smithi112.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.66065.log.gz 2024-09-11T06:53:18.063 INFO:teuthology.orchestra.run.smithi112.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.59655.log.gz 2024-09-11T06:53:18.063 INFO:teuthology.orchestra.run.smithi112.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.64662.log.gz 2024-09-11T06:53:18.063 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.69511.log.gz 2024-09-11T06:53:18.064 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.75584.log.gz 2024-09-11T06:53:18.064 INFO:teuthology.orchestra.run.smithi112.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.75991.log.gz 2024-09-11T06:53:18.064 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.29398.log 2024-09-11T06:53:18.065 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.116391.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.116391.log.gz 2024-09-11T06:53:18.065 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49469.log 2024-09-11T06:53:18.065 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.29398.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128496.log 2024-09-11T06:53:18.066 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.49469.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30500.log 2024-09-11T06:53:18.074 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.128496.log: 57.7% 56.0% -- replaced with /var/log/ceph/cluster1-client.mirror.29415.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.29398.log.gz 2024-09-11T06:53:18.074 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-11T06:53:18.074 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.73061.log.gz 2024-09-11T06:53:18.074 INFO:teuthology.orchestra.run.smithi112.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.84329.log.gz 2024-09-11T06:53:18.085 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39820.log 2024-09-11T06:53:18.096 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.30500.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34143.log 2024-09-11T06:53:18.107 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.39820.log: /var/log/ceph/cluster2-client.mirror.34143.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57283.log 2024-09-11T06:53:18.120 INFO:teuthology.orchestra.run.smithi112.stderr: 95.2% -- replaced with /var/log/ceph/cluster2-client.mirror.93155.log.gz 2024-09-11T06:53:18.120 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112870.log 2024-09-11T06:53:18.121 INFO:teuthology.orchestra.run.smithi112.stderr: 93.3% -- replaced with /var/log/ceph/cluster2-client.mirror.86565.log.gz 2024-09-11T06:53:18.122 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.57283.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44034.log 2024-09-11T06:53:18.123 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.112870.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.112870.log.gz 2024-09-11T06:53:18.124 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113592.log 2024-09-11T06:53:18.125 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.44034.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127301.log 2024-09-11T06:53:18.125 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.113592.log: 93.3% -- replaced with /var/log/ceph/cluster2-client.mirror.86342.log.gz 2024-09-11T06:53:18.126 INFO:teuthology.orchestra.run.smithi112.stderr: 91.7% -- replaced with /var/log/ceph/cluster2-client.mirror.113592.log.gz 2024-09-11T06:53:18.127 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52399.log 2024-09-11T06:53:18.127 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.127301.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.127301.log.gz 2024-09-11T06:53:18.128 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46574.log 2024-09-11T06:53:18.128 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.52399.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112484.log 2024-09-11T06:53:18.133 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.46574.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123982.log 2024-09-11T06:53:18.133 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.112484.log: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.128805.log.gz 2024-09-11T06:53:18.133 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37472.log 2024-09-11T06:53:18.134 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.123982.log: 95.9% -- replaced with /var/log/ceph/cluster2-client.mirror.112484.log.gz 2024-09-11T06:53:18.135 INFO:teuthology.orchestra.run.smithi112.stderr: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.123982.log.gz 2024-09-11T06:53:18.135 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61112.log 2024-09-11T06:53:18.136 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.37472.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122335.log 2024-09-11T06:53:18.136 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43755.log 2024-09-11T06:53:18.137 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.61112.log: /var/log/ceph/cluster2-client.mirror.122335.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82930.log 2024-09-11T06:53:18.137 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.43755.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49597.log 2024-09-11T06:53:18.138 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.82930.log: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.122335.log.gz 2024-09-11T06:53:18.138 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122617.log 2024-09-11T06:53:18.139 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73809.log 2024-09-11T06:53:18.139 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.49597.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123669.log 2024-09-11T06:53:18.140 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.122617.log: /var/log/ceph/cluster2-client.mirror.73809.log: 96.7% -- replaced with /var/log/ceph/cluster2-client.mirror.80922.log.gz 2024-09-11T06:53:18.141 INFO:teuthology.orchestra.run.smithi112.stderr: 90.8% -- replaced with /var/log/ceph/cluster2-client.mirror.122617.log.gz 2024-09-11T06:53:18.141 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.118608.log 2024-09-11T06:53:18.142 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107309.log 2024-09-11T06:53:18.143 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.123669.log: /var/log/ceph/cluster1-client.mirror.118608.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.123669.log.gz 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.106679.log.gz 2024-09-11T06:53:18.143 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-11T06:53:18.143 INFO:teuthology.orchestra.run.smithi112.stderr: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.118608.log.gz 2024-09-11T06:53:18.143 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35790.log 2024-09-11T06:53:18.143 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.107309.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57880.log 2024-09-11T06:53:18.145 INFO:teuthology.orchestra.run.smithi112.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.107309.log.gz/var/log/ceph/cluster2-client.mirror.35790.log: 2024-09-11T06:53:18.145 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108552.log 2024-09-11T06:53:18.146 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45491.log 2024-09-11T06:53:18.146 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.108552.log: /var/log/ceph/cluster2-client.mirror.57880.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77452.log 2024-09-11T06:53:18.146 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.45491.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.108552.log.gz 2024-09-11T06:53:18.146 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40510.log 2024-09-11T06:53:18.146 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.77452.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76890.log 2024-09-11T06:53:18.146 INFO:teuthology.orchestra.run.smithi112.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.105725.log.gz 2024-09-11T06:53:18.146 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.40510.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127144.log 2024-09-11T06:53:18.148 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.76890.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121875.log 2024-09-11T06:53:18.148 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.127144.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112462.log 2024-09-11T06:53:18.148 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.121875.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.127144.log.gz 2024-09-11T06:53:18.148 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74836.log 2024-09-11T06:53:18.148 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.112462.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.121875.log.gz 2024-09-11T06:53:18.148 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81197.log 2024-09-11T06:53:18.148 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.54017.log 2024-09-11T06:53:18.148 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.74836.log: /var/log/ceph/cluster2-client.mirror.81197.log: 91.8% -- replaced with /var/log/ceph/cluster1-client.mirror.112462.log.gz 2024-09-11T06:53:18.153 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28891.log 2024-09-11T06:53:18.153 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.admin.54017.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42144.log 2024-09-11T06:53:18.153 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86632.log 2024-09-11T06:53:18.153 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.admin.28891.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28891.log.gz 2024-09-11T06:53:18.154 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.42144.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124772.log 2024-09-11T06:53:18.154 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109880.log 2024-09-11T06:53:18.154 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.86632.log: /var/log/ceph/cluster2-client.mirror.124772.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40847.log 2024-09-11T06:53:18.154 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.109880.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123576.log 2024-09-11T06:53:18.154 INFO:teuthology.orchestra.run.smithi112.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.124772.log.gz 2024-09-11T06:53:18.154 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.109880.log.gz 2024-09-11T06:53:18.154 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28254.log 2024-09-11T06:53:18.154 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.40847.log: /var/log/ceph/cluster2-client.mirror.123576.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46981.log 2024-09-11T06:53:18.154 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.admin.28254.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92970.log 2024-09-11T06:53:18.154 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.46981.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.29620.log 2024-09-11T06:53:18.154 INFO:teuthology.orchestra.run.smithi112.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.123576.log.gz 2024-09-11T06:53:18.154 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.92970.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37786.log 2024-09-11T06:53:18.154 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.29620.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32147.log 2024-09-11T06:53:18.155 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.37786.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28326.log 2024-09-11T06:53:18.155 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.32147.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79262.log 2024-09-11T06:53:18.155 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.admin.28326.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30876.log 2024-09-11T06:53:18.155 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.28326.log.gz 2024-09-11T06:53:18.158 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.79262.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92486.log 2024-09-11T06:53:18.159 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.30876.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104662.log 2024-09-11T06:53:18.159 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.92486.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56198.log 2024-09-11T06:53:18.160 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.104662.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.104662.log.gz 2024-09-11T06:53:18.160 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107514.log 2024-09-11T06:53:18.160 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.56198.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122912.log 2024-09-11T06:53:18.161 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65910.log 2024-09-11T06:53:18.162 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.107514.log: /var/log/ceph/cluster2-client.mirror.122912.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.122912.log.gz 2024-09-11T06:53:18.162 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117812.log 2024-09-11T06:53:18.164 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.65910.log: 92.0% -- replaced with /var/log/ceph/cluster1-client.mirror.107514.log.gz 2024-09-11T06:53:18.164 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85627.log 2024-09-11T06:53:18.164 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.117812.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.117812.log.gz 2024-09-11T06:53:18.165 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83023.log 2024-09-11T06:53:18.165 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71286.log 2024-09-11T06:53:18.166 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.85627.log: /var/log/ceph/cluster2-client.mirror.83023.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71751.log 2024-09-11T06:53:18.166 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36162.log 2024-09-11T06:53:18.166 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.71286.log: /var/log/ceph/cluster2-client.mirror.71751.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37096.log 2024-09-11T06:53:18.167 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.36162.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82177.log 2024-09-11T06:53:18.167 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102245.log 2024-09-11T06:53:18.168 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.37096.log: /var/log/ceph/cluster2-client.mirror.82177.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73247.log 2024-09-11T06:53:18.168 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.102245.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.102245.log.gz 2024-09-11T06:53:18.169 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55694.log 2024-09-11T06:53:18.169 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.73247.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59093.log 2024-09-11T06:53:18.170 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87911.log 2024-09-11T06:53:18.170 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.55694.log: /var/log/ceph/cluster2-client.mirror.59093.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73561.log 2024-09-11T06:53:18.171 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.87911.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69081.log 2024-09-11T06:53:18.171 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126064.log 2024-09-11T06:53:18.171 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.73561.log: /var/log/ceph/cluster2-client.mirror.69081.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70015.log 2024-09-11T06:53:18.172 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.126064.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55357.log 2024-09-11T06:53:18.173 INFO:teuthology.orchestra.run.smithi112.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.126064.log.gz 2024-09-11T06:53:18.173 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.70015.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57973.log 2024-09-11T06:53:18.173 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.55357.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60364.log 2024-09-11T06:53:18.174 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.57973.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.90423.log 2024-09-11T06:53:18.174 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.60364.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52864.log 2024-09-11T06:53:18.175 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50062.log 2024-09-11T06:53:18.175 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.90423.log: /var/log/ceph/cluster2-client.mirror.52864.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107469.log 2024-09-11T06:53:18.176 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.50062.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125569.log 2024-09-11T06:53:18.178 INFO:teuthology.orchestra.run.smithi112.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.33174.log.gz 2024-09-11T06:53:18.178 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45247.log.gz 2024-09-11T06:53:18.179 INFO:teuthology.orchestra.run.smithi112.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.45398.log.gz 2024-09-11T06:53:18.180 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.107469.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.107469.log.gz 2024-09-11T06:53:18.180 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40696.log 2024-09-11T06:53:18.181 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.125569.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.125569.log.gz 2024-09-11T06:53:18.181 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120700.log 2024-09-11T06:53:18.182 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.40696.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83581.log 2024-09-11T06:53:18.183 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.120700.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.120700.log.gz 2024-09-11T06:53:18.184 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34329.log 2024-09-11T06:53:18.185 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.83581.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83453.log 2024-09-11T06:53:18.186 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.34329.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122293.log 2024-09-11T06:53:18.187 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.83453.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62515.log 2024-09-11T06:53:18.188 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.122293.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.122293.log.gz 2024-09-11T06:53:18.189 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67864.log 2024-09-11T06:53:18.189 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.62515.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54667.log 2024-09-11T06:53:18.191 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.67864.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.38255.log.gz 2024-09-11T06:53:18.192 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114776.log 2024-09-11T06:53:18.192 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.54667.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123263.log 2024-09-11T06:53:18.200 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.114776.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.39297.log.gz 2024-09-11T06:53:18.201 INFO:teuthology.orchestra.run.smithi112.stderr: 96.9% -- replaced with /var/log/ceph/cluster2-client.mirror.114776.log.gz 2024-09-11T06:53:18.202 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.46981.log.gz 2024-09-11T06:53:18.202 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.47039.log.gz 2024-09-11T06:53:18.202 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67922.log 2024-09-11T06:53:18.202 INFO:teuthology.orchestra.run.smithi112.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.58872.log.gz 2024-09-11T06:53:18.203 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.123263.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.30876.log.gz 2024-09-11T06:53:18.203 INFO:teuthology.orchestra.run.smithi112.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.31155.log.gz 2024-09-11T06:53:18.203 INFO:teuthology.orchestra.run.smithi112.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.32426.log.gz 2024-09-11T06:53:18.204 INFO:teuthology.orchestra.run.smithi112.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.40696.log.gz 2024-09-11T06:53:18.204 INFO:teuthology.orchestra.run.smithi112.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.41444.log.gz 2024-09-11T06:53:18.204 INFO:teuthology.orchestra.run.smithi112.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.73809.log.gz 2024-09-11T06:53:18.204 INFO:teuthology.orchestra.run.smithi112.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.123263.log.gz 2024-09-11T06:53:18.204 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.74123.log.gz 2024-09-11T06:53:18.204 INFO:teuthology.orchestra.run.smithi112.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.34329.log.gz 2024-09-11T06:53:18.205 INFO:teuthology.orchestra.run.smithi112.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.34891.log.gz 87.8% 2024-09-11T06:53:18.205 INFO:teuthology.orchestra.run.smithi112.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.35604.log.gz 2024-09-11T06:53:18.205 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49469.log.gz 2024-09-11T06:53:18.205 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115162.log 2024-09-11T06:53:18.207 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.67922.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.50961.log.gz 2024-09-11T06:53:18.208 INFO:teuthology.orchestra.run.smithi112.stderr: 96.7% -- replaced with /var/log/ceph/cluster2-client.mirror.85490.log.gz 2024-09-11T06:53:18.208 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.49597.log.gz 2024-09-11T06:53:18.208 INFO:teuthology.orchestra.run.smithi112.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.57283.log.gz 2024-09-11T06:53:18.208 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.57318.log.gz 2024-09-11T06:53:18.208 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.48008.log.gz 2024-09-11T06:53:18.209 INFO:teuthology.orchestra.run.smithi112.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.43007.log.gz 2024-09-11T06:53:18.209 INFO:teuthology.orchestra.run.smithi112.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.83581.log.gz 2024-09-11T06:53:18.210 INFO:teuthology.orchestra.run.smithi112.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.67771.log.gz 2024-09-11T06:53:18.210 INFO:teuthology.orchestra.run.smithi112.stderr: 88.7% 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.78944.log.gz 2024-09-11T06:53:18.210 INFO:teuthology.orchestra.run.smithi112.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.70015.log.gz 2024-09-11T06:53:18.210 INFO:teuthology.orchestra.run.smithi112.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.53705.log.gz 2024-09-11T06:53:18.210 INFO:teuthology.orchestra.run.smithi112.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.67922.log.gz 2024-09-11T06:53:18.211 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.60306.log.gz 2024-09-11T06:53:18.211 INFO:teuthology.orchestra.run.smithi112.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.60364.log.gz 2024-09-11T06:53:18.211 INFO:teuthology.orchestra.run.smithi112.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.64042.log.gz 2024-09-11T06:53:18.211 INFO:teuthology.orchestra.run.smithi112.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.66437.log.gz 2024-09-11T06:53:18.211 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.64848.log.gz 2024-09-11T06:53:18.212 INFO:teuthology.orchestra.run.smithi112.stderr: 84.7% -- replaced with /var/log/ceph/cluster2-client.mirror.88260.log.gz 2024-09-11T06:53:18.212 INFO:teuthology.orchestra.run.smithi112.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.74836.log.gz 2024-09-11T06:53:18.212 INFO:teuthology.orchestra.run.smithi112.stderr: 92.9% -- replaced with /var/log/ceph/cluster2-client.mirror.85801.log.gz 2024-09-11T06:53:18.213 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125456.log 2024-09-11T06:53:18.214 INFO:teuthology.orchestra.run.smithi112.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.74964.log.gz 2024-09-11T06:53:18.214 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.75022.log.gz 2024-09-11T06:53:18.214 INFO:teuthology.orchestra.run.smithi112.stderr: 2.3% -- replaced with /var/log/ceph/cluster2-client.mirror.29569.log.gz 2024-09-11T06:53:18.214 INFO:teuthology.orchestra.run.smithi112.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.41816.log.gz 2024-09-11T06:53:18.214 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.76328.log.gz 2024-09-11T06:53:18.215 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.115162.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.42631.log.gz 2024-09-11T06:53:18.215 INFO:teuthology.orchestra.run.smithi112.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.81197.log.gz 2024-09-11T06:53:18.215 INFO:teuthology.orchestra.run.smithi112.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.84015.log.gz 2024-09-11T06:53:18.215 INFO:teuthology.orchestra.run.smithi112.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.36724.log.gz 2024-09-11T06:53:18.215 INFO:teuthology.orchestra.run.smithi112.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.37096.log.gz 2024-09-11T06:53:18.219 INFO:teuthology.orchestra.run.smithi112.stderr: 96.1% -- replaced with /var/log/ceph/cluster2-client.mirror.115162.log.gz 2024-09-11T06:53:18.219 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47167.log 2024-09-11T06:53:18.220 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.125456.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.125456.log.gz 2024-09-11T06:53:18.220 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38971.log 2024-09-11T06:53:18.221 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.47167.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57690.log 2024-09-11T06:53:18.222 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.38971.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.29381.log 2024-09-11T06:53:18.223 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.57690.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.91538.log 2024-09-11T06:53:18.223 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.29381.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58531.log 2024-09-11T06:53:18.224 INFO:teuthology.orchestra.run.smithi112.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.85340.log.gz 2024-09-11T06:53:18.224 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.91538.log: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.91782.log.gz 2024-09-11T06:53:18.225 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53333.log 2024-09-11T06:53:18.225 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.58531.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.29347.log 2024-09-11T06:53:18.226 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.53333.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113612.log 2024-09-11T06:53:18.227 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.29347.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104130.log 2024-09-11T06:53:18.229 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.113612.log: 91.7% -- replaced with /var/log/ceph/cluster1-client.mirror.113612.log.gz 2024-09-11T06:53:18.230 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88408.log 2024-09-11T06:53:18.231 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.104130.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.104130.log.gz 2024-09-11T06:53:18.231 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84984.log 2024-09-11T06:53:18.233 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.88408.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67219.log 2024-09-11T06:53:18.234 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.84984.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97103.log 2024-09-11T06:53:18.235 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.67219.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86001.log 2024-09-11T06:53:18.259 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.97103.log: 92.9% -- replaced with /var/log/ceph/cluster2-client.mirror.86205.log.gz 2024-09-11T06:53:18.266 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117869.log 2024-09-11T06:53:18.282 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.86001.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57504.log 2024-09-11T06:53:18.283 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.117869.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.74367.log.gz 2024-09-11T06:53:18.284 INFO:teuthology.orchestra.run.smithi112.stderr: 91.9% -- replaced with /var/log/ceph/cluster2-client.mirror.117869.log.gz 2024-09-11T06:53:18.284 INFO:teuthology.orchestra.run.smithi112.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.80583.log.gz 2024-09-11T06:53:18.284 INFO:teuthology.orchestra.run.smithi112.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.49748.log.gz 2024-09-11T06:53:18.286 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.51244.log.gz 2024-09-11T06:53:18.286 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47543.log.gz 2024-09-11T06:53:18.286 INFO:teuthology.orchestra.run.smithi112.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.48814.log.gz 2024-09-11T06:53:18.287 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.78045.log.gz 2024-09-11T06:53:18.288 INFO:teuthology.orchestra.run.smithi112.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.128496.log.gz 2024-09-11T06:53:18.289 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123647.log 2024-09-11T06:53:18.290 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.57504.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59465.log 2024-09-11T06:53:18.291 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.123647.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.123647.log.gz 2024-09-11T06:53:18.292 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62829.log 2024-09-11T06:53:18.292 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.68891.log.gz 2024-09-11T06:53:18.293 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.59465.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121924.log 2024-09-11T06:53:18.293 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.62829.log: 88.8%gzip -- replaced with /var/log/ceph/cluster2-client.mirror.69174.log.gz 2024-09-11T06:53:18.293 INFO:teuthology.orchestra.run.smithi112.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.102377.log 2024-09-11T06:53:18.293 INFO:teuthology.orchestra.run.smithi112.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.69325.log.gz 2024-09-11T06:53:18.294 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.121924.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.121924.log.gz 2024-09-11T06:53:18.295 INFO:teuthology.orchestra.run.smithi112.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.87008.log.gz 2024-09-11T06:53:18.295 INFO:teuthology.orchestra.run.smithi112.stderr: 84.6% -- replaced with /var/log/ceph/cluster2-client.mirror.88408.log.gz 2024-09-11T06:53:18.295 INFO:teuthology.orchestra.run.smithi112.stderr: 63.9% -- replaced with /var/log/ceph/cluster1-client.mirror.29381.log.gz 2024-09-11T06:53:18.295 INFO:teuthology.orchestra.run.smithi112.stderr: 93.1% -- replaced with /var/log/ceph/cluster2-client.mirror.80035.log.gz 2024-09-11T06:53:18.295 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.42724.log.gz 2024-09-11T06:53:18.296 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73468.log 2024-09-11T06:53:18.297 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.102377.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.102377.log.gz 2024-09-11T06:53:18.297 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.29734.log 2024-09-11T06:53:18.298 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.73468.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112144.log 2024-09-11T06:53:18.298 INFO:teuthology.orchestra.run.smithi112.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.78758.log.gz 2024-09-11T06:53:18.299 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.1.29734.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53112.log 2024-09-11T06:53:18.301 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.112144.log: 91.8% -- replaced with /var/log/ceph/cluster1-client.mirror.112144.log.gz 2024-09-11T06:53:18.303 INFO:teuthology.orchestra.run.smithi112.stderr: 79.7% -- replaced with /var/log/ceph/cluster1-client.admin.29296.log.gz 2024-09-11T06:53:18.304 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99036.log 2024-09-11T06:53:18.304 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.53112.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36910.log 2024-09-11T06:53:18.310 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.99036.log: 95.2% -- replaced with /var/log/ceph/cluster2-client.mirror.92486.log.gz 2024-09-11T06:53:18.310 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47915.log 2024-09-11T06:53:18.311 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.36910.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64418.log 2024-09-11T06:53:18.312 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.47915.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.29535.log 2024-09-11T06:53:18.313 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.64418.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126334.log 2024-09-11T06:53:18.314 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.admin.29535.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63166.log 2024-09-11T06:53:18.314 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.126334.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.126334.log.gz 2024-09-11T06:53:18.315 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46760.log 2024-09-11T06:53:18.315 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.63166.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43344.log 2024-09-11T06:53:18.318 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.46760.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128367.log 2024-09-11T06:53:18.318 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.43344.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80147.log 2024-09-11T06:53:18.320 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.128367.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.128367.log.gz 2024-09-11T06:53:18.320 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54760.log 2024-09-11T06:53:18.321 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.80147.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30748.log 2024-09-11T06:53:18.322 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.54760.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75805.log 2024-09-11T06:53:18.322 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.30748.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110392.log 2024-09-11T06:53:18.323 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.75805.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45526.log 2024-09-11T06:53:18.325 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.110392.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.110392.log.gz 2024-09-11T06:53:18.325 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126356.log 2024-09-11T06:53:18.326 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.45526.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52585.log 2024-09-11T06:53:18.327 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.126356.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.126356.log.gz 2024-09-11T06:53:18.327 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117426.log 2024-09-11T06:53:18.328 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.52585.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37600.log 2024-09-11T06:53:18.329 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.117426.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.117426.log.gz 2024-09-11T06:53:18.329 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74929.log 2024-09-11T06:53:18.334 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.37600.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130356.log 2024-09-11T06:53:18.335 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.74929.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69980.log 2024-09-11T06:53:18.336 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.130356.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.130356.log.gz 2024-09-11T06:53:18.336 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109915.log 2024-09-11T06:53:18.337 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.69980.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114863.log 2024-09-11T06:53:18.341 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.109915.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.109915.log.gz 2024-09-11T06:53:18.341 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121966.log 2024-09-11T06:53:18.341 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.114863.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.114863.log.gz 2024-09-11T06:53:18.341 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85464.log 2024-09-11T06:53:18.342 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.121966.log: 93.0% -- replaced with /var/log/ceph/cluster2-client.mirror.85844.log.gz 2024-09-11T06:53:18.342 INFO:teuthology.orchestra.run.smithi112.stderr: 90.8% -- replaced with /var/log/ceph/cluster2-client.mirror.121966.log.gz 2024-09-11T06:53:18.342 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78665.log 2024-09-11T06:53:18.342 INFO:teuthology.orchestra.run.smithi112.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.62515.log.gz 2024-09-11T06:53:18.343 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.62550.log.gz 2024-09-11T06:53:18.343 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.85464.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.33360.log.gz 2024-09-11T06:53:18.344 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42879.log 2024-09-11T06:53:18.344 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.78665.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.44875.log.gz 2024-09-11T06:53:18.344 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42124.log 2024-09-11T06:53:18.344 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.44747.log.gz 2024-09-11T06:53:18.344 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.42879.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32802.log 2024-09-11T06:53:18.345 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.42124.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.39448.log.gz 2024-09-11T06:53:18.346 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103752.log 2024-09-11T06:53:18.347 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.32802.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.39634.log.gz 2024-09-11T06:53:18.348 INFO:teuthology.orchestra.run.smithi112.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.39855.log.gz 2024-09-11T06:53:18.348 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.43065.log.gz 2024-09-11T06:53:18.348 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.43100.log.gz 2024-09-11T06:53:18.348 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.56442.log.gz 2024-09-11T06:53:18.348 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.46760.log.gz 2024-09-11T06:53:18.348 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.57039.log.gz 2024-09-11T06:53:18.349 INFO:teuthology.orchestra.run.smithi112.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.31527.log.gz 2024-09-11T06:53:18.349 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.31868.log.gz 2024-09-11T06:53:18.350 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28590.log 2024-09-11T06:53:18.351 INFO:teuthology.orchestra.run.smithi112.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.41502.log.gz 2024-09-11T06:53:18.351 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.103752.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.103752.log.gz 2024-09-11T06:53:18.352 INFO:teuthology.orchestra.run.smithi112.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.41595.log.gz 2024-09-11T06:53:18.352 INFO:teuthology.orchestra.run.smithi112.stderr: 92.8% -- replaced with /var/log/ceph/cluster1-client.mirror.53879.log.gz 2024-09-11T06:53:18.352 INFO:teuthology.orchestra.run.smithi112.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.34798.log.gz 2024-09-11T06:53:18.354 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.admin.28590.log: 0.0% 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.67585.log.gz 2024-09-11T06:53:18.354 INFO:teuthology.orchestra.run.smithi112.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.28590.log.gz 2024-09-11T06:53:18.355 INFO:teuthology.orchestra.run.smithi112.stderr: 97.1% -- replaced with /var/log/ceph/cluster2-client.mirror.80124.log.gz 2024-09-11T06:53:18.355 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84422.log 2024-09-11T06:53:18.355 INFO:teuthology.orchestra.run.smithi112.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.80397.log.gz 2024-09-11T06:53:18.356 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51616.log.gz 2024-09-11T06:53:18.356 INFO:teuthology.orchestra.run.smithi112.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.37600.log.gz 2024-09-11T06:53:18.356 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.50345.log.gz 2024-09-11T06:53:18.356 INFO:teuthology.orchestra.run.smithi112.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.57504.log.gz 2024-09-11T06:53:18.357 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.43623.log.gz 2024-09-11T06:53:18.357 INFO:teuthology.orchestra.run.smithi112.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.76518.log.gz 2024-09-11T06:53:18.358 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67264.log 2024-09-11T06:53:18.358 INFO:teuthology.orchestra.run.smithi112.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.78665.log.gz 2024-09-11T06:53:18.358 INFO:teuthology.orchestra.run.smithi112.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.53015.log.gz 2024-09-11T06:53:18.359 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.84422.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108587.log 2024-09-11T06:53:18.359 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.62794.log.gz 2024-09-11T06:53:18.359 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.67264.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84949.log 2024-09-11T06:53:18.361 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.108587.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.108587.log.gz 2024-09-11T06:53:18.361 INFO:teuthology.orchestra.run.smithi112.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.62829.log.gz 2024-09-11T06:53:18.362 INFO:teuthology.orchestra.run.smithi112.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48194.log.gz 2024-09-11T06:53:18.362 INFO:teuthology.orchestra.run.smithi112.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.64418.log.gz 2024-09-11T06:53:18.363 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.64697.log.gz 2024-09-11T06:53:18.364 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-osd.0.log 2024-09-11T06:53:18.364 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.84949.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128470.log 2024-09-11T06:53:18.365 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.59465.log.gz 2024-09-11T06:53:18.366 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.68856.log.gz 2024-09-11T06:53:18.366 INFO:teuthology.orchestra.run.smithi112.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.72220.log.gz 2024-09-11T06:53:18.367 INFO:teuthology.orchestra.run.smithi112.stderr: 19.0% -- replaced with /var/log/ceph/cluster1-client.mirror.1.29734.log.gz 2024-09-11T06:53:18.367 INFO:teuthology.orchestra.run.smithi112.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.85569.log.gz 2024-09-11T06:53:18.368 INFO:teuthology.orchestra.run.smithi112.stderr: 88.2% -- replaced with /var/log/ceph/cluster1-client.admin.39161.log.gz 2024-09-11T06:53:18.368 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-osd.0.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39355.log 2024-09-11T06:53:18.369 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.128470.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.128470.log.gz 2024-09-11T06:53:18.370 INFO:teuthology.orchestra.run.smithi112.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.41909.log.gz 2024-09-11T06:53:18.370 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66809.log 2024-09-11T06:53:18.371 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.39355.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.42879.log.gz 2024-09-11T06:53:18.371 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127391.log 2024-09-11T06:53:18.372 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.66809.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.73468.log.gz 2024-09-11T06:53:18.372 INFO:teuthology.orchestra.run.smithi112.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.73526.log.gz 2024-09-11T06:53:18.372 INFO:teuthology.orchestra.run.smithi112.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.81604.log.gz 2024-09-11T06:53:18.373 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109561.log 2024-09-11T06:53:18.373 INFO:teuthology.orchestra.run.smithi112.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.84422.log.gz 2024-09-11T06:53:18.373 INFO:teuthology.orchestra.run.smithi112.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.84949.log.gz 2024-09-11T06:53:18.374 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.127391.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.127391.log.gz 2024-09-11T06:53:18.374 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78886.log 2024-09-11T06:53:18.374 INFO:teuthology.orchestra.run.smithi112.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.86987.log.gz 2024-09-11T06:53:18.375 INFO:teuthology.orchestra.run.smithi112.stderr: 84.6% -- replaced with /var/log/ceph/cluster2-client.mirror.87998.log.gz 2024-09-11T06:53:18.376 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.109561.log: 88.0% 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.36538.log.gz 2024-09-11T06:53:18.376 INFO:teuthology.orchestra.run.smithi112.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.109561.log.gz 2024-09-11T06:53:18.377 INFO:teuthology.orchestra.run.smithi112.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.36910.log.gz 2024-09-11T06:53:18.378 INFO:teuthology.orchestra.run.smithi112.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.80833.log.gz 2024-09-11T06:53:18.379 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53391.log 2024-09-11T06:53:18.379 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.78886.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35918.log 2024-09-11T06:53:18.380 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.53391.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.98167.log 2024-09-11T06:53:18.381 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.35918.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42782.log 2024-09-11T06:53:18.382 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.98167.log: 95.9% -- replaced with /var/log/ceph/cluster2-client.mirror.92970.log.gz 2024-09-11T06:53:18.382 INFO:teuthology.orchestra.run.smithi112.stderr: 92.8% -- replaced with /var/log/ceph/cluster1-client.mirror.67172.log.gz 2024-09-11T06:53:18.386 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82647.log 2024-09-11T06:53:18.387 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.42782.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124721.log 2024-09-11T06:53:18.387 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.82647.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34984.log 2024-09-11T06:53:18.390 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.124721.log: 90.8% -- replaced with /var/log/ceph/cluster2-client.mirror.124721.log.gz 2024-09-11T06:53:18.390 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78231.log 2024-09-11T06:53:18.391 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.34984.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83209.log 2024-09-11T06:53:18.391 INFO:teuthology.orchestra.run.smithi112.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.32802.log.gz 2024-09-11T06:53:18.393 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.78231.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37414.log 2024-09-11T06:53:18.393 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.83209.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39262.log 2024-09-11T06:53:18.394 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.37414.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60271.log 2024-09-11T06:53:18.395 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.39262.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.54473.log 2024-09-11T06:53:18.396 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.60271.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51988.log 2024-09-11T06:53:18.397 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.admin.54473.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.39355.log.gz 2024-09-11T06:53:18.399 INFO:teuthology.orchestra.run.smithi112.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.40231.log.gz 2024-09-11T06:53:18.399 INFO:teuthology.orchestra.run.smithi112.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.61054.log.gz 2024-09-11T06:53:18.399 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118576.log 2024-09-11T06:53:18.400 INFO:teuthology.orchestra.run.smithi112.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.61488.log.gz 2024-09-11T06:53:18.400 INFO:teuthology.orchestra.run.smithi112.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.30934.log.gz 2024-09-11T06:53:18.401 INFO:teuthology.orchestra.run.smithi112.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.41537.log.gz 2024-09-11T06:53:18.401 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.51988.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.51651.log.gz 2024-09-11T06:53:18.401 INFO:teuthology.orchestra.run.smithi112.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.34984.log.gz 2024-09-11T06:53:18.402 INFO:teuthology.orchestra.run.smithi112.stderr: 84.6% -- replaced with /var/log/ceph/cluster2-client.mirror.87610.log.gz 2024-09-11T06:53:18.402 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97929.log 2024-09-11T06:53:18.403 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.118576.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.118576.log.gz 2024-09-11T06:53:18.403 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49655.log 2024-09-11T06:53:18.406 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.97929.log: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.90423.log.gz 2024-09-11T06:53:18.407 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103366.log 2024-09-11T06:53:18.407 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.49655.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120884.log 2024-09-11T06:53:18.409 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.103366.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.103366.log.gz 2024-09-11T06:53:18.409 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.118222.log 2024-09-11T06:53:18.410 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.120884.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.76704.log.gz 2024-09-11T06:53:18.411 INFO:teuthology.orchestra.run.smithi112.stderr: 88.7% -- replaced with /var/log/ceph/cluster1-client.admin.46164.log.gz 2024-09-11T06:53:18.412 INFO:teuthology.orchestra.run.smithi112.stderr: 90.8% -- replaced with /var/log/ceph/cluster2-client.mirror.120884.log.gz 2024-09-11T06:53:18.412 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110350.log 2024-09-11T06:53:18.413 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.118222.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.118222.log.gz 2024-09-11T06:53:18.413 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125147.log 2024-09-11T06:53:18.416 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.110350.log: 96.7% -- replaced with /var/log/ceph/cluster2-client.mirror.42213.log.gz 2024-09-11T06:53:18.416 INFO:teuthology.orchestra.run.smithi112.stderr: 92.1% -- replaced with /var/log/ceph/cluster2-client.mirror.110350.log.gz 2024-09-11T06:53:18.416 INFO:teuthology.orchestra.run.smithi112.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.83546.log.gz 2024-09-11T06:53:18.416 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126176.log 2024-09-11T06:53:18.416 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.62643.log.gz 2024-09-11T06:53:18.417 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.125147.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.125147.log.gz 2024-09-11T06:53:18.417 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31810.log 2024-09-11T06:53:18.417 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.126176.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.126176.log.gz 2024-09-11T06:53:18.418 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31062.log 2024-09-11T06:53:18.418 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.53391.log.gz 2024-09-11T06:53:18.419 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.31810.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63073.log 2024-09-11T06:53:18.419 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.66100.log.gz 2024-09-11T06:53:18.420 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.31062.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124892.log 2024-09-11T06:53:18.420 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.66809.log.gz 2024-09-11T06:53:18.420 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.63073.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.72313.log.gz 2024-09-11T06:53:18.421 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.37286.log.gz 2024-09-11T06:53:18.421 INFO:teuthology.orchestra.run.smithi112.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.75336.log.gz 2024-09-11T06:53:18.421 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66751.log 2024-09-11T06:53:18.422 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.124892.log: 57.1% -- replaced with /var/log/ceph/cluster2-client.mirror.29620.log.gz 2024-09-11T06:53:18.422 INFO:teuthology.orchestra.run.smithi112.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.36011.log.gz 2024-09-11T06:53:18.422 INFO:teuthology.orchestra.run.smithi112.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.42782.log.gz 2024-09-11T06:53:18.423 INFO:teuthology.orchestra.run.smithi112.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.124892.log.gz 2024-09-11T06:53:18.423 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster1-client.mirror.80808.log.gz 2024-09-11T06:53:18.424 INFO:teuthology.orchestra.run.smithi112.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.82612.log.gz 2024-09-11T06:53:18.427 INFO:teuthology.orchestra.run.smithi112.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.36759.log.gz 2024-09-11T06:53:18.430 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105213.log 2024-09-11T06:53:18.431 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.66751.log: /var/log/ceph/cluster2-client.mirror.105213.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.105213.log.gz 2024-09-11T06:53:18.434 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111114.log 2024-09-11T06:53:18.434 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125076.log 2024-09-11T06:53:18.438 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.111114.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.56163.log.gz 2024-09-11T06:53:18.439 INFO:teuthology.orchestra.run.smithi112.stderr: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.130019.log.gz 2024-09-11T06:53:18.439 INFO:teuthology.orchestra.run.smithi112.stderr: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.111114.log.gz 2024-09-11T06:53:18.439 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122173.log 2024-09-11T06:53:18.441 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.125076.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70573.log 2024-09-11T06:53:18.441 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.122173.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.125076.log.gz 2024-09-11T06:53:18.441 INFO:teuthology.orchestra.run.smithi112.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.122173.log.gz 2024-09-11T06:53:18.441 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113990.log 2024-09-11T06:53:18.442 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.70573.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69639.log 2024-09-11T06:53:18.442 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.113990.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33302.log 2024-09-11T06:53:18.442 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.69639.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44933.log 2024-09-11T06:53:18.443 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.33302.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130189.log 2024-09-11T06:53:18.443 INFO:teuthology.orchestra.run.smithi112.stderr: 91.8% -- replaced with /var/log/ceph/cluster1-client.mirror.113990.log.gz 2024-09-11T06:53:18.443 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41874.log 2024-09-11T06:53:18.444 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.44933.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33736.log 2024-09-11T06:53:18.445 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.130189.log: /var/log/ceph/cluster2-client.mirror.41874.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87096.log 2024-09-11T06:53:18.445 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.130189.log.gz 2024-09-11T06:53:18.445 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.33736.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48849.log 2024-09-11T06:53:18.446 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64976.log 2024-09-11T06:53:18.446 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.87096.log: /var/log/ceph/cluster2-client.mirror.48849.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105325.log 2024-09-11T06:53:18.447 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87071.log 2024-09-11T06:53:18.447 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.64976.log: /var/log/ceph/cluster1-client.mirror.105325.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.103298.log 2024-09-11T06:53:18.448 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.87071.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110258.log 2024-09-11T06:53:18.449 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.103298.log: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.105325.log.gz 2024-09-11T06:53:18.449 INFO:teuthology.orchestra.run.smithi112.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.103298.log.gz 2024-09-11T06:53:18.449 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86067.log 2024-09-11T06:53:18.449 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.110258.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.110258.log.gz 2024-09-11T06:53:18.450 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34108.log 2024-09-11T06:53:18.450 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.86067.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124053.log 2024-09-11T06:53:18.451 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.46195.log 2024-09-11T06:53:18.451 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.34108.log: /var/log/ceph/cluster2-client.mirror.124053.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113114.log 2024-09-11T06:53:18.453 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.admin.46195.log: 90.8% -- replaced with /var/log/ceph/cluster2-client.mirror.124053.log.gz 2024-09-11T06:53:18.453 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78010.log 2024-09-11T06:53:18.453 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115627.log 2024-09-11T06:53:18.454 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.113114.log: 88.2%/var/log/ceph/cluster2-client.mirror.78010.log: -- replaced with /var/log/ceph/cluster2-client.mirror.113114.log.gz 2024-09-11T06:53:18.454 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85739.log 2024-09-11T06:53:18.455 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.115627.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64197.log 2024-09-11T06:53:18.455 INFO:teuthology.orchestra.run.smithi112.stderr: 88.2%/var/log/ceph/cluster2-client.mirror.85739.log: -- replaced with /var/log/ceph/cluster2-client.mirror.115627.log.gz 2024-09-11T06:53:18.456 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28741.log 2024-09-11T06:53:18.456 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37693.log 2024-09-11T06:53:18.456 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.64197.log: /var/log/ceph/cluster1-client.admin.28741.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77018.log 2024-09-11T06:53:18.456 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28741.log.gz 2024-09-11T06:53:18.457 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64790.log 2024-09-11T06:53:18.458 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.37693.log: /var/log/ceph/cluster2-client.mirror.77018.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107855.log 2024-09-11T06:53:18.458 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126041.log 2024-09-11T06:53:18.459 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.64790.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55636.log 2024-09-11T06:53:18.459 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.107855.log: /var/log/ceph/cluster1-client.mirror.126041.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.126041.log.gz 2024-09-11T06:53:18.459 INFO:teuthology.orchestra.run.smithi112.stderr: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.107855.log.gz 2024-09-11T06:53:18.460 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64476.log 2024-09-11T06:53:18.460 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.55636.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35732.log 2024-09-11T06:53:18.460 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59523.log 2024-09-11T06:53:18.461 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.64476.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120359.log 2024-09-11T06:53:18.461 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.35732.log: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.91538.log.gz 2024-09-11T06:53:18.461 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67829.log 2024-09-11T06:53:18.462 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.59523.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38534.log 2024-09-11T06:53:18.462 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.120359.log: /var/log/ceph/cluster2-client.mirror.67829.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51895.log 2024-09-11T06:53:18.463 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32651.log 2024-09-11T06:53:18.463 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.38534.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.90945.log 2024-09-11T06:53:18.463 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.51895.log: /var/log/ceph/cluster2-client.mirror.32651.log: 91.9% -- replaced with /var/log/ceph/cluster1-client.mirror.120359.log.gz 2024-09-11T06:53:18.463 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115273.log 2024-09-11T06:53:18.464 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.90945.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37565.log 2024-09-11T06:53:18.464 INFO:teuthology.orchestra.run.smithi112.stderr: 84.6% -- replaced with /var/log/ceph/cluster2-client.mirror.90945.log.gz 2024-09-11T06:53:18.465 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.115273.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59279.log 2024-09-11T06:53:18.465 INFO:teuthology.orchestra.run.smithi112.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.115273.log.gz 2024-09-11T06:53:18.465 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.37565.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74743.log 2024-09-11T06:53:18.465 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.59279.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114477.log 2024-09-11T06:53:18.466 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.74743.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71976.log 2024-09-11T06:53:18.466 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.114477.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64232.log 2024-09-11T06:53:18.467 INFO:teuthology.orchestra.run.smithi112.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.114477.log.gz 2024-09-11T06:53:18.467 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.71976.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80695.log 2024-09-11T06:53:18.467 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.64232.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30297.log 2024-09-11T06:53:18.467 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.80695.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126401.log 2024-09-11T06:53:18.468 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.30297.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115970.log 2024-09-11T06:53:18.468 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.126401.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.126401.log.gz 2024-09-11T06:53:18.469 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.83302.log 2024-09-11T06:53:18.469 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.115970.log: gzip 95.7% -5 --verbose -- replaced with /var/log/ceph/cluster2-client.mirror.89433.log.gz -- 2024-09-11T06:53:18.469 INFO:teuthology.orchestra.run.smithi112.stderr: /var/log/ceph/cluster2-client.mirror.91759.log 2024-09-11T06:53:18.470 INFO:teuthology.orchestra.run.smithi112.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.115970.log.gz 2024-09-11T06:53:18.470 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.83302.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50775.log 2024-09-11T06:53:18.471 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.91759.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.79599.log 2024-09-11T06:53:18.471 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.50775.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68333.log 2024-09-11T06:53:18.471 INFO:teuthology.orchestra.run.smithi112.stderr: 84.5% -- replaced with /var/log/ceph/cluster2-client.mirror.91759.log.gz 2024-09-11T06:53:18.471 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.79599.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121320.log 2024-09-11T06:53:18.471 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.68333.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64135.log 2024-09-11T06:53:18.472 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.29720.log 2024-09-11T06:53:18.472 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.121320.log: /var/log/ceph/cluster2-client.mirror.64135.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61147.log 2024-09-11T06:53:18.472 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.0.29720.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58252.log 2024-09-11T06:53:18.472 INFO:teuthology.orchestra.run.smithi112.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.121320.log.gz 2024-09-11T06:53:18.472 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.61147.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39130.log 2024-09-11T06:53:18.473 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38813.log 2024-09-11T06:53:18.473 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.admin.39130.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114099.log 2024-09-11T06:53:18.474 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.38813.log: 93.1% -- replaced with /var/log/ceph/cluster2-client.mirror.86295.log.gz 2024-09-11T06:53:18.474 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40661.log 2024-09-11T06:53:18.474 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.58252.log: /var/log/ceph/cluster2-client.mirror.114099.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70480.log 2024-09-11T06:53:18.474 INFO:teuthology.orchestra.run.smithi112.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.114099.log.gz 2024-09-11T06:53:18.475 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.40661.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84480.log 2024-09-11T06:53:18.475 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.70480.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56721.log 2024-09-11T06:53:18.476 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.84480.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82426.log 2024-09-11T06:53:18.476 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.56721.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82705.log 2024-09-11T06:53:18.476 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.82426.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125238.log 2024-09-11T06:53:18.477 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.82705.log: gzip -5 --verbose -- /var/log/ceph/cluster2-osd.2.log 2024-09-11T06:53:18.477 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.125238.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109208.log 2024-09-11T06:53:18.477 INFO:teuthology.orchestra.run.smithi112.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.32744.log.gz 2024-09-11T06:53:18.478 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-osd.2.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.33081.log.gz 2024-09-11T06:53:18.478 INFO:teuthology.orchestra.run.smithi112.stderr: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.125238.log.gz 2024-09-11T06:53:18.479 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51054.log 2024-09-11T06:53:18.479 INFO:teuthology.orchestra.run.smithi112.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.33643.log.gz 2024-09-11T06:53:18.479 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.109208.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65352.log 2024-09-11T06:53:18.480 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.51054.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103032.log 2024-09-11T06:53:18.480 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.65352.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124750.log 2024-09-11T06:53:18.480 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.38534.log.gz 2024-09-11T06:53:18.481 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.103032.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.39262.log.gz 2024-09-11T06:53:18.481 INFO:teuthology.orchestra.run.smithi112.stderr: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.109208.log.gz 2024-09-11T06:53:18.482 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50310.log 2024-09-11T06:53:18.482 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.124750.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117911.log 2024-09-11T06:53:18.483 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.50310.log: 92.0% -- replaced with /var/log/ceph/cluster1-client.mirror.103032.log.gz 2024-09-11T06:53:18.483 INFO:teuthology.orchestra.run.smithi112.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.124750.log.gz 2024-09-11T06:53:18.483 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130948.log 2024-09-11T06:53:18.483 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.39483.log.gz 2024-09-11T06:53:18.483 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.117911.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31434.log 2024-09-11T06:53:18.483 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.130948.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.130948.log.gz 2024-09-11T06:53:18.484 INFO:teuthology.orchestra.run.smithi112.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.31062.log.gz 2024-09-11T06:53:18.485 INFO:teuthology.orchestra.run.smithi112.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.31810.log.gz 2024-09-11T06:53:18.485 INFO:teuthology.orchestra.run.smithi112.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.32519.log.gz 2024-09-11T06:53:18.486 INFO:teuthology.orchestra.run.smithi112.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.40661.log.gz 2024-09-11T06:53:18.486 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37224.log 2024-09-11T06:53:18.486 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.31434.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65445.log 2024-09-11T06:53:18.487 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.37224.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76084.log 2024-09-11T06:53:18.487 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.65445.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.74309.log.gz 2024-09-11T06:53:18.488 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59841.log 2024-09-11T06:53:18.488 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.76084.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126019.log 2024-09-11T06:53:18.489 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.59841.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.34519.log.gz 2024-09-11T06:53:18.490 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49655.log.gz 2024-09-11T06:53:18.490 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.126019.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68988.log 2024-09-11T06:53:18.490 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.50310.log.gz 2024-09-11T06:53:18.490 INFO:teuthology.orchestra.run.smithi112.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.126019.log.gz 2024-09-11T06:53:18.490 INFO:teuthology.orchestra.run.smithi112.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.51054.log.gz 2024-09-11T06:53:18.490 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78138.log 2024-09-11T06:53:18.490 INFO:teuthology.orchestra.run.smithi112.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.65410.log.gz 2024-09-11T06:53:18.490 INFO:teuthology.orchestra.run.smithi112.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.65445.log.gz 2024-09-11T06:53:18.491 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.68988.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.62608.log.gz 2024-09-11T06:53:18.491 INFO:teuthology.orchestra.run.smithi112.stderr: 96.9% -- replaced with /var/log/ceph/cluster2-client.mirror.117911.log.gz 2024-09-11T06:53:18.492 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28357.log 2024-09-11T06:53:18.492 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.78138.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.70352.log.gz 2024-09-11T06:53:18.492 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57783.log 2024-09-11T06:53:18.493 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.admin.28357.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108644.log 2024-09-11T06:53:18.493 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.28357.log.gz 2024-09-11T06:53:18.494 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.57783.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.62701.log.gz 2024-09-11T06:53:18.494 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.64790.log.gz 2024-09-11T06:53:18.494 INFO:teuthology.orchestra.run.smithi112.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.70573.log.gz 2024-09-11T06:53:18.495 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.65224.log.gz 2024-09-11T06:53:18.495 INFO:teuthology.orchestra.run.smithi112.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.59244.log.gz 2024-09-11T06:53:18.495 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.59841.log.gz 2024-09-11T06:53:18.495 INFO:teuthology.orchestra.run.smithi112.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.69639.log.gz 2024-09-11T06:53:18.495 INFO:teuthology.orchestra.run.smithi112.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.76084.log.gz 2024-09-11T06:53:18.495 INFO:teuthology.orchestra.run.smithi112.stderr: 64.5% -- replaced with /var/log/ceph/cluster2-client.mirror.29603.log.gz 2024-09-11T06:53:18.495 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster1-client.admin.39130.log.gz 2024-09-11T06:53:18.496 INFO:teuthology.orchestra.run.smithi112.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.33736.log.gz 2024-09-11T06:53:18.496 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130779.log 2024-09-11T06:53:18.497 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.108644.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.73247.log.gz 2024-09-11T06:53:18.497 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33550.log 2024-09-11T06:53:18.497 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.130779.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.130779.log.gz 2024-09-11T06:53:18.498 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.78514.log.gz 2024-09-11T06:53:18.498 INFO:teuthology.orchestra.run.smithi112.stderr: 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.108644.log.gz 2024-09-11T06:53:18.498 INFO:teuthology.orchestra.run.smithi112.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.82705.log.gz 2024-09-11T06:53:18.498 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43286.log 2024-09-11T06:53:18.500 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.33550.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127895.log 2024-09-11T06:53:18.500 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.43286.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119483.log 2024-09-11T06:53:18.500 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.127895.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61802.log 2024-09-11T06:53:18.500 INFO:teuthology.orchestra.run.smithi112.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.37038.log.gz 2024-09-11T06:53:18.500 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.119483.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.127895.log.gz 2024-09-11T06:53:18.500 INFO:teuthology.orchestra.run.smithi112.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.119483.log.gz 2024-09-11T06:53:18.502 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.29695.log 2024-09-11T06:53:18.502 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.61802.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40041.log 2024-09-11T06:53:18.502 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.admin.29695.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125479.log 2024-09-11T06:53:18.502 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.29695.log.gz 2024-09-11T06:53:18.502 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.40041.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72910.log 2024-09-11T06:53:18.502 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.125479.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128622.log 2024-09-11T06:53:18.502 INFO:teuthology.orchestra.run.smithi112.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.125479.log.gz 2024-09-11T06:53:18.502 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.72910.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.131235.log 2024-09-11T06:53:18.502 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster1-client.mirror.128622.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33829.log 2024-09-11T06:53:18.503 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.admin.131235.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.128622.log.gz 2024-09-11T06:53:18.503 INFO:teuthology.orchestra.run.smithi112.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.131235.log.gz 2024-09-11T06:53:18.506 INFO:teuthology.orchestra.run.smithi112.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74216.log 2024-09-11T06:53:18.518 INFO:teuthology.orchestra.run.smithi112.stderr:/var/log/ceph/cluster2-client.mirror.33829.log: /var/log/ceph/cluster2-client.mirror.74216.log: 92.9% -- replaced with /var/log/ceph/cluster2-client.mirror.86024.log.gz 2024-09-11T06:53:18.523 INFO:teuthology.orchestra.run.smithi112.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.72720.log.gz 2024-09-11T06:53:18.532 INFO:teuthology.orchestra.run.smithi112.stderr: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.97103.log.gz 2024-09-11T06:53:18.534 INFO:teuthology.orchestra.run.smithi112.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.43755.log.gz 2024-09-11T06:53:18.535 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.52399.log.gz 2024-09-11T06:53:18.537 INFO:teuthology.orchestra.run.smithi112.stderr: 95.4% -- replaced with /var/log/ceph/cluster2-client.mirror.97929.log.gz 2024-09-11T06:53:18.545 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.33550.log.gz 2024-09-11T06:53:18.546 INFO:teuthology.orchestra.run.smithi112.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.34143.log.gz 2024-09-11T06:53:18.547 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.46574.log.gz 2024-09-11T06:53:18.548 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.46795.log.gz 2024-09-11T06:53:18.549 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.58531.log.gz 2024-09-11T06:53:18.549 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.58589.log.gz 2024-09-11T06:53:18.550 INFO:teuthology.orchestra.run.smithi112.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.31027.log.gz 2024-09-11T06:53:18.550 INFO:teuthology.orchestra.run.smithi112.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.31434.log.gz 2024-09-11T06:53:18.550 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.32147.log.gz 2024-09-11T06:53:18.550 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.40510.log.gz 2024-09-11T06:53:18.550 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.74216.log.gz 2024-09-11T06:53:18.550 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.50062.log.gz 2024-09-11T06:53:18.555 INFO:teuthology.orchestra.run.smithi112.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.57783.log.gz 2024-09-11T06:53:18.556 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.57880.log.gz 2024-09-11T06:53:18.556 INFO:teuthology.orchestra.run.smithi112.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.43286.log.gz 2024-09-11T06:53:18.557 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.77355.log.gz 2024-09-11T06:53:18.557 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.71286.log.gz 2024-09-11T06:53:18.557 INFO:teuthology.orchestra.run.smithi112.stderr: 88.7% -- replaced with /var/log/ceph/cluster1-client.admin.54017.log.gz 2024-09-11T06:53:18.558 INFO:teuthology.orchestra.run.smithi112.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.55357.log.gz 2024-09-11T06:53:18.558 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.61802.log.gz 2024-09-11T06:53:18.560 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.67864.log.gz 2024-09-11T06:53:18.561 INFO:teuthology.orchestra.run.smithi112.stderr: 70.2% -- replaced with /var/log/ceph/cluster2-client.admin.29535.log.gz 2024-09-11T06:53:18.562 INFO:teuthology.orchestra.run.smithi112.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.75805.log.gz 2024-09-11T06:53:18.562 INFO:teuthology.orchestra.run.smithi112.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.84984.log.gz 2024-09-11T06:53:18.563 INFO:teuthology.orchestra.run.smithi112.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.86920.log.gz 2024-09-11T06:53:18.563 INFO:teuthology.orchestra.run.smithi112.stderr: 93.3% -- replaced with /var/log/ceph/cluster2-client.mirror.80078.log.gz 2024-09-11T06:53:18.564 INFO:teuthology.orchestra.run.smithi112.stderr: 93.2% -- replaced with /var/log/ceph/cluster1-client.mirror.82197.log.gz 2024-09-11T06:53:18.564 INFO:teuthology.orchestra.run.smithi112.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.78138.log.gz 2024-09-11T06:53:18.589 INFO:teuthology.orchestra.run.smithi112.stderr: 95.1% -- replaced with /var/log/ceph/cluster2-client.mirror.98167.log.gz 2024-09-11T06:53:18.592 INFO:teuthology.orchestra.run.smithi112.stderr: 93.1% -- replaced with /var/log/ceph/cluster1-client.mirror.86589.log.gz 2024-09-11T06:53:18.598 INFO:teuthology.orchestra.run.smithi112.stderr: 95.5% -- replaced with /var/log/ceph/cluster2-client.mirror.99036.log.gz 2024-09-11T06:53:18.613 INFO:teuthology.orchestra.run.smithi112.stderr: 80.3% -- replaced with /var/log/ceph/cluster2-client.admin.28254.log.gz 2024-09-11T06:53:18.614 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.45491.log.gz 2024-09-11T06:53:18.618 INFO:teuthology.orchestra.run.smithi112.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.46326.log.gz 2024-09-11T06:53:18.629 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.67550.log.gz 2024-09-11T06:53:18.633 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.49504.log.gz 2024-09-11T06:53:18.634 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.54667.log.gz 2024-09-11T06:53:18.634 INFO:teuthology.orchestra.run.smithi112.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.57973.log.gz 2024-09-11T06:53:18.638 INFO:teuthology.orchestra.run.smithi112.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.83453.log.gz 2024-09-11T06:53:18.644 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.55694.log.gz 2024-09-11T06:53:18.644 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.52585.log.gz 2024-09-11T06:53:18.647 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.52864.log.gz 2024-09-11T06:53:18.648 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.53112.log.gz 2024-09-11T06:53:18.648 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.53333.log.gz 2024-09-11T06:53:18.648 INFO:teuthology.orchestra.run.smithi112.stderr: 2.3% -- replaced with /var/log/ceph/cluster1-client.mirror.29347.log.gz 2024-09-11T06:53:18.649 INFO:teuthology.orchestra.run.smithi112.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.82647.log.gz 2024-09-11T06:53:18.666 INFO:teuthology.orchestra.run.smithi112.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.83678.log.gz 2024-09-11T06:53:18.669 INFO:teuthology.orchestra.run.smithi112.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.38971.log.gz 2024-09-11T06:53:18.671 INFO:teuthology.orchestra.run.smithi112.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.41688.log.gz 2024-09-11T06:53:18.673 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47167.log.gz 2024-09-11T06:53:18.699 INFO:teuthology.orchestra.run.smithi112.stderr: 93.3% -- replaced with /var/log/ceph/cluster2-client.mirror.86655.log.gz 2024-09-11T06:53:18.722 INFO:teuthology.orchestra.run.smithi112.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.83023.log.gz 2024-09-11T06:53:18.727 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.61616.log.gz 2024-09-11T06:53:18.727 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.61674.log.gz 2024-09-11T06:53:18.729 INFO:teuthology.orchestra.run.smithi112.stderr: 93.3% -- replaced with /var/log/ceph/cluster2-client.mirror.86475.log.gz 2024-09-11T06:53:18.729 INFO:teuthology.orchestra.run.smithi112.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.37786.log.gz 2024-09-11T06:53:18.732 INFO:teuthology.orchestra.run.smithi112.stderr: 92.8% -- replaced with /var/log/ceph/cluster1-client.mirror.54260.log.gz 2024-09-11T06:53:18.732 INFO:teuthology.orchestra.run.smithi112.stderr: 92.9% -- replaced with /var/log/ceph/cluster1-client.mirror.85864.log.gz 2024-09-11T06:53:18.732 INFO:teuthology.orchestra.run.smithi112.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.51988.log.gz 2024-09-11T06:53:18.733 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.30748.log.gz 2024-09-11T06:53:18.733 INFO:teuthology.orchestra.run.smithi112.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.38499.log.gz 2024-09-11T06:53:18.734 INFO:teuthology.orchestra.run.smithi112.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.40847.log.gz 2024-09-11T06:53:18.735 INFO:teuthology.orchestra.run.smithi112.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.50624.log.gz 2024-09-11T06:53:18.736 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.74088.log.gz 2024-09-11T06:53:18.737 INFO:teuthology.orchestra.run.smithi112.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.54760.log.gz 2024-09-11T06:53:18.738 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.43193.log.gz 2024-09-11T06:53:18.739 INFO:teuthology.orchestra.run.smithi112.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.47915.log.gz 2024-09-11T06:53:18.739 INFO:teuthology.orchestra.run.smithi112.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.77824.log.gz 2024-09-11T06:53:18.740 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.78886.log.gz 2024-09-11T06:53:18.741 INFO:teuthology.orchestra.run.smithi112.stderr: 88.4% -- replaced with /var/log/ceph/cluster1-client.admin.54473.log.gz 2024-09-11T06:53:18.741 INFO:teuthology.orchestra.run.smithi112.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.64325.log.gz 2024-09-11T06:53:18.741 INFO:teuthology.orchestra.run.smithi112.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.69546.log.gz 2024-09-11T06:53:18.741 INFO:teuthology.orchestra.run.smithi112.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.85627.log.gz 2024-09-11T06:53:18.741 INFO:teuthology.orchestra.run.smithi112.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.41990.log.gz 2024-09-11T06:53:18.744 INFO:teuthology.orchestra.run.smithi112.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.87096.log.gz 2024-09-11T06:53:18.751 INFO:teuthology.orchestra.run.smithi112.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.82837.log.gz 2024-09-11T06:53:18.752 INFO:teuthology.orchestra.run.smithi112.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.84480.log.gz 2024-09-11T06:53:18.869 INFO:teuthology.orchestra.run.smithi112.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.86811.log.gz 2024-09-11T06:53:18.892 INFO:teuthology.orchestra.run.smithi112.stderr: 92.8% -- replaced with /var/log/ceph/cluster2-client.mirror.86115.log.gz 2024-09-11T06:53:18.893 INFO:teuthology.orchestra.run.smithi112.stderr: 97.0% -- replaced with /var/log/ceph/cluster2-client.mirror.67241.log.gz 2024-09-11T06:53:18.893 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.70852.log.gz 2024-09-11T06:53:18.907 INFO:teuthology.orchestra.run.smithi112.stderr: 92.8% -- replaced with /var/log/ceph/cluster1-client.mirror.85821.log.gz 2024-09-11T06:53:18.910 INFO:teuthology.orchestra.run.smithi112.stderr: 93.3% -- replaced with /var/log/ceph/cluster2-client.mirror.86432.log.gz 2024-09-11T06:53:18.921 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.39820.log.gz 2024-09-11T06:53:18.921 INFO:teuthology.orchestra.run.smithi112.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.44034.log.gz 2024-09-11T06:53:18.922 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.44592.log.gz 2024-09-11T06:53:18.922 INFO:teuthology.orchestra.run.smithi112.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46609.log.gz 2024-09-11T06:53:18.929 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.61112.log.gz 2024-09-11T06:53:18.930 INFO:teuthology.orchestra.run.smithi112.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.51558.log.gz 2024-09-11T06:53:18.930 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.51709.log.gz 2024-09-11T06:53:18.931 INFO:teuthology.orchestra.run.smithi112.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.37565.log.gz 2024-09-11T06:53:18.933 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50775.log.gz 2024-09-11T06:53:18.934 INFO:teuthology.orchestra.run.smithi112.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.83302.log.gz 2024-09-11T06:53:18.937 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.48907.log.gz 2024-09-11T06:53:18.937 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.48849.log.gz 2024-09-11T06:53:18.937 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.53577.log.gz 2024-09-11T06:53:18.938 INFO:teuthology.orchestra.run.smithi112.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.71751.log.gz 2024-09-11T06:53:18.943 INFO:teuthology.orchestra.run.smithi112.stderr: 93.0% -- replaced with /var/log/ceph/cluster1-client.mirror.80055.log.gz 2024-09-11T06:53:18.945 INFO:teuthology.orchestra.run.smithi112.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.80660.log.gz 2024-09-11T06:53:18.955 INFO:teuthology.orchestra.run.smithi112.stderr: 92.8% -- replaced with /var/log/ceph/cluster1-client.mirror.86001.log.gz 2024-09-11T06:53:18.961 INFO:teuthology.orchestra.run.smithi112.stderr: 93.2% -- replaced with /var/log/ceph/cluster1-client.mirror.86542.log.gz 2024-09-11T06:53:18.962 INFO:teuthology.orchestra.run.smithi112.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.36162.log.gz 2024-09-11T06:53:18.968 INFO:teuthology.orchestra.run.smithi112.stderr: 93.0% -- replaced with /var/log/ceph/cluster1-client.mirror.86315.log.gz 2024-09-11T06:53:18.976 INFO:teuthology.orchestra.run.smithi112.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.86723.log.gz 2024-09-11T06:53:18.978 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.63073.log.gz 2024-09-11T06:53:18.979 INFO:teuthology.orchestra.run.smithi112.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.37414.log.gz 2024-09-11T06:53:18.986 INFO:teuthology.orchestra.run.smithi112.stderr: 92.9% -- replaced with /var/log/ceph/cluster2-client.mirror.54240.log.gz 2024-09-11T06:53:18.989 INFO:teuthology.orchestra.run.smithi112.stderr: 92.9% -- replaced with /var/log/ceph/cluster2-client.mirror.67152.log.gz 2024-09-11T06:53:18.990 INFO:teuthology.orchestra.run.smithi112.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.83209.log.gz 2024-09-11T06:53:19.022 INFO:teuthology.orchestra.run.smithi112.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.44933.log.gz 2024-09-11T06:53:19.022 INFO:teuthology.orchestra.run.smithi112.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.45526.log.gz 2024-09-11T06:53:19.023 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45716.log.gz 2024-09-11T06:53:19.030 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.38813.log.gz 2024-09-11T06:53:19.037 INFO:teuthology.orchestra.run.smithi112.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.61147.log.gz 2024-09-11T06:53:19.038 INFO:teuthology.orchestra.run.smithi112.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.37693.log.gz 2024-09-11T06:53:19.039 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.78010.log.gz 2024-09-11T06:53:19.051 INFO:teuthology.orchestra.run.smithi112.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.79599.log.gz 2024-09-11T06:53:19.051 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.55880.log.gz 2024-09-11T06:53:19.054 INFO:teuthology.orchestra.run.smithi112.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.61860.log.gz 2024-09-11T06:53:19.054 INFO:teuthology.orchestra.run.smithi112.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.67829.log.gz 2024-09-11T06:53:19.055 INFO:teuthology.orchestra.run.smithi112.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.64232.log.gz 2024-09-11T06:53:19.055 INFO:teuthology.orchestra.run.smithi112.stderr: 88.7% 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.64135.log.gz -- replaced with /var/log/ceph/cluster2-client.mirror.64197.log.gz 2024-09-11T06:53:19.055 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-11T06:53:19.056 INFO:teuthology.orchestra.run.smithi112.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.64976.log.gz 2024-09-11T06:53:19.056 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.59523.log.gz 2024-09-11T06:53:19.057 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.66751.log.gz 2024-09-11T06:53:19.059 INFO:teuthology.orchestra.run.smithi112.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.71976.log.gz 2024-09-11T06:53:19.060 INFO:teuthology.orchestra.run.smithi112.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.85739.log.gz 2024-09-11T06:53:19.060 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.74685.log.gz 2024-09-11T06:53:19.063 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.75956.log.gz 2024-09-11T06:53:19.064 INFO:teuthology.orchestra.run.smithi112.stderr: 19.0% -- replaced with /var/log/ceph/cluster1-client.mirror.0.29720.log.gz 2024-09-11T06:53:19.065 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.41874.log.gz 2024-09-11T06:53:19.065 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.42503.log.gz 2024-09-11T06:53:19.066 INFO:teuthology.orchestra.run.smithi112.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.80695.log.gz 2024-09-11T06:53:19.068 INFO:teuthology.orchestra.run.smithi112.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.87071.log.gz 2024-09-11T06:53:19.069 INFO:teuthology.orchestra.run.smithi112.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.82930.log.gz 2024-09-11T06:53:19.089 INFO:teuthology.orchestra.run.smithi112.stderr: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.86679.log.gz 2024-09-11T06:53:19.097 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster1.osd.2 has been restored 2024-09-11T06:53:19.144 INFO:teuthology.orchestra.run.smithi112.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.40041.log.gz 2024-09-11T06:53:19.145 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.44092.log.gz 2024-09-11T06:53:19.145 INFO:teuthology.orchestra.run.smithi112.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.33829.log.gz 2024-09-11T06:53:19.145 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.58252.log.gz 2024-09-11T06:53:19.145 INFO:teuthology.orchestra.run.smithi112.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.34108.log.gz 2024-09-11T06:53:19.146 INFO:teuthology.orchestra.run.smithi112.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.58310.log.gz 2024-09-11T06:53:19.163 INFO:teuthology.orchestra.run.smithi112.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.37472.log.gz 2024-09-11T06:53:19.163 INFO:teuthology.orchestra.run.smithi112.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.65875.log.gz 2024-09-11T06:53:19.163 INFO:teuthology.orchestra.run.smithi112.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.76739.log.gz 2024-09-11T06:53:19.164 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.65910.log.gz 2024-09-11T06:53:19.164 INFO:teuthology.orchestra.run.smithi112.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.77018.log.gz 2024-09-11T06:53:19.164 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.77452.log.gz 2024-09-11T06:53:19.167 INFO:teuthology.orchestra.run.smithi112.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.55636.log.gz 2024-09-11T06:53:19.168 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.68333.log.gz 2024-09-11T06:53:19.175 INFO:teuthology.orchestra.run.smithi112.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.85135.log.gz 2024-09-11T06:53:19.179 INFO:teuthology.orchestra.run.smithi112.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.69081.log.gz 2024-09-11T06:53:19.180 INFO:teuthology.orchestra.run.smithi112.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.56198.log.gz 2024-09-11T06:53:19.180 INFO:teuthology.orchestra.run.smithi112.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.56256.log.gz 2024-09-11T06:53:19.181 INFO:teuthology.orchestra.run.smithi112.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.56721.log.gz 2024-09-11T06:53:19.182 INFO:teuthology.orchestra.run.smithi112.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.42099.log.gz 2024-09-11T06:53:19.182 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.73561.log.gz 2024-09-11T06:53:19.191 INFO:teuthology.orchestra.run.smithi112.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.72910.log.gz 2024-09-11T06:53:19.191 INFO:teuthology.orchestra.run.smithi112.stderr: 93.3% -- replaced with /var/log/ceph/cluster2-client.mirror.82177.log.gz 2024-09-11T06:53:19.197 INFO:teuthology.orchestra.run.smithi112.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.35918.log.gz 2024-09-11T06:53:19.204 INFO:teuthology.orchestra.run.smithi112.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.69980.log.gz 2024-09-11T06:53:19.205 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.78103.log.gz 2024-09-11T06:53:19.205 INFO:teuthology.orchestra.run.smithi112.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.78231.log.gz 2024-09-11T06:53:19.286 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.49934.log.gz 2024-09-11T06:53:19.293 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.78386.log.gz 2024-09-11T06:53:19.299 INFO:teuthology.orchestra.run.smithi112.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.79262.log.gz 2024-09-11T06:53:19.306 INFO:teuthology.orchestra.run.smithi112.stderr: 92.9% -- replaced with /var/log/ceph/cluster2-client.mirror.53902.log.gz 2024-09-11T06:53:19.312 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.74929.log.gz 2024-09-11T06:53:19.326 INFO:teuthology.orchestra.run.smithi112.stderr: 92.9% -- replaced with /var/log/ceph/cluster2-client.mirror.86162.log.gz 2024-09-11T06:53:19.332 INFO:teuthology.orchestra.run.smithi112.stderr: 93.2% -- replaced with /var/log/ceph/cluster1-client.mirror.86452.log.gz 2024-09-11T06:53:19.336 INFO:teuthology.orchestra.run.smithi112.stderr: 93.0% -- replaced with /var/log/ceph/cluster1-client.mirror.86272.log.gz 2024-09-11T06:53:19.341 INFO:teuthology.orchestra.run.smithi112.stderr: 93.3% -- replaced with /var/log/ceph/cluster2-client.mirror.85444.log.gz 2024-09-11T06:53:19.350 INFO:teuthology.orchestra.run.smithi112.stderr: 92.8% -- replaced with /var/log/ceph/cluster1-client.mirror.86091.log.gz 2024-09-11T06:53:19.355 INFO:teuthology.orchestra.run.smithi112.stderr: 92.9% -- replaced with /var/log/ceph/cluster1-client.mirror.86225.log.gz 2024-09-11T06:53:19.413 INFO:teuthology.orchestra.run.smithi112.stderr: 93.0% -- replaced with /var/log/ceph/cluster2-client.mirror.42124.log.gz 2024-09-11T06:53:19.417 INFO:teuthology.orchestra.run.smithi112.stderr: 88.4% -- replaced with /var/log/ceph/cluster1-client.admin.46195.log.gz 2024-09-11T06:53:19.419 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.63166.log.gz 2024-09-11T06:53:19.421 INFO:teuthology.orchestra.run.smithi112.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.69360.log.gz 2024-09-11T06:53:19.421 INFO:teuthology.orchestra.run.smithi112.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.69418.log.gz 2024-09-11T06:53:19.423 INFO:teuthology.orchestra.run.smithi112.stderr: 92.8% -- replaced with /var/log/ceph/cluster1-client.mirror.86044.log.gz 2024-09-11T06:53:19.431 INFO:teuthology.orchestra.run.smithi112.stderr: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.86405.log.gz 2024-09-11T06:53:19.432 INFO:teuthology.orchestra.run.smithi112.stderr: 93.3% -- replaced with /var/log/ceph/cluster2-client.mirror.86385.log.gz 2024-09-11T06:53:19.437 INFO:teuthology.orchestra.run.smithi112.stderr: 84.8% -- replaced with /var/log/ceph/cluster2-client.mirror.87911.log.gz 2024-09-11T06:53:19.438 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.51895.log.gz 2024-09-11T06:53:19.438 INFO:teuthology.orchestra.run.smithi112.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.58907.log.gz 2024-09-11T06:53:19.440 INFO:teuthology.orchestra.run.smithi112.stderr: 92.9% -- replaced with /var/log/ceph/cluster1-client.mirror.39048.log.gz 2024-09-11T06:53:19.440 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.76890.log.gz 2024-09-11T06:53:19.446 INFO:teuthology.orchestra.run.smithi112.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.81418.log.gz 2024-09-11T06:53:19.446 INFO:teuthology.orchestra.run.smithi112.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.64476.log.gz 2024-09-11T06:53:19.446 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.65352.log.gz 2024-09-11T06:53:19.446 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.59093.log.gz 2024-09-11T06:53:19.446 INFO:teuthology.orchestra.run.smithi112.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.74743.log.gz 2024-09-11T06:53:19.447 INFO:teuthology.orchestra.run.smithi112.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.59279.log.gz 2024-09-11T06:53:19.457 INFO:teuthology.orchestra.run.smithi112.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.82426.log.gz 2024-09-11T06:53:19.467 INFO:teuthology.orchestra.run.smithi112.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.30500.log.gz 2024-09-11T06:53:19.476 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.60085.log.gz 2024-09-11T06:53:19.480 INFO:teuthology.orchestra.run.smithi112.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.32930.log.gz 2024-09-11T06:53:19.553 INFO:teuthology.orchestra.run.smithi112.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.35732.log.gz 2024-09-11T06:53:19.558 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.37224.log.gz 2024-09-11T06:53:19.574 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48663.log.gz 2024-09-11T06:53:19.575 INFO:teuthology.orchestra.run.smithi112.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.60682.log.gz 2024-09-11T06:53:19.578 INFO:teuthology.orchestra.run.smithi112.stderr: 92.7% -- replaced with /var/log/ceph/cluster1-client.mirror.46028.log.gz 2024-09-11T06:53:19.580 INFO:teuthology.orchestra.run.smithi112.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.70480.log.gz 2024-09-11T06:53:19.580 INFO:teuthology.orchestra.run.smithi112.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.64883.log.gz 2024-09-11T06:53:19.581 INFO:teuthology.orchestra.run.smithi112.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.68988.log.gz 2024-09-11T06:53:19.582 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45758.log.gz 2024-09-11T06:53:19.584 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.60271.log.gz 2024-09-11T06:53:19.589 INFO:teuthology.orchestra.run.smithi112.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.33302.log.gz 2024-09-11T06:53:19.590 INFO:teuthology.orchestra.run.smithi112.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.43344.log.gz 2024-09-11T06:53:19.598 INFO:teuthology.orchestra.run.smithi112.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.57690.log.gz 2024-09-11T06:53:19.601 INFO:teuthology.orchestra.run.smithi112.stderr: 93.3% -- replaced with /var/log/ceph/cluster2-client.mirror.80876.log.gz 2024-09-11T06:53:19.632 INFO:teuthology.orchestra.run.smithi112.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.80490.log.gz 2024-09-11T06:53:19.939 INFO:teuthology.orchestra.run.smithi112.stderr: 93.2% -- replaced with /var/log/ceph/cluster1-client.mirror.86632.log.gz 2024-09-11T06:53:19.941 INFO:teuthology.orchestra.run.smithi112.stderr: 92.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67219.log.gz 2024-09-11T06:53:19.941 INFO:teuthology.orchestra.run.smithi112.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.31961.log.gz 2024-09-11T06:53:19.941 INFO:teuthology.orchestra.run.smithi112.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.35790.log.gz 2024-09-11T06:53:19.942 INFO:teuthology.orchestra.run.smithi112.stderr: 92.8% -- replaced with /var/log/ceph/cluster1-client.mirror.86182.log.gz 2024-09-11T06:53:19.942 INFO:teuthology.orchestra.run.smithi112.stderr: 92.7% -- replaced with /var/log/ceph/cluster1-client.mirror.85954.log.gz 2024-09-11T06:53:19.942 INFO:teuthology.orchestra.run.smithi112.stderr: 92.9% -- replaced with /var/log/ceph/cluster1-client.mirror.42144.log.gz 2024-09-11T06:53:19.942 INFO:teuthology.orchestra.run.smithi112.stderr: 94.1% -- replaced with /var/log/ceph/cluster2.audit.log.gz 2024-09-11T06:53:20.112 INFO:teuthology.orchestra.run.smithi112.stderr: 94.5% -- replaced with /var/log/ceph/cluster1-client.mirror.29862.log.gz 2024-09-11T06:53:20.113 INFO:teuthology.orchestra.run.smithi112.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.32651.log.gz 2024-09-11T06:53:20.137 INFO:teuthology.orchestra.run.smithi112.stderr: 94.4% -- replaced with /var/log/ceph/cluster1-client.mirror.29802.log.gz 2024-09-11T06:53:20.205 INFO:teuthology.orchestra.run.smithi112.stderr: 93.3% -- replaced with /var/log/ceph/cluster2-client.mirror.86522.log.gz 2024-09-11T06:53:20.231 INFO:teuthology.orchestra.run.smithi112.stderr: 96.2% -- replaced with /var/log/ceph/cluster2-client.mirror.82219.log.gz 2024-09-11T06:53:20.540 INFO:teuthology.orchestra.run.smithi112.stderr: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.85464.log.gz 2024-09-11T06:53:20.542 INFO:teuthology.orchestra.run.smithi112.stderr: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.86362.log.gz 2024-09-11T06:53:20.649 INFO:teuthology.orchestra.run.smithi112.stderr: 92.9% -- replaced with /var/log/ceph/cluster2-client.mirror.86067.log.gz 2024-09-11T06:53:21.418 INFO:teuthology.orchestra.run.smithi112.stderr: 94.4% -- replaced with /var/log/ceph/cluster2-client.mirror.1.29862.log.gz 2024-09-11T06:53:23.468 INFO:teuthology.orchestra.run.smithi112.stderr: 95.5% -- replaced with /var/log/ceph/cluster2-client.mirror.67389.log.gz 2024-09-11T06:53:25.453 INFO:teuthology.orchestra.run.smithi112.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.80232.log.gz 2024-09-11T06:53:25.511 INFO:teuthology.orchestra.run.smithi112.stderr: 95.5% -- replaced with /var/log/ceph/cluster2-client.mirror.87635.log.gz 2024-09-11T06:53:25.947 INFO:teuthology.orchestra.run.smithi112.stderr: 94.1% -- replaced with /var/log/ceph/cluster2-client.mirror.0.29802.log.gz 2024-09-11T06:53:27.917 INFO:teuthology.orchestra.run.smithi112.stderr: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.42340.log.gz 2024-09-11T06:53:27.998 INFO:teuthology.orchestra.run.smithi112.stderr: 96.0% -- replaced with /var/log/ceph/cluster2-client.mirror.87709.log.gz 2024-09-11T06:53:28.146 INFO:teuthology.orchestra.run.smithi112.stderr: 96.1% -- replaced with /var/log/ceph/cluster2-client.mirror.87260.log.gz 2024-09-11T06:53:28.492 INFO:teuthology.orchestra.run.smithi112.stderr: 95.5% -- replaced with /var/log/ceph/cluster2-client.mirror.82358.log.gz 2024-09-11T06:53:30.852 INFO:teuthology.orchestra.run.smithi112.stderr: 95.9% -- replaced with /var/log/ceph/cluster2-client.mirror.88635.log.gz 2024-09-11T06:53:31.036 INFO:teuthology.orchestra.run.smithi112.stderr: 95.5% -- replaced with /var/log/ceph/cluster2-client.mirror.80945.log.gz 2024-09-11T06:53:31.726 INFO:teuthology.orchestra.run.smithi112.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.82242.log.gz 2024-09-11T06:53:32.420 INFO:teuthology.orchestra.run.smithi112.stderr: 95.5% -- replaced with /var/log/ceph/cluster2-client.mirror.80147.log.gz 2024-09-11T06:53:33.048 INFO:teuthology.orchestra.run.smithi112.stderr: 95.8% -- replaced with /var/log/ceph/cluster2-client.mirror.53971.log.gz 2024-09-11T06:53:33.065 INFO:teuthology.orchestra.run.smithi112.stderr: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.87463.log.gz 2024-09-11T06:53:33.349 INFO:teuthology.orchestra.run.smithi112.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.87542.log.gz 2024-09-11T06:53:34.193 INFO:teuthology.orchestra.run.smithi112.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.81030.log.gz 2024-09-11T06:53:34.447 INFO:teuthology.orchestra.run.smithi112.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.88533.log.gz 2024-09-11T06:53:35.851 INFO:teuthology.orchestra.run.smithi112.stderr: 95.8% -- replaced with /var/log/ceph/cluster2-client.mirror.42236.log.gz 2024-09-11T06:53:37.970 INFO:teuthology.orchestra.run.smithi112.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.30297.log.gz 2024-09-11T06:53:39.399 INFO:teuthology.orchestra.run.smithi112.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.54501.log.gz 2024-09-11T06:53:39.412 INFO:teuthology.orchestra.run.smithi112.stderr: 95.9% -- replaced with /var/log/ceph/cluster2-client.mirror.87934.log.gz 2024-09-11T06:53:40.391 INFO:teuthology.orchestra.run.smithi112.stderr: 95.8% -- replaced with /var/log/ceph/cluster2-client.mirror.46223.log.gz 2024-09-11T06:53:40.818 INFO:teuthology.orchestra.run.smithi112.stderr: 95.8% -- replaced with /var/log/ceph/cluster2-client.mirror.30173.log.gz 2024-09-11T06:53:41.914 INFO:teuthology.orchestra.run.smithi112.stderr: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.88148.log.gz 2024-09-11T06:53:41.997 INFO:teuthology.orchestra.run.smithi112.stderr: 95.8% -- replaced with /var/log/ceph/cluster2-client.mirror.88431.log.gz 2024-09-11T06:53:44.246 INFO:teuthology.orchestra.run.smithi112.stderr: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.46120.log.gz 2024-09-11T06:53:45.307 INFO:teuthology.orchestra.run.smithi112.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.88021.log.gz 2024-09-11T06:53:45.334 INFO:teuthology.orchestra.run.smithi112.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.39189.log.gz 2024-09-11T06:53:46.132 INFO:teuthology.orchestra.run.smithi112.stderr: 95.8% -- replaced with /var/log/ceph/cluster2-client.mirror.87788.log.gz 2024-09-11T06:53:46.188 INFO:teuthology.orchestra.run.smithi112.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.87390.log.gz 2024-09-11T06:53:46.945 INFO:teuthology.orchestra.run.smithi112.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.54076.log.gz 2024-09-11T06:53:47.925 INFO:teuthology.orchestra.run.smithi112.stderr: 95.9% -- replaced with /var/log/ceph/cluster2-client.mirror.67264.log.gz 2024-09-11T06:53:48.277 INFO:teuthology.orchestra.run.smithi112.stderr: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.87317.log.gz 2024-09-11T06:53:51.322 INFO:teuthology.orchestra.run.smithi112.stderr: 96.1% -- replaced with /var/log/ceph/cluster2-client.mirror.88283.log.gz 2024-09-11T06:53:52.258 INFO:teuthology.orchestra.run.smithi112.stderr: 91.1% -- replaced with /var/log/ceph/cluster2-mon.a.log.gz 2024-09-11T06:53:52.757 INFO:teuthology.orchestra.run.smithi112.stderr: 95.9% -- replaced with /var/log/ceph/cluster2-client.mirror.54352.log.gz 2024-09-11T06:53:55.500 INFO:teuthology.orchestra.run.smithi112.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.39093.log.gz 2024-09-11T06:53:57.439 INFO:teuthology.orchestra.run.smithi112.stderr: 94.3% -- replaced with /var/log/ceph/cluster2-client.mirror.29757.log.gz 2024-09-11T06:54:06.764 INFO:teuthology.orchestra.run.smithi006.stderr: 94.3% -- replaced with /var/log/ceph/cluster1-osd.1.log.gz 2024-09-11T06:54:10.179 INFO:teuthology.orchestra.run.smithi006.stderr: 94.4% -- replaced with /var/log/ceph/cluster1-osd.2.log.gz 2024-09-11T06:54:24.091 INFO:teuthology.orchestra.run.smithi006.stderr: 94.3% -- replaced with /var/log/ceph/cluster1-osd.0.log.gz 2024-09-11T06:54:24.092 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-11T06:54:24.092 INFO:teuthology.orchestra.run.smithi006.stderr:real 1m7.724s 2024-09-11T06:54:24.092 INFO:teuthology.orchestra.run.smithi006.stderr:user 2m43.301s 2024-09-11T06:54:24.093 INFO:teuthology.orchestra.run.smithi006.stderr:sys 0m5.606s 2024-09-11T06:54:26.480 INFO:teuthology.orchestra.run.smithi112.stderr: 94.4% -- replaced with /var/log/ceph/cluster1-client.mirror.1.29757.log.gz 2024-09-11T06:54:31.265 INFO:teuthology.orchestra.run.smithi112.stderr: 94.5% -- replaced with /var/log/ceph/cluster2-client.mirror.29729.log.gz 2024-09-11T06:55:09.992 INFO:teuthology.orchestra.run.smithi112.stderr: 94.4% -- replaced with /var/log/ceph/cluster1-client.mirror.0.29729.log.gz 2024-09-11T06:58:32.395 INFO:teuthology.orchestra.run.smithi112.stderr: 94.3% -- replaced with /var/log/ceph/cluster2-osd.2.log.gz 2024-09-11T06:58:38.298 INFO:teuthology.orchestra.run.smithi112.stderr: 94.1% -- replaced with /var/log/ceph/cluster2-osd.1.log.gz 2024-09-11T06:59:04.029 INFO:teuthology.orchestra.run.smithi112.stderr: 94.1% -- replaced with /var/log/ceph/cluster2-osd.0.log.gz 2024-09-11T06:59:04.030 INFO:teuthology.orchestra.run.smithi112.stderr: 2024-09-11T06:59:04.030 INFO:teuthology.orchestra.run.smithi112.stderr:real 5m47.642s 2024-09-11T06:59:04.030 INFO:teuthology.orchestra.run.smithi112.stderr:user 7m37.867s 2024-09-11T06:59:04.030 INFO:teuthology.orchestra.run.smithi112.stderr:sys 0m23.069s 2024-09-11T06:59:04.031 INFO:tasks.ceph:Archiving logs... 2024-09-11T06:59:04.032 DEBUG:teuthology.misc:Transferring archived files from smithi006:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-10_21:16:03-rbd-squid-distro-default-smithi/7900605/remote/smithi006/log 2024-09-11T06:59:04.033 DEBUG:teuthology.orchestra.run.smithi006:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-11T06:59:22.638 DEBUG:teuthology.misc:Transferring archived files from smithi112:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-10_21:16:03-rbd-squid-distro-default-smithi/7900605/remote/smithi112/log 2024-09-11T06:59:22.639 DEBUG:teuthology.orchestra.run.smithi112:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-11T06:59:57.602 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-11T06:59:57.631 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-11T06:59:57.632 DEBUG:teuthology.orchestra.run.smithi006:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-11T06:59:57.635 DEBUG:teuthology.orchestra.run.smithi112:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-11T06:59:57.710 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-11T06:59:57.941 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-11T06:59:57.941 DEBUG:teuthology.orchestra.run.smithi006:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-11T06:59:57.944 DEBUG:teuthology.orchestra.run.smithi112:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-11T06:59:57.971 INFO:teuthology.orchestra.run.smithi006.stdout: remote refid st t when poll reach delay offset jitter 2024-09-11T06:59:57.971 INFO:teuthology.orchestra.run.smithi006.stdout:============================================================================== 2024-09-11T06:59:57.971 INFO:teuthology.orchestra.run.smithi006.stdout:*hv01.front.sepi 67.205.162.81 3 u 122 128 377 0.085 -1.974 0.143 2024-09-11T06:59:57.971 INFO:teuthology.orchestra.run.smithi006.stdout:+hv02.front.sepi 63.231.80.2 3 u 110 128 377 0.123 +0.461 0.050 2024-09-11T06:59:57.971 INFO:teuthology.orchestra.run.smithi006.stdout:+hv03.front.sepi 74.6.168.72 3 u 127 128 377 0.124 +0.350 0.115 2024-09-11T06:59:57.971 INFO:teuthology.orchestra.run.smithi006.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2024-09-11T06:59:58.019 INFO:teuthology.orchestra.run.smithi112.stdout: remote refid st t when poll reach delay offset jitter 2024-09-11T06:59:58.020 INFO:teuthology.orchestra.run.smithi112.stdout:============================================================================== 2024-09-11T06:59:58.020 INFO:teuthology.orchestra.run.smithi112.stdout:+hv01.front.sepi 67.205.162.81 3 u 33 64 377 0.108 -3.853 1.682 2024-09-11T06:59:58.020 INFO:teuthology.orchestra.run.smithi112.stdout:*hv02.front.sepi 63.231.80.2 3 u 36 64 377 0.104 -3.913 3.096 2024-09-11T06:59:58.020 INFO:teuthology.orchestra.run.smithi112.stdout:+hv03.front.sepi 74.6.168.72 3 u 51 64 377 0.085 +0.648 2.819 2024-09-11T06:59:58.020 INFO:teuthology.orchestra.run.smithi112.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2024-09-11T06:59:58.021 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-11T06:59:58.182 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-11T06:59:58.183 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-11T06:59:58.323 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-11T06:59:58.457 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-11T06:59:58.598 INFO:teuthology.task.internal:Duration was 3327.866253 seconds 2024-09-11T06:59:58.599 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-11T06:59:58.770 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-11T06:59:58.771 DEBUG:teuthology.orchestra.run.smithi006:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-11T06:59:58.773 DEBUG:teuthology.orchestra.run.smithi112:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-11T06:59:58.921 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-11T06:59:58.921 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi006.front.sepia.ceph.com 2024-09-11T06:59:58.921 DEBUG:teuthology.orchestra.run.smithi006:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-11T06:59:58.932 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi112.front.sepia.ceph.com 2024-09-11T06:59:58.932 DEBUG:teuthology.orchestra.run.smithi112:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-11T06:59:58.943 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-11T06:59:58.943 DEBUG:teuthology.orchestra.run.smithi006:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-11T06:59:58.975 DEBUG:teuthology.orchestra.run.smithi112:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-11T06:59:59.017 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-11T06:59:59.017 DEBUG:teuthology.orchestra.run.smithi006:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-11T06:59:59.055 DEBUG:teuthology.orchestra.run.smithi112:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-11T06:59:59.264 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-11T06:59:59.275 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-11T06:59:59.275 DEBUG:teuthology.orchestra.run.smithi006:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-11T06:59:59.299 DEBUG:teuthology.orchestra.run.smithi112:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-11T06:59:59.338 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-11T06:59:59.355 DEBUG:teuthology.orchestra.run.smithi006:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-11T06:59:59.357 DEBUG:teuthology.orchestra.run.smithi112:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-11T06:59:59.389 INFO:teuthology.orchestra.run.smithi006.stdout:kernel.core_pattern = core 2024-09-11T06:59:59.405 INFO:teuthology.orchestra.run.smithi112.stdout:kernel.core_pattern = core 2024-09-11T06:59:59.436 DEBUG:teuthology.orchestra.run.smithi006:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-11T06:59:59.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T06:59:59.474 DEBUG:teuthology.orchestra.run.smithi112:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-11T06:59:59.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T06:59:59.481 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-11T06:59:59.491 INFO:teuthology.task.internal:Transferring archived files... 2024-09-11T06:59:59.492 DEBUG:teuthology.misc:Transferring archived files from smithi006:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-10_21:16:03-rbd-squid-distro-default-smithi/7900605/remote/smithi006 2024-09-11T06:59:59.492 DEBUG:teuthology.orchestra.run.smithi006:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-11T06:59:59.539 DEBUG:teuthology.misc:Transferring archived files from smithi112:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-10_21:16:03-rbd-squid-distro-default-smithi/7900605/remote/smithi112 2024-09-11T06:59:59.539 DEBUG:teuthology.orchestra.run.smithi112:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-11T06:59:59.561 INFO:teuthology.task.internal:Removing archive directory... 2024-09-11T06:59:59.562 DEBUG:teuthology.orchestra.run.smithi006:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-11T06:59:59.578 DEBUG:teuthology.orchestra.run.smithi112:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-11T06:59:59.606 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-11T06:59:59.616 INFO:teuthology.task.internal:Not uploading archives. 2024-09-11T06:59:59.616 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-11T06:59:59.671 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-11T06:59:59.671 DEBUG:teuthology.orchestra.run.smithi006:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-11T06:59:59.673 DEBUG:teuthology.orchestra.run.smithi112:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-11T06:59:59.677 INFO:teuthology.orchestra.run.smithi006.stdout: 658672 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 11 06:59 /home/ubuntu/cephtest 2024-09-11T06:59:59.679 INFO:teuthology.orchestra.run.smithi112.stdout: 658615 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 11 06:59 /home/ubuntu/cephtest 2024-09-11T06:59:59.680 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-11T06:59:59.750 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-11T06:59:59.773 INFO:teuthology.run:Summary data: description: rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-zlib supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-stress-workunit-min-compat-client-octopus} duration: 3327.86625289917 flavor: default owner: scheduled_teuthology@teuthology success: true 2024-09-11T06:59:59.774 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-11T06:59:59.854 INFO:teuthology.run:pass